initial commit

README.rst

@@ -0,0 +1,76 @@
+Frank CI
+========
+
+Frank CI aims to be a minimal (but smart) continuous integration system. 
+
+To get started you simply deploy from with a WSGI server (e.g NGNIX) and configure
+your git repository (github and others) to post a load after each push. 
+
+If your repository includes a top level file called `.frank.yaml` your
+frank instance will react uppon the commands found in the file.
+
+Frank commands
+--------------
+
+Frank understands the following file format::
+
+   .. text: yaml
+
+    commands:
+     - do_this
+     - do_that
+     - run_function_python
+     - deploy_there
+
+    do_this:
+      - shell: echo "this"
+     
+    do_that:
+      shell:
+       - cd foo
+       - touch bar.txt 
+
+    run_function_python:
+      python:
+       - foo.bar:callable_function
+
+
+The `commands` is a mandatory list of commnads to excute after each push.
+The order of the commands is how the will be executed. 
+
+Following the key `commands` are definitions of each command. Command can
+by eith `shell` commands or some callable python object when specified as `python`
+and given with full import path `foo.bar` and following callable object after
+the collon.
+
+
+Example 1:
+---------- 
+The following example will run the tests, build sphinx documentation and 
+publish static HTML files via rsync to another server::
+
+    .. text: yaml
+
+    commads:
+     - test
+     - build_sphinx
+     - publish
+
+    test:
+      shell:
+       - pip install -e .
+       - python setup.py test
+
+    build_sphinx:
+      shell:
+       cwd: docs
+       cmd: make html
+
+    publish:
+      shell:
+       - rsync -avz docs/build/html/ docserver.company.com:/var/www/docs/
+   
+This example demonstrates running multiple commands after the keyword `shell`.
+You can also specify any keyword that the function `subprocess.Popen` accepts.
+For example you can specify envirnment variables or the working directroy, as 
+in the example with the keyword `cwd`. 

frank/conf.py

@@ -0,0 +1,4 @@
+
+GITHUB_USER, GITUB_PASSWORD = "user", "secret"
+POST_KEY = '6361675'
+

frank/frank.py

@@ -0,0 +1,296 @@
+"""
+Frank can be configured using the following yaml::
+	.frank.yml
+
+	commads:
+	 - build
+	 - test
+	 - publish
+	 - deploy
+
+
+	deploy:
+	 - cmd 
+	 - runif:
+	   - test
+	   - newtag
+	 
+	newtag:
+	 - python:
+	   - frank.actions:detect_new_tag
+
+	publish:
+	  - shell:
+		- cd docs; make html
+	  - python:
+		- frank.actions:recursive_copy 
+
+The sections commands is a simple list of command you would like to define. 
+This section is optional. It's main purpose is for you to decalre 
+which commands frank should execute upon recieving a load.
+
+Each command is a dictionary with the following possible keys::
+
+    cmd_name:
+      - cmd  # not mandatory if you include it in the list of commands 
+      - runif # a conditional to determine whether to run or skip the command
+              # it can contain multiple directives which can be python 
+              # code to execute 
+              # or shell code 
+ 
+
+For example, let's say you would like to build the sphinx documentation 
+of your project after every push. You start by defining a command 
+build_docs in the following way::
+
+   build_docs:
+    - cmd
+    - shell:
+      - cd docs
+      - make html
+
+You could also specify::
+
+   build_docs:
+    - cmd
+    - shell: cd docs; make html
+
+or::
+
+   build_docs:
+    - shell:
+      - cwd: docs
+      - cmd: make html
+
+
+This tells that `build_docs` is a command that executes after every push.
+It will execute a shell and will run the shell commands
+`cd docs; make html`. Pretty straight forward!
+
+Now, let's refine. Suppose we want to build the docs only if the docs changed,
+thus ignoring completely, changes in the embeded docs strings*::
+
+   build_docs:
+    - cmd
+    - runif:
+      - shell:
+        - git --no-pager diff --name-status HEAD~1 | grep -v docs
+    - shell: cd docs; make html
+
+Now, the command will run if the latest git commit have changed the docs
+directory. Since the grep command will return exit with 1.
+Alternatively, the conditional to run can be some python code that returns
+True or False. Here is how to specify this:
+
+   build_docs:
+    - cmd
+    - runif:
+      - python: frank.githubparsers:detect_changes_in_docs
+    - shell: cd docs; make html
+
+This, will invoke the method ``detect_changes_in_docs`` which is
+found in the module `frank.githubparsers`.
+If this function will return True the shell command will execute.
+If the methods takes some arguments, they could also be given.
+Suppose the function is called `detect_changes_in` and this function
+takes a single paramter called path we can configure the
+build_docs command in the following way:
+
+   build_docs:
+    - cmd
+    - runif:
+      - python:
+        - function: frank.githubparsers:detect_changes_in
+        - args:
+          - path: docs
+
+* This is probably a lame idea, but it's good for demonstration.
+So, do write doc-strings, and do build  your docs often.
+"""
+
+import hmac
+import os
+import subprocess
+import subprocess as sp
+import hashlib
+import yaml
+from flask import Flask, request, abort
+import conf
+from shell import Shell
+from collections import OrderedDict, namedtuple
+
+PythonCode = namedtupe('PythonCode', path, args, kwargs, code)
+
+def override_run(self, command, **kwargs):
+    """
+    Override Shell.run to handle exceptions and accept kwargs
+    that Popen accepts
+    """
+    self.last_command = command
+    command_bits = self._split_command(command)
+    _kwargs = {
+        'stdout': subprocess.PIPE,
+        'stderr': subprocess.PIPE,
+        'universal_newlines': True,
+    }
+    if kwargs:
+        for kw in kwargs:
+            _kwargs[kw] = kwargs[kw]
+
+        _kwargs['shell'] = True
+
+    if self.has_input:
+        _kwargs['stdin'] = subprocess.PIPE
+
+    try:
+        self._popen = subprocess.Popen(
+            command_bits,
+            **_kwargs
+        )
+    except Exception as E:
+        self.exception = E
+        return self
+
+    self.pid = self._popen.pid
+
+    if not self.has_input:
+        self._communicate()
+    return self
+
+Shell.run = override_run
+
+
+def ordered_load(stream, Loader=yaml.Loader, selfect_pairs_hook=OrderedDict):
+
+    class OrderedLoader(Loader):
+        pass
+
+    def construct_mapping(loader, node):
+        loader.flatten_mapping(node)
+        return selfect_pairs_hook(loader.construct_pairs(node))
+
+    OrderedLoader.add_constructor(
+        yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG,
+        construct_mapping)
+    return yaml.load(stream, OrderedLoader)
+
+app = Flask(__name__)
+app.config.from_object(conf)
+
+
+def parse_branch_gh(request_json):
+    """
+    parse the branch to clone from a github payload
+    "ref": "refs/heads/develop", -> should return develop
+    """
+    return request_json['ref'].split('/')[-1]
+
+
+def parse_yaml(clone_dest):
+    os.chdir(clone_dest)
+    if os.path.exists('.frank.yaml'):
+        with open('.frank.yaml') as f:
+            y = ordered_load(f, yaml.SafeLoader)
+    return y
+
+
+def load_actions(yaml):
+    pass
+
+
+def report_success(results):
+    pass
+
+
+def report_failure(results):
+    pass
+
+
+def run_action(axn):
+    results = []
+    if isinstance(axn, list):
+
+        if axn[0] == 'shell':
+            for cmd in axn[1:]:
+                sh = Shell()
+                assert isinstance(cmd, str)
+                sh.run(cmd)
+                results.append(sh)
+                if sh.code:
+                    break
+        if axn[0] == 'python':
+            for func in axn[1:]:
+                mod, f = func.split(':')
+                mod = importlib.import(mod)
+                f = getattr(mod, f)
+                res = f()
+                results.append(PythonCode(func, None, None, res))
+
+    elif isinstance(axn, OrderedDict):
+
+        if 'shell' in axn:
+
+            sh = Shell()
+            cmd = axn['shell'].pop('cmd')
+            assert isinstance(cmd, str)
+            kwargs = axn['shell']
+            sh.run(cmd, **kwargs)
+            results.append(sh)
+
+    return results
+
+
+def clone(clone_url, branch, depth=1):
+    cmd = ('git clone --depth={d} -b {branch} --single-branch '
+           '{git_url} {dir}'.format(d=depth, branch=branch,
+                                    git_url=clone_url, dir=branch))
+    pull = sp.Popen(cmd, stderr=sp.STDOUT, shell=True)
+    out, err = pull.communicate()
+    return out, err
+
+
+@app.route('/', methods=['POST'])
+def start():
+    """
+    main logic:
+     1. listen to post
+     2a if authenticated post do:
+     3.    clone the latest commit (--depth 1)
+     4.    parse yaml config
+     5.    for each command in the config
+     7.        run command
+     8.        report success or failure
+    """
+    # This is authentication for github only
+    # We could\should check for other hostings
+    ans = hmac.new(app.config['POST_KEY'], request.data,
+                   hashlib.sha1).hexdigest()
+    secret = request.headers['X-Hub-Signature'].split('=')[-1]
+    if ans != secret:
+        return abort(500)
+    request_as_json = request.get_json()
+    clone_dest = parse_branch_gh(request_as_json)
+    repo_name = request_as_json["repository"]['name']
+    try:
+        o, e = clone(request_as_json['repository']['ssh_url'], clone_dest)
+    except Exception as E:
+        print E, E.message	
+    # parse yaml is still very crude ...
+    # it could yield selfect with a run method 
+    # thus:
+    # for action in parse_yaml(clone_dest):
+    #     action.run()
+    # 
+    # this should also handle dependencies, 
+    # the following implementation is very crud 
+    failed = None
+    for action in parse_yaml(clone_dest):
+        results = run_action(action)
+    
+    if any([result.code for result in results]):
+        report_failure(results)
+    else:
+        report_success(results)
+
+if __name__ == '__main__':
+    app.run(host='0.0.0.0', debug=True)

requirements.txt

@@ -0,0 +1,4 @@
+flask
+PyYAML
+shell
+spur

setup.py

@@ -0,0 +1,13 @@
+from setuptools import setup, find_packages
+
+
+setup(name='frank',
+      version='0.0.1',
+      license="AGPL",
+      packages=find_packages(exclude=['tests']),
+      )
+
+
+# licenses
+
+# Add mention that we use Shell, flask and PyYAML

test/test_frank.py

@@ -0,0 +1,137 @@
+from collections import OrderedDict
+from frank.frank import parse_yaml, ordered_load, run_action
+from frank.frank import Shell
+
+test_yaml = """
+    commands:
+     - test
+     - build_sphinx
+     - publish
+
+    test:
+     shell:
+      - pip install -e .
+      - python setup.py test
+
+    build_sphinx:
+      shell:
+       cwd: docs
+       cmd: make html
+    """
+
+
+def mk_yaml():
+    with open('.frank.yaml', 'w') as f:
+        f.write(test_yaml)
+
+mk_yaml()
+
+
+def test_parse_yaml():
+    y = parse_yaml('.')
+    assert isinstance(y, dict)
+
+
+def test_ordered_load():
+    ans = ordered_load("""
+    commands:
+     - test
+     - build_sphinx
+     - publish""")
+
+    assert isinstance(ans['commands'], list)
+    ans = ordered_load("""
+    commands:
+     - test
+     - build_sphinx
+     - publish
+
+    test:
+     shell:
+      - pip install -e .
+      - python setup.py test
+     """)
+    assert isinstance(ans['test'], OrderedDict)
+    assert isinstance(ans['test']['shell'], list)
+
+    # If 'shell' is a list run the commands as ordered
+
+    ans = ordered_load("""
+    commands:
+     - test
+     - build_sphinx
+     - publish
+
+    test:
+     shell:
+      - pip install -e .
+      - python setup.py test
+
+    build_sphinx:
+      shell:
+       cwd: docs
+       cmd: make html
+    """)
+    assert isinstance(ans['test']['shell'], list)
+    assert isinstance(ans['build_sphinx'], OrderedDict)
+    assert isinstance(ans['build_sphinx']['shell'], OrderedDict)
+    # If 'shell' is an OrderedDict run the command with the keywords given
+
+
+def test_Shell_run():
+    sh = Shell()
+    sh.run('ls -l', cwd='/tmp/')
+    assert not sh.code
+    assert isinstance(sh.output(), list)
+    sh = Shell()
+    sh.run('ls -l', cwd='/no-such-directory/')
+    assert sh.pid is None
+    assert isinstance(sh.exception, OSError)
+
+
+def test_run_action_shell():
+    # test shell commands
+    axn = ['shell', 'touch /tmp/testaxn', 'touch /tmp/testaxn2',
+           'rm /tmp/testaxn']
+    results = run_action(axn)
+
+    # all these commands should have succeeded
+    assert not any([result.code for result in results])
+    axn = ['shell', 'touch /nosuchdir/testaxn', 'touch /tmp/testaxn2',
+           'rm /tmp/testaxn']
+    results = run_action(axn)
+    assert len(results) == 1
+    assert results[0].code == 1
+    axn = ['shell', 'touch /tmp/works', 'touch /nosuchdir/fails',
+           'touch /tmp/testaxn2', 'rm /tmp/testaxn']
+    results = run_action(axn)
+    assert len(results) == 2
+    assert results[0].code == 0
+    assert results[1].code == 1
+
+
+def test_run_action_shell_with_kwargs():
+    # test shell commands
+    axn = OrderedDict([('shell', OrderedDict([('cwd', 'tmp'),
+                      ('cmd', 'touch foo')]))])
+    results = run_action(axn)
+    assert len(results) == 1
+    assert results[0].code == 0
+    axn = OrderedDict([('shell', OrderedDict([('cwd', 'nosuchdir'),
+                      ('cmd', 'touch foo')]))])
+    results = run_action(axn)
+    assert len(results) == 1
+    assert isinstance(results[0].exception, OSError)
+    axn = OrderedDict([('shell', OrderedDict([('cwd', '/tmp'),
+                      ('cmd', 'touch /nosuchdir/foo')]))])
+    results = run_action(axn)
+    assert len(r)esults) == 1
+    assert results[0].code == 1
+    # missing arguments to touch
+    axn = OrderedDict([('shell', OrderedDict([('cwd', '/tmp'),
+                      ('cmd', 'touch')]))])
+    assert len(results) == 1
+    assert results[0].code == 1
+    assert ['touch: missing file operand',
+            "Try `touch --help' for more information."] == 
+                results[0].errors()

test_requirements.txt

@@ -0,0 +1 @@
+pytest-flask