A Mercurial Testing and Coverage Hook
A Mercurial Testing and Coverage Hook
I've recently migrated from Subversion to Mercurial for my personal coding projects. I wanted to get more familiar with distributed version control systems and after reading up on the choices, I felt Mercurial was the best fit for my needs.
At the same time, I've been trying to be more diligent about applying good testing practices to my development routine so I thought this would be a good opportunity to take advantage of Mercurial's event hooks.
Like most version control systems, Mercurial provides a way for custom actions to be executed at certain points in the version control work flow (e.g. commit, update, push, pull). Common uses of these "hooks" are to send out email notifications when changes are checked in or to verify that a bug number is referenced in the check-in comments. One other fairly common use of an event hook—that I was particularly interested in—is to run a test suite across the code base to verify that all tests pass. This type of action can be tied to Mercurial's precommit event and if the test fails, the commit is aborted. This ensures that you never check in changes that fail any of your tests.
Running Nose on precommit
Adding the following to your repository's .hg/hgrc file tells Mercurial to
run the external command nosetests tests prior to committing a change to the
repository. If Nose succeeds (exits with 0), Mercurial will continue with the
commit. If Nose fails (exits with a non-zero value), Mercurial will abort the
change.
[hooks] precommit = nosetests tests
The syntax used here is for an external hook, that is, Mercurial calls the
external command with the specified arguments. The script has access to
limited information through a set of environment variables prefixed with
HG_. Mercurial also supports in-process hooks defined as a Python
callable. These hooks are passed arguments that allows the script to access
detailed information about the changeset, files and revisions. The syntax is
slightly different:
[hooks] precommit = python:myext.myhooks.run_test
In this case, Mercurial imports the myext.myhooks module (which must be in
your PYTHONPATH) and calls run_test. For the simple case of running Nose
on a standard test suite, either of these approaches works well, although the
external version is probably a bit simpler and easier to wrap in a shell
script if your test command requires a long list of arguments.
Adding a Minimum Code Coverage Hook
Checking if the code repository passes or fails a test suite is pretty simple,
especially since the exit code of Nose is just what we need to signal the
success or failure to Mercurial. But I also wanted to verify that my test
suite had a reasonable amount of coverage of my code base. While I can just
enable Nose's coverage plugin to calculate the percent coverage, I can't
simply test the exit code since it says nothing about the coverage amount.
For that I needed to parse the output of coverage.py. This is fairly simple
but there is one wrinkle. The output for covering one module is slightly
different than for covering multiple modules. There is no "TOTAL" line when
only one module is being processed.
When you run Nose with code coverage enabled, you get output like the following:
pfa-ct-imac01:code david$ nosetests --with-coverage calc.py E... ====================================================================== ERROR: test_constants (calc.CalcTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/david/code/calc.py", line 18, in test_constants self.assertAlmostEquals(3.1415926, calc(['pi'])) File "/Users/david/code/calc.py", line 6, in calc val = eval(expr) File "<string>", line 1, in <module> NameError: name 'pi' is not defined Name Stmts Exec Cover Missing ------------------------------------- calc 20 16 80% 21-25 ---------------------------------------------------------------------- Ran 4 tests in 0.005s FAILED (errors=1) pfa-ct-imac01:code david$
I needed to parse the coverage summary (the 80% value) from this output, but
since I was going to that trouble, I thought I'd parse out some of the testing
statistics, too:
def run_nose(args): """Run nosetests with coverage enabled, passing along the additional arguments. Parse out the test results (number of tests run, number of errors, number of failures, execution time) as well as the percent code coverage. Return all of these stats along with the nose output. """ def parse_tests(out): """Parse test stats out of summary and status messages. Status message will contain number of errors and failures if there were any, else just an OK message. """ ntests, time = 0, 0.0 m = re.search(r'^Ran (\d+) tests in ([\d\.]+)s$', out, re.M) if m: ntests = int(m.group(1)) time = float(m.group(2)) nerrs, nfails = 0, 0 m = re.search(r'^FAILED (.*)$', out, re.M) if m: msgs = m.group(1)[1:-1].split(', ') msgdict = dict([msg.split('=') for msg in msgs]) nerrs = int(msgdict['errors']) if 'errors' in msgdict else 0 nfails = int(msgdict['failures']) if 'failures' in msgdict else 0 return ntests, nerrs, nfails, time def parse_coverage(out): """Find the last line of dashes and parse the percent coverage from the preceding line. Works for both single module reports and for multi-module reports that contain a final total line. """ lines = out.split('\n') for i in reversed(range(len(lines))): if re.match('^-{20,}$', lines[i]): if i: m = re.search('\d+\s+\d+\s+(\d+)%', lines[i - 1]) if m: return int(m.group(1)) return 0 command = ['nosetests', '--with-coverage', '--cover-erase'] + args out, err = Popen(command, stdout=PIPE, stderr=PIPE).communicate() tests = parse_tests(out + err) cover = parse_coverage(out + err) return list(tests) + [cover, out + err]
This is part of a script I call verifycode which I tied to Mercurial's
precommit hook:
[hooks] precommit = verifycode 70 --exclude=__slow \ --cover-package=nn,ns projects
Now whenever I commit code, the script runs a suite of verification tests to make sure I haven't broken anything. It also makes sure that my tests cover a reasonable percent of my code base. I can reuse this script across different repositories and simply change the arguments for the test coverage threshold and the other arguments which are passed on to Nose.
Aside: the
--exclude=__slowargument excludes any test methods with__slowin their name. This is a convention I follow to make it easy to run a subset of my tests quickly. This particular project involves testing a neural network which can be slow.
The complete script is available here. The following is the output from a commit.
david$ hg ci -m "finished first draft of article" verifycode...ok [3.1s, 10/10 tests passed, 79% coverage (>70%)]
Tips for Writing Mercurial Hooks
As I was going through this, I ran into a few problems and learned a few tips along the way. The first problem was getting in-process hooks to work properly on Windows XP. It turns out there is a known bug with the binary installer for Mercurial on Windows which messes up the importer. The workaround is to use the explicit path of the Python source file for an in-process hook:
[hooks] precommit = python:c:/path/to/hook/file/myhooks.py:hookfunc
That was ugly enough to convince me to use external hooks instead. Additionally, I didn't really need the detailed information available through in-process hooks so it seems the general advice you'll find is to stick with external hooks when possible.
The second thing I found useful was the fact that you can install multiple hooks under the same event and Mercurial will call them in alphabetical order. You just need to append something to the name, after a period, to keep the hooks unique. You can use this to add a hook that will always be called last and will fail so you can test your hook without actually committing any changes:
[hooks] precommit = my.real.hook precommit.zstop = false
Mercurial will call the hook with the .zstop suffix after the real
one. Since it simply calls the *nix false command, the commit will fail.
The third trick I learned to help me debug hooks, when I was struggling with in-process hooks, was to write a debug hook that can be called in-process and externally. The following script shows the approach I used.
#! /usr/bin/env python import os import sys from mercurial import ui, hg def debug_hook(ui, repo, **kwargs): ui.status('repo.root: %s\n' % repo.root) fmod, fadd, frem, fdel, funk, fign, fcln = repo.status() files = [('M', f) for f in fmod] + \ [('A', f) for f in fadd] + \ [('R', f) for f in frem] + \ [('D', f) for f in fdel] ui.status('repo.status:\n') for f in files: ui.status(' %s %s\n' % f) changectx = repo['tip'] ui.status('change.desc: %s\n' % changectx.description()) return False if __name__ == '__main__': print 'environment:' keys = sorted([k for k in os.environ.keys() if k.startswith('HG')]) size = max(map(len, keys)) for k in keys: print " %-*s: %s" % (size, k, os.environ[k]) hgui = ui.ui() repo = hg.repository(hgui, '.') debug_hook(hgui, repo) sys.exit(0)
You can use it as a both an in-process hook or an external hook like this:
[hooks] # external pretxncommit.ext = ~/code/hgext/hgdebug.py # in-process pretxncommit.in = python:hgext.hgdebug.debug_hook
You have to have the module in your Python path, of course. Also, you need to
make the script executable so you can call it directly as an external hook (or
you could invoke it as python ~/code/hgext/hgdebug.py).
Mercurial over Subversion
I moved to Mercurial from Subversion and I'm still getting my feet wet at this point, but I really like it so far. It's fairly easy to learn, is similar enough to Subversion in many ways that the commands don't seem too foreign and was very easy to convert my Subversion projects over to try it out. I still use Subversion at work and haven't found it confusing to bounce back and forth between the two.
As I'm not using it on a team, I'm not getting the full benefits of a DVCS, but the nice thing is that most of the work flows scale nicely down to a team of one. If you want to get started, I found these resources very helpful:
- Mercurial: The Definitive Guide: I bought this book (and I suggest you do, too), but it's also available online for free. After reading it, I find it nice to just browse to it online and review certain sections.
- Hg Init: a Mercurial tutorial: Joel Spolsky's company, Fog Creek Software, switched over to Mercurial from Subversion and now offers a hosted Mercurial service, with a code review work flow that integrates nicely with their FogBugz bug tracking system. He wrote this nice, short tutorial on how to use Mercurial and what he found the advantages to be.