I use Nose to run my automated tests. It can report on code coverage like this:

$ nosetests --quiet --with-coverage --cover-package pitz
Name Stmts Exec Cover Missing
------------------------------------------------------------
pitz 29 29 100%
pitz.bag 108 107 99% 150
pitz.cmdline 50 12 24% 23-54, 62-79, 92-93, 96, 109-114, 119-130
pitz.entity 105 105 100%
pitz.exceptions 1 1 100%
pitz.junkyard 0 0 100%
pitz.junkyard.ditzloader 22 15 68% 31-37, 45-47
pitz.pitzfiles 0 0 100%
pitz.project 52 52 100%
pitz.projecttypes 0 0 100%
pitz.projecttypes.agilepitz 55 54 98% 66
pitz.projecttypes.simplepitz 66 61 92% 84-90
------------------------------------------------------------
TOTAL 488 436 89%
----------------------------------------------------------------------
Ran 69 tests in 6.350s

OK (SKIP=6)

Most of the uncovered code is in the cmdline module, which does a lot of work with the filesystem and starting up IPython, and I’m having trouble writing tests there. You could help, you know 🙂.

I’m keenly aware that running with coverage makes tests much slower. Normally, my pitz tests run in about half a second:


$ nosetests --quiet
----------------------------------------------------------------------
Ran 69 tests in 0.504s

OK (SKIP=6)


Fortunately, I don”t have to rerun all your tests to see what lines are uncovered. I can query the .coverage file created by nose afterward to get details. It is really easy to get coverage for just one module:

$ coverage -r -m pitz/cmdline.py
Name Stmts Exec Cover Missing
--------------------------------------------
pitz/cmdline 50 12 24% 23-54, 62-79, 92-93, 96, 109-114, 119-130

And getting coverage on multiple modules is straightforward, but kind of tedious:

$ coverage -r -m pitz/cmdline.py pitz/__init__.py pitz/entity.py
Name Stmts Exec Cover Missing
---------------------------------------------
pitz/__init__ 29 29 100%
pitz/cmdline 50 12 24% 23-54, 62-79, 92-93, 96, 109-114, 119-130
pitz/entity 105 105 100%
---------------------------------------------
TOTAL 184 146 79%

I don’t know of an elegant solution to do what nosetests does, where it shows me all coverage for a package. Running coverage -r without any module lists dumps out everything, which is never what I want:

$ coverage -r
Name Stmts Exec Cover
--------------------------------------------------------------------------------------------------------------------------------------
/home/matt/checkouts/myfiles/bin/__init__ 0 0 100%
/home/matt/virtualenvs/pitz/lib/python2.6/site-packages/Jinja2-2.1.1-py2.6-linux-i686.egg/jinja2/__init__ 12 11 91%
/home/matt/virtualenvs/pitz/lib/python2.6/site-packages/Jinja2-2.1.1-py2.6-linux-i686.egg/jinja2/bccache 111 36 32%
Traceback (most recent call last):
KeyboardInterrupt


A while back on twitter I posted that I wanted my editor to read my .coverage file and highlight the lines that aren’t covered. I don’t know anything about how to control syntax highlighting, but I run this little deal from within vim to get the same facts:

:! coverage -r -m %

My edit buffer disappears and I see this instead:

Name Stmts Exec Cover Missing
---------------------------------------
cmdline 50 12 24% 23-54, 62-79, 92-93, 96, 109-114, 119-130

Press ENTER or type command to continue

In vim, :! is how you run a command in a shell. Incidentally, it is also possible to pull the results from the command into vim and write data from vim to the command, but that’s not what I want right now.

Vim replaces the % symbol with the file I’m currently editing. Of course, this command only works when I start vim in the same directory as the .coverage file. If I ain’t in that directory, then I have to specify how to find the .coverage file by setting the environmental variable COVERAGE_FILE like this:

:! COVERAGE_FILE=../.coverage coverage -r -m %

Setting it that way means it doesn’t last beyond that one shell. If I want vim to remember the setting, I could set COVERAGE_FILE when I start vim like this:

$ COVERAGE_FILE=../.coverage vi cmdline.py

Or I could export it like this:

$ export COVERAGE_FILE=../.coverage
$ vi cmdline.py


In summary, coverage is a neat tool, but it is silly to think that 100% test coverage guarantees anything. Coverage won’t warn you when your calculator returns 3 for 1 + 1.