Some notes on nosetests and coverage

I use Nose to run my automated tests. It can report on code coverage like this:

$ nosetests --quiet --with-coverage --cover-package pitz
Name                           Stmts   Exec  Cover   Missing
------------------------------------------------------------
pitz                              29     29   100%   
pitz.bag                         108    107    99%   150
pitz.cmdline                      50     12    24%   23-54, 62-79, 92-93, 96, 109-114, 119-130
pitz.entity                      105    105   100%   
pitz.exceptions                    1      1   100%   
pitz.junkyard                      0      0   100%   
pitz.junkyard.ditzloader          22     15    68%   31-37, 45-47
pitz.pitzfiles                     0      0   100%   
pitz.project                      52     52   100%   
pitz.projecttypes                  0      0   100%   
pitz.projecttypes.agilepitz       55     54    98%   66
pitz.projecttypes.simplepitz      66     61    92%   84-90
------------------------------------------------------------
TOTAL                            488    436    89%   
----------------------------------------------------------------------
Ran 69 tests in 6.350s

OK (SKIP=6)

Most of the uncovered code is in the cmdline module, which does a lot of work with the filesystem and starting up IPython, and I’m having trouble writing tests there. You could help, you know .

I’m keenly aware that running with coverage makes tests much slower. Normally, my pitz tests run in about half a second:

$ nosetests --quiet
----------------------------------------------------------------------
Ran 69 tests in 0.504s

OK (SKIP=6)

Fortunately, I don”t have to rerun all your tests to see what lines are uncovered. I can query the .coverage file created by nose afterward to get details. It is really easy to get coverage for just one module:

$ coverage -r -m pitz/cmdline.py
Name           Stmts   Exec  Cover   Missing
--------------------------------------------
pitz/cmdline      50     12    24%   23-54, 62-79, 92-93, 96, 109-114, 119-130

And getting coverage on multiple modules is straightforward, but kind of tedious:

$ coverage -r -m pitz/cmdline.py pitz/__init__.py pitz/entity.py
Name            Stmts   Exec  Cover   Missing
---------------------------------------------
pitz/__init__      29     29   100%   
pitz/cmdline       50     12    24%   23-54, 62-79, 92-93, 96, 109-114, 119-130
pitz/entity       105    105   100%   
---------------------------------------------
TOTAL             184    146    79%   

I don’t know of an elegant solution to do what nosetests does, where it shows me all coverage for a package. Running coverage -r without any module lists dumps out everything, which is never what I want:

$ coverage -r
Name                                                                                                               Stmts   Exec  Cover
--------------------------------------------------------------------------------------------------------------------------------------
/home/matt/checkouts/myfiles/bin/__init__                                                                              0      0   100%
/home/matt/virtualenvs/pitz/lib/python2.6/site-packages/Jinja2-2.1.1-py2.6-linux-i686.egg/jinja2/__init__             12     11    91%
/home/matt/virtualenvs/pitz/lib/python2.6/site-packages/Jinja2-2.1.1-py2.6-linux-i686.egg/jinja2/bccache             111     36    32%
Traceback (most recent call last):
KeyboardInterrupt

A while back on twitter I posted that I wanted my editor to read my .coverage file and highlight the lines that aren’t covered. I don’t know anything about how to control syntax highlighting, but I run this little deal from within vim to get the same facts:

:! coverage -r -m %

My edit buffer disappears and I see this instead:

Name      Stmts   Exec  Cover   Missing
---------------------------------------
cmdline      50     12    24%   23-54, 62-79, 92-93, 96, 109-114, 119-130

Press ENTER or type command to continue

In vim, :! is how you run a command in a shell. Incidentally, it is also possible to pull the results from the command into vim and write data from vim to the command, but that’s not what I want right now.

Vim replaces the % symbol with the file I’m currently editing. Of course, this command only works when I start vim in the same directory as the .coverage file. If I ain’t in that directory, then I have to specify how to find the .coverage file by setting the environmental variable COVERAGE_FILE like this:

:! COVERAGE_FILE=../.coverage coverage -r -m %

Setting it that way means it doesn’t last beyond that one shell. If I want vim to remember the setting, I could set COVERAGE_FILE when I start vim like this:

$ COVERAGE_FILE=../.coverage vi cmdline.py

Or I could export it like this:

$ export COVERAGE_FILE=../.coverage
$ vi cmdline.py

In summary, coverage is a neat tool, but it is silly to think that 100% test coverage guarantees anything. Coverage won’t warn you when your calculator returns 3 for 1 + 1.