I wrote the doctests for the function below and then wrote the code to satisfy them in a total of about 30 seconds. As an extra plus, these doctests immediately clarify behavior in corner cases.
def has_no(s):
"""
Return False if string s doesn't have the word 'no' inside.
>>> has_no('no problem')
True
>>> has_no('not really')
False
>>> has_no('no')
True
>>> has_no('oh nothing')
False
"""
if s.lower() == 'no': return True
if s.lower().startswith('no '): return True
if s.lower().endswith(' no'): return True
if ' no ' in s.lower(): return True
return False
Writing tests in any other testing framework would have taken me much longer. Compared to writing these tests with nose, writing this:
assert not has_no('oh nothing')
wouldn’t take me any more time than
>>> has_no('oh nothing')
False
But that’s not all there is to it. With nose, I’d need to open a new test_blah.py file, then import my original blah.py module, then I would have to decide between putting each assert in a separate test function or just writing a single function with all my asserts.
That’s how a 30-second task turns into a 5-minute task.
Anyhow, I’m surprised doctests don’t get a lot more attention. They’re beautiful. Adding tests to an existing code base couldn’t be any simpler. Just load functions into an interpreter and then play around with it (ipython has a %doctest_mode, by the way).
For a lot of simple functions (like the one above) it is easy to just write out the expected results manually rather than record from a session.
It is also possible to store doctests in external text files. The Django developers use this trick frequently.
Finally, I don’t try to solve every testing problem with doctests. I avoid doctests when I need elaborate test fixtures or mock objects. Most of my modules have a mix of functions with doctests and nose tests somewhere else to exercise the weird or composite stuff.
Incidentally, this post is where Tim Peters introduced the doctests module.
But then you end up with your tests for a module in several places.
Although typing this:
>>> has_no('oh nothing')
False
May not take much longer than:
assert not has_no('oh nothing')
but typing in docstrings means I lose the support of my IDE and my real code gets bloated with tests – not are useful documentation.
Personally I think doctest is *great* for testing documentation, but unittest is more suitable for unit testing.
moved this comment to the reply it was meant to be.
Hi Michael, Thanks for the comment!
I'm not sure I understand your point about tests in several places. The tests are in two places at most; the docstring, and then whatever external tests I write. It isn't difficult to trace them all down quickly because everything is in predictable locations.
I think that the *right kind of tests* are useful as documentation. It's sometimes easier to show rather than tell. Sure, a 200-line docstring is goofy. But I don't find myself writing those kinds of doctests.
I agree with Michael, and shall invoke The Zen:
“There should be one– and preferably only one –obvious way to do it”
If you have useful tests in doctests AND in test modules, you've created two ways to write tests, have violated the Zen, and are condemned to burn in hell surrounded by Javangelists for all eternity.
In practical terms, though, you're probably OK. I think there just has to be a very clear line drawn about what goes where.
Hi Olivstor, I don't think you can invoke the Zen since Tim Peters is the guy that wrote both the Zen AND came up with the idea for doctests 🙂
But in terms of practicality, I think it would be straightforward to come up with some rules for when to use which test framework; e.g., if you need a big long list of fixtures, please use something besides a doctest.
So, the “obvious” way to do it would be to look at the testing needs and then let dictate which tool to use.
The “one obvious way to do it” doesn't mean (by my interpretation, anyway), to use the same tool for everything.
Python has for-loops, iterators, generator expressions, and map. They're all appropriate in different contexts.
Anyhow, thanks for the comment!
Matt –
Couldn't you just use nose's doctest plugin?
<my snip>
% nosetests –with-doctest has.py -v -s
Doctest: has.has_no … ok
———————————————————————-
Ran 1 test in 0.007s
OK
</my snip>
I love doctests. I also love the nose doctest plugin.
Hi Terry, Yeah, I do use the doctests plugin for nose. So when I run my doctests, I do $ nosetests –with-doctest blah.py
Well, actually, with-doctest=1 is in my $HOME/.noserc, so I don't need the –with-doctest option.
The fact that other important frameworks such as nosetests and unit tests have hooks into doctests is great. It shows that Python unit testers recognise its importance.
The big battle is getting people to write any meaningful tests. Doctests can help break down that barrier.
– Paddy.
P.S. More doctest info here: http://en.wikipedia.org/wiki/Doctest
> but typing in docstrings means I lose the support of my IDE
Get yourself a real IDE. Emacs handles doctests without any issue with doctest-mode 0.5 in mmm-mode.
> and my real code gets bloated with tests – not are useful documentation.
Your tests clearly aren't good enough.
Also Matt, all you boolean fugling could be replaced by `return bool(re.search(r'bnob', s, re.I))`
Masklinn — thanks for the regexp help! I hoped there was a way to do it. Code updated.
Yeah, doctests are so simple to write that *anybody* can start writing tests, and I believe that any automated testing, even if it only covers 10% of the code, is still a huge help.
“Get yourself a real IDE. Emacs handles doctests without any issue”
Yes, because it's autocompletion is brain dead in the first place! At least compared to a real IDE like Wing.
Pingback: Interstellar Medium: the Free Software carnival » Free Software Carnival: 12 – 18 July
Other important frameworks such as nosetests and unit tests have hooks into doctests is great. It shows that Python unit testers recognise its importance. The big battle is getting people to write any meaningful tests.
I found this blog very handy. I also wonder why docktests do not grab much attention. It’s nice that you have focused on this matter. But I have one question. Can you please explain me how I can store doctests in external text files?
Hey I'v been reading your blog from quite some time now and I just wanted to say keep up the good work.