t+1: Matt Wilson’s blog

This is not ideal:

----------------------------------------------------------------------
Ran 84 tests in 370.741s
 
OK


My tests take so long for two reasons. First of all, most of them use twill to simulate a browser walking through a version of the web app running on localhost. Second, my test code reads like a novel. Here’s an example, slightly embellished to make a point:

setup: connect to the database and find or create a hospital and an employee named “Nurse Ratched.” Find or create a bunch of open shifts in the emergency department. Find or create another nurse named Lunchlady Doris*.

test: Nurse Ratched wants to see what shifts are available to be picked up. So she logs into the app. Then she navigates to the “open shifts” screen, and then filters down to shifts in the emergency department over the next seven days. Then she wants to sign up for the shift starting at midnight on Saturday night. So, she clicks the “sign up” icon. The system verifies that this shift + her already-scheduled hours won’t push her into overtime, and she has no other flags on her account, so she is automatically scheduled.

Then the system sends her a confirmation message, which according to her preferences, is sent to her email address. Then the system queues an SMS message to be delivered an hour before the shift starts in order to remind her (also according to her preferences).

Finally, the test verifies that the shift is now not listed as available by simulating Lunchlady Doris logging in and checking that same “open shifts” screen.

If everything checks out, print a dot, and move on to the next chapter.

teardown: Unassign Nurse Ratched from the shift she picked up.

I think twill in itself is fine. Marching through a series of pages is problematic. I do this to set up conditions for testing later on. As a side benefit, I verify everything checks out along the way.

On the plus side, I’m confident that the integration of all these components do in fact play nice together. I don’t think it’s safe to abandon end-to-end testing like this, but I would like not to depend it every time I want to make some slight change to a component. It would be nice to run these right before a commit, but only run some super-fast tests after each save.

[*]People that understand this reference should reevaluate their priorities in life. back

I wrote the doctests for the function below and then wrote the code to satisfy them in a total of about 30 seconds. As an extra plus, these doctests immediately clarify behavior in corner cases.

def has_no(s):
    """
    Return False if string s doesn't have the word 'no' inside.
 
    >>> has_no('no problem')
    True
 
    >>> has_no('not really')
    False
 
    >>> has_no('no')
    True
 
    >>> has_no('oh nothing')
    False
    """
 
    if s.lower() == 'no': return True
    if s.lower().startswith('no '): return True
    if s.lower().endswith(' no'): return True
    if ' no ' in s.lower(): return True
    
    return False


Writing tests in any other testing framework would have taken me much longer. Compared to writing these tests with nose, writing this:

assert not has_no('oh nothing')

wouldn’t take me any more time than

>>> has_no('oh nothing')
False

But that’s not all there is to it. With nose, I’d need to open a new test_blah.py file, then import my original blah.py module, then I would have to decide between putting each assert in a separate test function or just writing a single function with all my asserts.

That’s how a 30-second task turns into a 5-minute task.

Anyhow, I’m surprised doctests don’t get a lot more attention. They’re beautiful. Adding tests to an existing code base couldn’t be any simpler. Just load functions into an interpreter and then play around with it (ipython has a %doctest_mode, by the way).

For a lot of simple functions (like the one above) it is easy to just write out the expected results manually rather than record from a session.

It is also possible to store doctests in external text files. The Django developers use this trick frequently.

Finally, I don’t try to solve every testing problem with doctests. I avoid doctests when I need elaborate test fixtures or mock objects. Most of my modules have a mix of functions with doctests and nose tests somewhere else to exercise the weird or composite stuff.

Incidentally, this post is where Tim Peters introduced the doctests module.

I’ve been writing way more tests for my code lately, and they’ve become the backbone of my development style.

I used to write some web code, then play around with it in the browser and look for something to blow up. Automated testing was usually an afterthought or used for confirmation. Or maybe in some cases I would write tests dilligently at the beginning of a project, but start skipping them as deadlines approached. Following TDD has always felt like sticking to high-school abstinence pledges. I wrote tests because I thought I should do it, not because I wanted to do it.

But now I’ve found a way to make writing tests central to getting my work done. It’s because of the computer I’m using now. It is such a crappy box that it takes about 5 minutes for firefox to start up. Then each page load takes about another 30 seconds at least. In the time it takes to click through three or four pages using firefox or epiphany, my twill scripts can run through about a hundred pages.

There’s a scene in Star Wars: a New Hope where Obiwan trains Luke to use the light saber while blindfolded. Well, he’s not blindfolded, really, he’s wearing a helmet with the blast shield down, but the idea is the same. Luke has to use “the force” to feel out where the floating droid is, rather than relying on his vision.

Anyway, writing web pages with this Compaq Presario 1200 feels kind of like that. It’s too frustrating to check my pages with Firefox. The only way I can make sure that anything really works is to write a test for it.

PS: I wrote this posting with lynx.

I wanted to add a list of checkboxes to the “create employee” and “edit employee” pages. Each checkbox adds that employee to a group, and each group has certain privileges. Really standard stuff. I wrote the code in my controller and template in about 20 minutes

Then I played around with the new pages in my browser to check for obvious errors. I created some employees, monkeyed with their groups, then saved them, then opened up the edit screen, verified everything worked right, then monkeyed with them some more, and so forth. That probably took 5 minutes of clicking around lazilly.

In the past, that’s when I would have committed the code to the repository and moved on to something else. I played with it, it didn’t break, so we’re done. Total dev time: 30 minutes.

This time, I wrote a series of twill scripts to go through the different combinatorials of users and groups.

For example, if I have groups A and B, I would really test creating 4 different employees:

• new user with no group membership
• new user in group A
• new user in group B
• new user in groups A and B

After each creation, I verify that the page displays the data correctly and I then hit the database and make sure that everything is set correctly.

For the screen that allows editing employees, the most thorough possible test would take those four new employees and loop until each has been changed to every other possible configuration.

This took about another two hours by the time it was done. The next time I have to write code like this, it will be much faster because I figured out how to write code that yields tests iteratively. Using TDD, total dev time hit about 2.5 hours.

So, in this particular case, is it worth it?

Here’s the reasons why I would say that it was worth it:

I’m still learning how to write good tests. Writing thorough tests requires a different mindset, for me anyway. If I wait until I face some really gnarly complex code to write tests, I’m likely to write some crappy incomplete tests. Each test I write makes me faster at writing tests. Also, when I’m regularly writing tests, I write application code with testing in mind. I think more about design and protecting against what could go wrong, rather than just reaching the finish line any way I can.

And here’s the contrarian view:

Time is scarce and writing tests takes time. In a time-constrained environment, writing needless tests is as silly as blowing off real work to write blogs.

I’m not sure which voice in my head I will listen to on this one.