We worked in pairs for 45-minute blocks of time, building Conway’s Game of Life. In each session, we started from the beginning. We didn’t build on previous code.

After each session, teams talked about what stuff they discovered. Then we reorganized into new pairs and started working from the beginning again.

First pairing

At first I paired with a C# developer. While other teams went right into writing test programs, we spent a fair amount of time talking and sketching out ideas about how to model the game.

Then we made a grid that knew how to generate the next state just by copying itself into a new grid.

We planned to add in code to support each of the four rules of the game of life one by one.

So, we wrote a test that created a grid with just a single living cell. Then we told that grid to make a grid for the next generation, and our test verified that the single cell survived the copy.

Then we started working on the first rule: if a cell is alive and has less than two neighbors, it dies.

We burned through the reminder of time sketching out ideas on my clipboard. By the end of our time, it was clear that we only needed to pay attention to living cells and neighbors of living cells.

It was also clear that the time spent writing the first test and then writing the code to satisfy the first test did nothing to move us toward a real solution.

Second pairing

Next I worked with a Java guy. This time, we did something a little closer to TDD.

We decided to start from the point of view of an end-user firing up the game. The user would want to make a grid, add a few living cells, and then tell the grid to generate its next state.

The first code we wrote was a test and we started writing tests almost immediately. In the test, we instantiated a Grid class. That test blew up with an error, since no Grid class existed. So then we wrote a grid class. It had no methods or attributes, but that was all we needed to write in order to satisfy our test.

Next we wrote a new test that instantiated a grid and then added a single cell by calling an (undefined) addCell(…) method.

After seeing that test crash with an error, we wrote an addCell(…) method that was a no-op, because that’s all our tests required us to do.

Then we wrote a test for rule one, where a living cell with less than two neighbors dies.

In this test, we made a grid, added a single cell, and then told the grid to create a new grid that represented itself in the future. Then we used an assert to verify the new grid had no cells.

First we added a “cells” instance-level ListArray attribute to our grid class. Then we wrote a generate_next_grid method on our grid class, and all that did was instantiate and return a new grid with an empty cells.

And bingo, we had enough code again to pass our test.

At this point, I became convinced that the kind of testing we were doing was harming our productivity. We had fallen into some kind of xeno’s paradox* where we would never actually get anything done, because we could always think of some new test that applied to a trivial subset of the project, and then work on writing code to satisfy that test.

*A dude shoots an arrow at a target. Before the arrow gets to the target, it has to cross half the distance to the target. Then it can cross the other half of the distance. But now it has to cross half of the remaining distance, so it does that.

Then it has to cross half of that remaining distance, and then half of the remaining distance after that, and so on for infinity.

Since the arrow has an infinite number of partial distances to cross, the arrow will never reach the target.

Read more about it here.

I was itching to solve a real problem, rather than just watch lights blink from red to green, so I cajoled my partner into writing a test for the blinker pattern, which is when a 3×1 rectangle of cells converts to a 1×3 rectangle of cells on the second turn, and then on the third turn, converts back to a 3×1 rectangle.

So, we started talking about how to implement the rules. We decided we needed to figure out for every cell that was near a living cell, how many living neighbors it had. So we wrote a method to generate the coordinates for the eight neighboring cell to a particular cell, and wrote a test for that.

Then we were going to make a neighbor_count hashmap attribute on our grid and we were going to use that to link cells to how many neighbors each had, and then the time ran out.

At this point, I had a good feeling about the algorithms involved in solving this project.

Third pairing

Right before the third pairing, a much older fellow raised his hand and asked if anyone wanted to work together on a solution that only tracked the living cells. That’s the approach I was using, so I paired with this guy. He said he didn’t care what language we used, but he was a lousy typist. So I said I could do the typing and we could use python.

At first we excitedly talked over each other and drew pictures until we confirmed we had the same solution in mind. It turned out my partner was an old common lisp hacker, and once he realized that I understood what he meant when he talked about stuff like &rest and cadr, we got along really well.

The first bit of code I wrote was a cell class with an x and a y attribute, and a method to generate a list of eight nearby cells.

I wrote a quick test in the same file as my code for that to make sure the execution was correct.

Of course, that test failed. I had lots of syntax errors because I was writing so fast and defending against good-natured jabs about how vim was so vastly inferior to emacs.

After a few cycles of fixing typos and rerunning the tests, we got the syntax errors out of the code. Then we attacked the project of generating the next state of the grid.

I had already seen how easy it was to write a test for the blinker pattern, so I wrote a quick test for that. The test made a grid with three cells in a vertical rectangle, then told the grid to generate the next state, then it tested that in the next state, the three cells were in a horizontal rectangle.

We wrote a grid class with a dictionary attribute of living cells in the current generation and a dictionary attribute of cells that would live in the next generation.

Then we wrote the function in our grid class to generate another grid. The function did an outer loop on all the living cells, then for each living cell, it looped through all the neighbors of that living cell, and then did an inner loop of all neighbors of that neighbor, and then counted up how many were alive.

Then we implemented the four rules of the game with two if-clauses.

We ran the test again, and of course, the code was again littered with indentation errors and mismatched parentheses because of how fast I had been typing and moving stuff around.

Then the timer went off. I stayed put while everybody was gathering in the middle to talk about how everything went. Within about 30 seconds I got the last few silly typing glitches fixed, and BAM. The test passed.

I walked over to my partner and told him we solved it.

Then we listened to other teams talk. The conversation centered around naming conventions that people were coming up with, like “sustainable neighborhood” and “four horsemen”.

I raised my hand and asked if anybody had written tests for any of the patterns in game of life beyond the blinker pattern, like tests for the glider pattern, or any of the other patterns shown on the wikipedia page.

Nobody replied to that. Instead the conversation shifted to a discussion of whether or not booleans hidden behind getters and setters were a sign of a bad design.

I think it was while somebody was talking about writing tests for their (nonexistent) display code that my partner leaned over and said “This is a drunkard’s search.” Of course, I didn’t know what meant, so he explained it. Here’s the wikipedia entry:

Conducting a drunkard’s search is to look in the place that’s easiest, rather than in the place most likely to yield results. Taken from an old joke about a drunkard who loses his car keys while unlocking his car and is found looking under a streetlamp down the road because the light is better, it has been an object of consideration in the social sciences since at least 1964.

Fourth pairing

I asked if anybody wanted to see a python solution, and four of us worked together, with me doing the typing.

First I described how we would need to find the eight neighboring cells of a particular cell. So I wrote a test for that, and then wrote the code to satisfy that test.

Then once that was finished, I described what the blinker looked like, and I wrote a test for that.

Then I reused the approach that we came up with in the third pairing, and then got it working fairly quickly.

At one point I showed the code we had written to one of the people running the event, and he pointed out two things (this is roughly paraphrased, and I hope I am getting these ideas correct):

1. I had four if-clauses in a single function, and each if-clause tested two independent variables. So that means there were dozens of paths through the function that would have to be tested.
2. My blinker test could be satisfied by code that just hardcoded the expected results, rather than built an actual game-of-life system. My test didn’t force a complete solution.

Both points illustrate the biggest difference in the TDD mindset versus whatever it is called that I do. (design by intuition?)

The fact that I didn’t have a test for every path that I created didn’t really concern me, because I was focused on getting the blinker pattern to work. If that blinker pattern test failed, then I might go in to my code and make sure that what I wrote was really what I meant, or maybe I’d even throw it all out and start from scratch. But I wouldn’t worry about covering all corners of this version until I was confident I was on the right track.

As for the second point, I use tests to catch errors, not to tell me what to write. I wouldn’t write code that returns a hard-coded list of values just to pass a test, unless I actually thought I could solve the real problem by doing that.

Anyhow, we got everything working pretty quickly this second time. You can see the output from my fourth pairing here.

After you download that code, you can run the tests like this:

    $ python life.py
    ..
    --------------------------------------------------------------------
    Ran 2 tests in 0.001s

    OK

Fifth pairing

I worked with two guys and we used ruby and rspec. I described the algorithm that I had used in the last two times, and we decided to work on building that.

We went with the approach of not writing any more code than was required to satisfy the tests.

So, first, we wrote some tests to instantiate a cell class, then we wrote some tests to call a neighbors method on the cell class and verify that it return a list of eight things, and then we verified that the first and last and elements in the neighbors list were at the position that we expected them to be at.

Then we started work on the grid class. We wanted to make sure that two different objects with the same values for their x and y attributes would be considered equal. But none of us knew how to redefine the == operator in ruby, and while we were researching this, the time ran out.

Again, I felt like we had fallen into the trap of slicing off trivial subsets of the real problem, and solving them, rather than attacking the real problem.

Commentary

Writing tests to check for errors (which is what I do) is not test-driven development. I daydream about solutions until I find one that I like, and then I build it, and I see if it works by writing tests to cover all the use cases I can think of.

My design comes from idle thoughts and doodles on my notepad, not from the testing.

Final note: I’ve tried my best to keep any generalizations and extrapolations out of this post. I don’t mean for this to be seen as an attack on anyone’s style of work. I’m very, very grateful for everything I’ve learned about how to write tests for software from the TDD community.

gunicorn crazyproject.webapp:make_application

Other frameworks often have a development mode that forks off a helper process that watches the source code folder. When some file changes, the helper tells the server to reload.

I’m not using any frameworks this time, so I don’t have that feature. I never really liked the fact that every time I saved a file, I would restart the server during development. Instead, in this case, I want to easily restart gunicorn, but only when I want to, and I don’t want to leave my editor.

It turned out to be pretty easy to do.

First I read in the excellent gunicorn docs how to tell gunicorn to reload my app:

How do I reload my application in Gunicorn?

You can gracefully reload by sending HUP signal to gunicorn:

$ kill -HUP masterpid

Next I added a –pid /tmp/gunicorn.pid option to the command that I use to start gunicorn, so that gunicorn would write the parent’s process ID into /tmp/gunicorn.pid.

Now any time I want gunicorn to reload, I can do this little command in a terminal window:

$ kill -HUP `cat /tmp/gunicorn.pid`

Those backticks around cat /tmp/gunicorn.pid tell the shell to do that part of the command first, and then feed the result into the rest of the command.

You can always use ! in vim to run some command-line program. If I had to explain the difference between vim and emacs, the one difference that is most interesting to me is that vim makes it super-easy to send buffer contents to other programs or read buffer contents from other programs, while the people behind emacs seem to think that any external dependency should ultimately be moved into emacs itself. I get the feeling that vim wants to be my text editor, but emacs wants to be my OS.

Anyhow, while I’m in vim, I run that command to restart like this:

: !kill -HUP `cat /tmp/gunicorn.pid`

After using that code for a while, and feeling confident it worked right, I mapped F12 in vim to do that action for me by adding this little thing to the end of my ~/.vimrc:

:map <F12> :!kill -HUP `cat /tmp/gunicorn.pid`<CR>

Now that means I hit F12 when I want to reload gunicorn.

If you’re smart, live near me, and have at least eight hours a week to work, let me know.

This will be for some combination of money up front and equity.

I expect to launch in anywhere from six weeks to three months after the start.

I have no patience for ideologues, jackanapes, or freeloaders.

The –quick option

Normally, running pitz-add-task will prompt for a title and then open $EDITOR so I can write a description. After that, pitz will ask for choices for milestone, owner, status, estimate, and tags.

Sometimes I want to make a quick task without getting prompted for all this stuff.

I already had a --no-description option that would tell pitz-add-task to not open $EDITOR for a description. And I already had a --use-defaults option to just choose the default values for milestone, owner, status, estimate, and tags.

But when I just want to make a quick to-do task as a placeholder, writing out all this stuff:

$ pitz-add-task --no-description --use-defaults -t "Fix fibityfoo"

is kind of a drag. So I made a --quick option (also available as -q) that does the same thing as --no-description --use-defaults.

The -1 alias for –one-line-view

This is the typical way that a to-do list looks:

$ pitz-my-todo -n 3
==============================
slice from To-do list for matt
==============================

(3 task entities, ordered by ['milestone', 'status', 'pscore'])

Write new CLI scripts (witz!) to talk to pitz-webapp           9f1c76
matt | paused | difficult | 1.0 | 1
webapp, CLI
Starting up and loading all the pitz data at the beginning of ever...

Experiment with different task summarized views                0f6fee
matt | unstarted | straightforward | 1.0 | 1
CLI
Right now, the summarized view of a task looks a little like this:...

Add more supported URLs to pitz-webapp                         295b5f
matt | unstarted | straightforward | 1.0 | 0
webapp
I want to allow these actions through the webapp:  *   Insert a ne...

Incidentally, notice the -n 3 option limits the output to the first three tasks.

Tasks also have a one-line view:

$ pitz-my-todo -n 3 --one-line-view
==============================
slice from To-do list for matt
==============================

(3 task entities, ordered by ['milestone', 'status', 'pscore'])

Write new CLI scripts (witz!) to talk to pitz-webapp           9f1c76
Experiment with different task summarized views                0f6fee
Add more supported URLs to pitz-webapp                         295b5f

Typing out --one-line-view is tedious, so now, -1 is an alias that works as well:

$ pitz-my-todo -n 3 -1
==============================
slice from To-do list for matt
==============================

(3 task entities, ordered by ['milestone', 'status', 'pscore'])

Write new CLI scripts (witz!) to talk to pitz-webapp           9f1c76
Experiment with different task summarized views                0f6fee
Add more supported URLs to pitz-webapp                         295b5f

The text and programs from my talk on building your own are available here on github. You can probably absorb the information most quickly by reading the talk.rst file.

Please let me know if you find spelling errors or goofy sentences that are hard to understand that I should rewrite.

This is the description for the talk:

My talk is based on a project that seemed very simple at first. I wanted an object like the regular python dictionary, but with a few small tweaks:

• values for some keys should be restricted to elements of a set
• values for some keys should be restricted to instances of a type

For example, pretend I want a dictionary called favorites, and I want the value for the “color” key to be any instance of my Color class. Meanwhile, for the “movie” key, I want to make sure that the value belongs to my set of movies.

In the talk, I’ll walk through how I used tests to validate my different implementations until I came up with a winner.

Unlike my talk last year on metaclass tomfoolery, and the year before that on fun with decorators (and decorator factories) I’m hoping to make this talk straightforward and friendly to beginning programmers.

You’ll see:

• how I use tests to solve a real-world problem
• a few little gotchas with the super keyword
• a little about how python works under the hood

Building your own kind of dictionary

Python level: novice

Description

My talk is based on a project that seemed very simple at first. I wanted an object like the regular python dictionary, but with a few small tweaks:

• values for some keys should be restricted to elements of a set
• values for some keys should be restricted to instances of a type

For example, pretend I want a dictionary called favorites, and I want the value for the “color” key to be any instance of my Color class. Meanwhile, for the “movie” key, I want to make sure that the value belongs to my set of movies.

In the talk, I’ll walk through how I used tests to validate my different implementations until I came up with a winner.

Unlike my talk last year on metaclass tomfoolery, and the year before that on fun with decorators (and decorator factories) I’m hoping to make this talk straightforward and friendly to beginning programmers.

You’ll see:

• how I use tests to solve a real-world problem
• gotchas with the super keyword
• a little about how python works under the hood.

Extended description

I’m not done with the slides, but all my code examples are finished. You can read it all online at my github repository here.

Outline

1. What kind of object I want
   1. Tests define the expected behavior
   2. How to run those tests
2. First implementation (subclass dict)
   1. How the implementation is defined
   2. Examine test results
   3. Examine the C code behind the dict class to see why my subclassed __setitem__ method won’t get called from the parent class
3. Composition-based implementation
   1. Explain composition vs inheritance
   2. Examine test results
   3. Point out irritating need to manually redefine every related dictionary method on the container class
   4. Show how to use __getattr__ to avoid all that boring wrapper code
   5. Show how __getattr__ doesn’t play nice with inspection tools
4. UserDict.UserDict
   1. Explain implementation
   2. Examine test results
   3. Add a new test that uses this class as a parent for a subclass
   4. Explain how UserDict.UserDict is not a new-style class, so the super keyword behaves differently
5. UserDict.DictMixin
   1. Explain implementation
   2. Examine test results
6. PEP 3119 and why it is nice
   1. duck-typing, why it is awesome, why it isn’t perfect
   2. abstract base classes
   3. As of python 2.6, don’t use UserDict.DictMixin; use collections.MutableMapping

pretty colors

It took me a while to figure out how to include the jinja2 template files with the package, but you can now download pitz from PyPi here.

1. What’s the coolest Python application, framework or library you have discovered in 2009 ?

I love the restructured text tools. I use rst2pdf at least once a week.

2. What new programming technique did you learn in 2009?

This was a year of giving up on techniques, paradigms, development methods, frameworks, and fanciness in general and going back to basics. I’m like Rocky Balboa in Rocky IV — jogging in the snow and chopping wood with an axe.

3. What’s the name of the open source project you contributed the most in 2009? What did you do?

I’ve been working on pitz for about 15 months now. I’ve learned about a lot of stuff that I didn’t expect to learn when I decided to build a to-do manager. Because of that project, I’ve learned about how templating systems like jinja2 work internally, how to use cProfile, how to write tests for all sorts of weird situations, how to write a simple querying system, etc.

4. What was the Python blog or website you read the most in 2009?

I just went through my bookmarks tagged with python and didn’t see an obvious pattern. These days, the hundred or so people I follow on twitter keep me supplied with too much to read.

5. What are the three top things you want to learn in 2010 ?

psycopg2 internals, datetime module internals (it is written in C and there is a patch I want to submit), and maybe stuff like ast.

]]> http://blog.tplus1.com/index.php/2010/01/04/new-years-python-meme/feed/ 1 http://blog.tplus1.com/index.php/2009/12/24/add-an-option-for-your-script-to-drop-into-the-debugger/ http://blog.tplus1.com/index.php/2009/12/24/add-an-option-for-your-script-to-drop-into-the-debugger/#comments Thu, 24 Dec 2009 19:49:04 +0000 matt http://blog.tplus1.com/?p=523 I like using the python debugger to, umm, debug stuff. Just recently I wanted to use pdb to poke around in a script that would intermittently crash. I wanted to add an option to my script that would drop me into pdb when the script dies.

This turned out to be trivial to write. The script scratch.py doesn’t do much, but if you add the --drop-into-debugger option, you’ll get dumped into a pdb session, where you can study the problem more closely, like this:

$ python scratch.py --drop-into-debugger
> /home/matt/scratch.py(15)main()
-> 1/0
(Pdb) 

When I don’t add the --drop-into-debugger option, then of course the uncaught exception just crashes the script:

$ python scratch.py
Traceback (most recent call last):
  File "scratch.py", line 23, in <module>
    main()
  File "scratch.py", line 15, in main
    1/0
ZeroDivisionError: integer division or modulo by zero

The code involved was really easy to write. I wrote a decorator called into_decorator with the decorator module, but that’s not strictly necessary. Here’s the code:

# vim: set expandtab ts=4 sw=4 filetype=python:

import pdb, sys
from decorator import decorator

@decorator
def into_debugger(f, *args, **kwargs):
    try:
        return f(*args, **kwargs)
    except:
        pdb.post_mortem()

def main():
    x = 99
    1/0 
    
if __name__ == '__main__':
    if len(sys.argv) > 1 and sys.argv[1] == '--drop-into-debugger':
        into_debugger(main)()
    else:
        main()

That pdb.post_mortem function does the interesting work of inspecting the most recently caught exception. The pdb documentation is pretty dang sparse, but in this case, I got what I needed.

Anyhow, I hope this helps somebody out. I’ll probably be adding a –drop-into-debugger option on every script I write from now on.

]]> http://blog.tplus1.com/index.php/2009/12/24/add-an-option-for-your-script-to-drop-into-the-debugger/feed/ 5 http://blog.tplus1.com/index.php/2009/11/17/pitz-has-a-cli/ http://blog.tplus1.com/index.php/2009/11/17/pitz-has-a-cli/#comments Wed, 18 Nov 2009 02:32:08 +0000 matt http://blog.tplus1.com/?p=514 I’ve exposed lots and lots of pitz functionality as command-line scripts. Here’s the list so far:

$ pitz-help
    pitz-abandon-task          Abandon a task
    pitz-add-task              Walks through the setup of a new Task.
    pitz-components            All components in the project
    pitz-estimates             All estimates in the project
    pitz-everything            No description
    pitz-finish-task           Finish a task
    pitz-milestones            No description
    pitz-my-tasks              List my tasks
    pitz-people                All people in the project
    pitz-prioritize-above      Put one task in front of another task
    pitz-prioritize-below      Put one task behind another task
    pitz-recent-activity       10 recent activities
    pitz-setup                 No description
    pitz-shell                 Start an ipython session after loading in a ...
    pitz-show                  Show detailed view of one entity
    pitz-start-task            Begin a task
    pitz-statuses              All statuses in the project
    pitz-tasks                 All tasks in the project
    pitz-todo                  List every unstarted and started task in the...
    pitz-unassign-task         Take this task off somebody's list of stuff ...

]]> http://blog.tplus1.com/index.php/2009/11/17/pitz-has-a-cli/feed/ 7