I just submitted my talk for PyOhio 2010

I’d be happy to get any crititicism ahead of the presentation. I’m hosting the code examples and the talk material here.

Building your own kind of dictionary

Python level: novice

Description

My talk is based on a project that seemed very simple at first. I wanted an object like the regular python dictionary, but with a few small tweaks:

  • values for some keys should be restricted to elements of a set
  • values for some keys should be restricted to instances of a type

For example, pretend I want a dictionary called favorites, and I want the value for the “color” key to be any instance of my Color class. Meanwhile, for the “movie” key, I want to make sure that the value belongs to my set of movies.

In the talk, I’ll walk through how I used tests to validate my different implementations until I came up with a winner.

Unlike my talk last year on metaclass tomfoolery, and the year before that on fun with decorators (and decorator factories) I’m hoping to make this talk straightforward and friendly to beginning programmers.

You’ll see:

  • how I use tests to solve a real-world problem
  • gotchas with the super keyword
  • a little about how python works under the hood.

Extended description

I’m not done with the slides, but all my code examples are finished. You can read it all online at my github repository here.

Outline

  1. What kind of object I want
    1. Tests define the expected behavior
    2. How to run those tests
  2. First implementation (subclass dict)
    1. How the implementation is defined
    2. Examine test results
    3. Examine the C code behind the dict class to see why my subclassed __setitem__ method won’t get called from the parent class
  3. Composition-based implementation
    1. Explain composition vs inheritance
    2. Examine test results
    3. Point out irritating need to manually redefine every related dictionary method on the container class
    4. Show how to use __getattr__ to avoid all that boring wrapper code
    5. Show how __getattr__ doesn’t play nice with inspection tools
  4. UserDict.UserDict
    1. Explain implementation
    2. Examine test results
    3. Add a new test that uses this class as a parent for a subclass
    4. Explain how UserDict.UserDict is not a new-style class, so the super keyword behaves differently
  5. UserDict.DictMixin
    1. Explain implementation
    2. Examine test results
  6. PEP 3119 and why it is nice
    1. duck-typing, why it is awesome, why it isn’t perfect
    2. abstract base classes
    3. As of python 2.6, don’t use UserDict.DictMixin; use collections.MutableMapping

Moving on

Last week marked the end of my employment with OnShift Inc. What started off as possibly the worst idea for a business conceivable (I’ll justify this claim later) is now an enterprise.

It has been a really good experience. I got to build something all myself. I had some really patient customers that politely filed bug report after bug report, and met with me over and over again to discuss their frustrations.

And then the business crossed one of those inflection points.

We started getting sales. Customers became happy to be references. More than once, people offered to pay for the product out of their own salary after they sat through a demo.

Some fantastic developers and designers joined the team.

In March I sat in the back of the room during a training. At the end, the trainer pointed me out. Usually I take that moment to do my little talk about how this is a new product, and these users are going to run into bugs or things that just seem goofy. I tell them I really want to hear about them, so that I can fix them as soon as possible.

But before I started my speech, everyone started clapping.

So that’s why I have to / get to / want to / need to leave. For about four years now, I’ve been banging rocks together trying to start a fire. And now I got one. So now I’m going to take a break, and perhaps more importantly to OnShift, get out of the way.

I’ll be happy if the business makes it big. I’ve been fully vested in my shares for a while now and I keep those after I leave. But this was never about getting rich. This was about proving something. I aimed to prove that I could design a product and then start a business around it.

I’ve been focused on getting that fire lit for the last four years. Now it is lit.

As for that worst possible biz plan, I’m not going to spell it out. Instead I’ll cram all the bullet points from my partner’s slides into one run-on word salad:

Orthodox Judaism SMS location-based time of Sunset on Friday mobile microblogging opt-in advertising

Farm report spring 2010

Help me prep my new 8 x 16 garden bed

I’ve already got one 8×16 bed that I’ve been growing vegetables in for the last four years, and I want to expand. So I set up 2″x12″ boards around a new 8’x16′ section of my backyard that I intend to use for a new garden.

Right now, the space is a big mess. Imagine a yard that nobody mowed for 15 years and you’ll have a fairly good picture.

My soil is dense clay with lots of really nasty roots throughout. It drains poorly and is very compacted.

Here’s what I’m thinking about doing:

  1. Cut everything down as close as possible.
  2. Double dig the area and mix in compost and straw.
  3. Plant a cover crop.
  4. Dig in the cover crop in the early summer
  5. Grow some vegetables, mostly stuff that can handle dense clay soils.

What kind of cover crop should I use? I need something cold-tolerant. I’m more concerned with improving soil structure and adding more humus to the soil rather than boosting nitrogen levels right now.

Side note: I know all about the idea of just turning this into a raised bed, filled with a truckload of fancy store-bought dirt. There’s something about that idea that just doesn’t appeal to me. I want to enrich what I have, not just import the finished product from somewhere else.

Sure, this idea doesn’t really withstand scrutiny; after all, the straw I’ll be mixing in and mulching with later is purchased.

But somehow buying an $8 bale of straw, and $40 worth of composted horse shit to mix in seems more in the spirit of sustainability than spending hundreds of dollars on a truck load of topsoil shipped in from who knows where.

New year’s python meme

Doing this after reading about it on Ben’s blog.

1. What’s the coolest Python application, framework or library you have discovered in 2009 ?

I love the restructured text tools. I use rst2pdf at least once a week.

2. What new programming technique did you learn in 2009?

This was a year of giving up on techniques, paradigms, development methods, frameworks, and fanciness in general and going back to basics. I’m like Rocky Balboa in Rocky IV — jogging in the snow and chopping wood with an axe.

3. What’s the name of the open source project you contributed the most in 2009? What did you do?

I’ve been working on pitz for about 15 months now. I’ve learned about a lot of stuff that I didn’t expect to learn when I decided to build a to-do manager. Because of that project, I’ve learned about how templating systems like jinja2 work internally, how to use cProfile, how to write tests for all sorts of weird situations, how to write a simple querying system, etc.

4. What was the Python blog or website you read the most in 2009?

I just went through my bookmarks tagged with python and didn’t see an obvious pattern. These days, the hundred or so people I follow on twitter keep me supplied with too much to read.

5. What are the three top things you want to learn in 2010 ?

Add an option for your script to drop into the debugger

I like using the python debugger to, umm, debug stuff. Just recently I wanted to use pdb to poke around in a script that would intermittently crash. I wanted to add an option to my script that would drop me into pdb when the script dies.

This turned out to be trivial to write. The script scratch.py doesn’t do much, but if you add the --drop-into-debugger option, you’ll get dumped into a pdb session, where you can study the problem more closely, like this:$ python scratch.py --drop-into-debugger
> /home/matt/scratch.py(15)main()
-> 1/0
(Pdb)

When I don’t add the --drop-into-debugger option, then of course the uncaught exception just crashes the script:
$ python scratch.py
Traceback (most recent call last):
File "scratch.py", line 23, in
main()
File "scratch.py", line 15, in main
1/0
ZeroDivisionError: integer division or modulo by zero

The code involved was really easy to write. I wrote a decorator called into_decorator with the decorator module, but that’s not strictly necessary. Here’s the code:

# vim: set expandtab ts=4 sw=4 filetype=python:

import pdb, sys
from decorator import decorator

@decorator
def into_debugger(f, *args, **kwargs):
try:
return f(*args, **kwargs)
except:
pdb.post_mortem()

def main():
x = 99
1/0

if __name__ == '__main__':
if len(sys.argv) > 1 and sys.argv[1] == '--drop-into-debugger':
into_debugger(main)()
else:
main()

That pdb.post_mortem function does the interesting work of inspecting the most recently caught exception. The pdb documentation is pretty dang sparse, but in this case, I got what I needed.

Anyhow, I hope this helps somebody out. I’ll probably be adding a –drop-into-debugger option on every script I write from now on.

The postgreSQL crosstab function is really powerful

I’m just beginning to wrap my brain around it. I’ve got some data that looks like this:

select rr as row_name, dt, c from mm limit 3;
+----------+----------+------+
| row_name | dt | c |
+----------+----------+------+
| xxx | 2009 mar | 552 |
| xxx | 2009 nov | 2179 |
| xxx | 2009 jun | 1101 |
+----------+----------+------+
(3 rows)

And now I’m going to transpose it like this:

select * from crosstab('select rr as row_name, dt, c from mm limit 3')
as foo(row_name text, dt_1 bigint, dt_2 bigint);
+----------+------+------+
| row_name | dt_1 | dt_2 |
+----------+------+------+
| xxx | 552 | 2179 |
+----------+------+------+
(1 row)

Neat, right?