New year’s python meme

Doing this after reading about it on Ben’s blog.

1. What’s the coolest Python application, framework or library you have discovered in 2009 ?

I love the restructured text tools. I use rst2pdf at least once a week.

2. What new programming technique did you learn in 2009?

This was a year of giving up on techniques, paradigms, development methods, frameworks, and fanciness in general and going back to basics. I’m like Rocky Balboa in Rocky IV — jogging in the snow and chopping wood with an axe.

3. What’s the name of the open source project you contributed the most in 2009? What did you do?

I’ve been working on pitz for about 15 months now. I’ve learned about a lot of stuff that I didn’t expect to learn when I decided to build a to-do manager. Because of that project, I’ve learned about how templating systems like jinja2 work internally, how to use cProfile, how to write tests for all sorts of weird situations, how to write a simple querying system, etc.

4. What was the Python blog or website you read the most in 2009?

I just went through my bookmarks tagged with python and didn’t see an obvious pattern. These days, the hundred or so people I follow on twitter keep me supplied with too much to read.

5. What are the three top things you want to learn in 2010 ?

Add an option for your script to drop into the debugger

I like using the python debugger to, umm, debug stuff. Just recently I wanted to use pdb to poke around in a script that would intermittently crash. I wanted to add an option to my script that would drop me into pdb when the script dies.

This turned out to be trivial to write. The script scratch.py doesn’t do much, but if you add the --drop-into-debugger option, you’ll get dumped into a pdb session, where you can study the problem more closely, like this:$ python scratch.py --drop-into-debugger
> /home/matt/scratch.py(15)main()
-> 1/0
(Pdb)

When I don’t add the --drop-into-debugger option, then of course the uncaught exception just crashes the script:
$ python scratch.py
Traceback (most recent call last):
File "scratch.py", line 23, in
main()
File "scratch.py", line 15, in main
1/0
ZeroDivisionError: integer division or modulo by zero

The code involved was really easy to write. I wrote a decorator called into_decorator with the decorator module, but that’s not strictly necessary. Here’s the code:

# vim: set expandtab ts=4 sw=4 filetype=python:

import pdb, sys
from decorator import decorator

@decorator
def into_debugger(f, *args, **kwargs):
try:
return f(*args, **kwargs)
except:
pdb.post_mortem()

def main():
x = 99
1/0

if __name__ == '__main__':
if len(sys.argv) > 1 and sys.argv[1] == '--drop-into-debugger':
into_debugger(main)()
else:
main()

That pdb.post_mortem function does the interesting work of inspecting the most recently caught exception. The pdb documentation is pretty dang sparse, but in this case, I got what I needed.

Anyhow, I hope this helps somebody out. I’ll probably be adding a –drop-into-debugger option on every script I write from now on.

The postgreSQL crosstab function is really powerful

I’m just beginning to wrap my brain around it. I’ve got some data that looks like this:

select rr as row_name, dt, c from mm limit 3;
+----------+----------+------+
| row_name | dt | c |
+----------+----------+------+
| xxx | 2009 mar | 552 |
| xxx | 2009 nov | 2179 |
| xxx | 2009 jun | 1101 |
+----------+----------+------+
(3 rows)

And now I’m going to transpose it like this:

select * from crosstab('select rr as row_name, dt, c from mm limit 3')
as foo(row_name text, dt_1 bigint, dt_2 bigint);
+----------+------+------+
| row_name | dt_1 | dt_2 |
+----------+------+------+
| xxx | 552 | 2179 |
+----------+------+------+
(1 row)

Neat, right?

What’s wrong with using somebody else’s framework

“I wish that you, oh exalted one, would not be angry with me,” said the
young man. “I have not spoken to you like this to argue with you, to
argue about words. You are truly right, there is little to opinions.
But let me say this one more thing: I have not doubted in you for a
single moment. I have not doubted for a single moment that you are
Buddha, that you have reached the goal, the highest goal towards which
so many thousands of Brahmans and sons of Brahmans are on their way.
You have found salvation from death. It has come to you in the course
of your own search, on your own path, through thoughts, through
meditation, through realizations, through enlightenment. It has not
come to you by means of teachings! And–thus is my thought, oh exalted
one,–nobody will obtain salvation by means of teachings! You will not
be able to convey and say to anybody, oh venerable one, in words and
through teachings what has happened to you in the hour of enlightenment!
The teachings of the enlightened Buddha contain much, it teaches many to
live righteously, to avoid evil. But there is one thing which these so
clear, these so venerable teachings do not contain: they do not contain
the mystery of what the exalted one has experienced for himself, he
alone among hundreds of thousands. This is what I have thought and
realized, when I have heard the teachings. This is why I am continuing
my travels–not to seek other, better teachings, for I know there are
none, but to depart from all teachings and all teachers and to reach my
goal by myself or to die. But often, I’ll think of this day, oh exalted
one, and of this hour, when my eyes beheld a holy man.”

From Siddhartha by Herman Hesse.

pitz has a CLI

I’ve exposed lots and lots of pitz functionality as command-line scripts. Here’s the list so far:

$ pitz-help
pitz-abandon-task Abandon a task
pitz-add-task Walks through the setup of a new Task.
pitz-components All components in the project
pitz-estimates All estimates in the project
pitz-everything No description
pitz-finish-task Finish a task
pitz-milestones No description
pitz-my-tasks List my tasks
pitz-people All people in the project
pitz-prioritize-above Put one task in front of another task
pitz-prioritize-below Put one task behind another task
pitz-recent-activity 10 recent activities
pitz-setup No description
pitz-shell Start an ipython session after loading in a ...
pitz-show Show detailed view of one entity
pitz-start-task Begin a task
pitz-statuses All statuses in the project
pitz-tasks All tasks in the project
pitz-todo List every unstarted and started task in the...
pitz-unassign-task Take this task off somebody's list of stuff ...

A few iPhone or android app ideas

Just like my post with some web app ideas, feel free to steal any of these.

A fake lottery game

This has already been done several times. You would start with some money (fake money) then buy some tickets and scratch them off. Some would win (more money) and most would lose. Maybe I would hide silly messages in the tickets. The gimmick would be the activity of scratching off the tickets, or maybe spending money on other side treats. Like cigarettes. While smoking, maybe the screen would be hazy. Or maybe you could buy crystal meth, too, and you could scratch tickets really really fast.

And so forth. On to idea number two!

Pick a scab

If you want high-brow, go somewhere else. So, the idea here is obvious, right? The screen shows an ugly scab. You can pick at it. Then it bleeds. Then maybe it heals, and the cycle starts again. Probably been done before.

AWESOME RIGHT?

Pop the pimple

Yeah, so apparently this has already been done too, damnit. But what if the pimples could be superimposed on top of the photos in your album?

Help me rewrite some repetitive scripts

I have about a dozen functions (that are run as scripts) that have very similar sections interleaved with specialized code. I copied two of the scripts below. Here’s the first:
def pitz_estimate_task():

p = optparse.OptionParser()

p.add_option('--version', action='store_true',
help='Print the version and exit')

# This script requires these arguments.
p.set_usage("%prog task [estimate]")
options, args = p.parse_args()

if options.version:
print_version()
return

# This is unique to this script.
if not args:
p.print_usage()
return

# And now we're back to boring generic stuff.
pitzdir = Project.find_pitzdir(options.pitzdir)
proj = Project.from_pitzdir(pitzdir)
proj.find_me()

# This section is specific to this script.
t = proj[args[0]]

if len(args) == 2:
est = proj[args[1]]

else:
est = Estimate.choose_from_already_instantiated()

t['estimate'] = est
# That was the last thing that was specific to just this script.

# Save the project (generic).
proj.save_entities_to_yaml_files()

That script does some “generic” stuff to build an object p, then adds on some extra tweaks to p, and uses p to build an options object and an args object.

Then the script does some generic stuff to build a proj object based on the data in the options.pitzdir object, and does some various method calls on the proj object.

And here’s another script:
def pitz_attach_file():

p = optparse.OptionParser()

p.add_option('--version', action='store_true',
help='Print the version and exit')

# Notice this line is different than the one in pitz_estimate_task.
p.set_usage("%prog entity file-to-attach")
options, args = p.parse_args()

if options.version:
print_version()
return

# This section is different too.
if len(args) != 2:
p.print_usage()
return

# Back to the generic code to build the project.
pitzdir = Project.find_pitzdir(options.pitzdir)
proj = Project.from_pitzdir(pitzdir)
proj.find_me()

# Some interesting stuff that is specific just for this script.
e, filepath = proj[args[0]], args[1]
e.save_attachment(filepath)

# Save the project. (Generic).
proj.save_entities_to_yaml_files()

So, the pattern in every script is: generic code, specific code, generic code, specific code, generic code. And each step depends on the previous step.

I know I could do stuff like wrap all the generic stuff into functions, but I’m not really a fan of that approach. I’m looking for an interesting way to reduce all repetition, but keep the legibility. I’m thinking some nested context managers or decorators might be the way to go. I like to hear ideas from other people, so, please, let me hear them.

By the way, all this code is from the command-line module of pitz, available here. That’s where you can see all the different variations on the same theme.

Why I’m hooked on parameters

I have this need to make everything into a parameter. Here’s some code I wrote recently:

import subprocess, tempfile

def edit_with_editor(s=None):
"""
Returns the text typed in the editor, after running strip().
"""
with tempfile.NamedTemporaryFile() as t:
if s:
t.write(str(s))
t.seek(0)

subprocess.call([os.environ.get('EDITOR', 'vi'), t.name])
return t.read().strip()

There’s a voice in my head that says I shouldn’t be importing the subprocess and tempfile modules outside my function and then referring to them from within. And, the voice goes on, it wouldn’t be OK to move the import within my function either. Instead, if my function needs to use NamedTemporaryFile, I should pass that function in as a parameter.

I think it all starts with the fact that I studied economics in college. I had a lot of lectures that started with a professor drawing something like this on the chalk board:

labor supply = f(...)

And then during the lecture, she would slowly replace the ellipses with parameters. By the end of class, the function might look like:

labor supply = f(income, wealth)

Then I’d also have a few pages of notes explaining the nature of the relationship between how hard somebody is willing to work, the wage they can earn, and the wealth they already have. The thing that sunk in deep is the idea that parameters (and only parameters) are what drives the dependent variable on the left hand site. Anything that drives labor supply is listed as a parameter. If it ain’t on the right hand side, then it is not relevant.

By the way, the war between the wealth effect and the income effect is one of the areas of economics that really does explain our behavior pretty dang well, and there’s neat charts involved, so go read about it.

How to download photos from phone over bluetooth with ubuntu

I have a samsung sync phone. It is about two years old now. It isn’t fancy, but it has a camera and a bluetooth device.

I wrote a python script to copy photos from my phone to my laptop. Any time I take a bunch of pictures, I just run
$ getpics.py
and if my phone is anywhere nearby, my computer will pull all the photos off my phone.

Bluetooth is a great little tool and I’m surprised it hasn’t caught on more with people like us. I’ve been able to access my phone from my laptop when I’m on the second floor and the phone is across the house and downstairs, still in my work bag. And bluetooth USB dongles are cheap (like around $9). There’s no need for carrying around USB keys or plugging in iPods. Everything should use bluetooth.

The getpics.py script requires that the phone and the laptop have already been bonded with each other. That’s not much work. Put your phone in to discoverable mode, then search for devices with your computer. When you find your phone, punch the same 4-digit PIN into the phone and your computer.

Anyway, here’s the script. I hope it helps somebody out.
#! /usr/bin/env python

# W. Matthew Wilson wrote this script and he released
# it into the public domain.

# vim: set expandtab ts=4 sw=4 filetype=python:

"""
Pull all the images off my phone.
"""

import commands, logging, os

from xml.etree import ElementTree
from xml.parsers.expat import ExpatError

logging.basicConfig(level=logging.DEBUG)

def get_list_of_pictures():

"""
Yield a list of strings, where each string is the file name of a
picture.
"""

# You'll need to change the 00:1D... stuff to your phone's
# MAC address.
# You might need to change /Graphics/ to where your phone
# stores its photos.
status, output = commands.getstatusoutput(
"obexftp -b 00:1D:F6:55:7E:D9 -l /Graphics/")

logging.debug("status of obexftp: %s" % status)

logging.info("Finished running obexftp...")

# Unfortunately, the output from obexftp includes some goofy status
# crap and then some XML data. So I parse each line separately,
# and skip the lines where parsing fails.

for line in output.split('\n'):

logging.debug("About to parse line %s." % line.strip())

try:
x = ElementTree.XML(line)
# Skip lines without a name attribute.
if 'name' not in x.keys(): continue

# Extract and yield the name
photofilename = x.get('name')

logging.debug("About to yield %s." % photofilename)

yield photofilename

logging.debug("Finished yielding %s." % photofilename)

except ExpatError:

logging.debug("This line is not valid XML: %s."
% line.strip())

def grab_pic(photofilename):

"""
Pull the photo named photofilename and save it locally.
"""

# You probably want to change this.
os.chdir("/home/matt/Documents/Photos/phonepics")

status, output = commands.getstatusoutput(
"obexftp -b 00:1D:F6:55:7E:D9 -g /Graphics/%s" % photofilename)

logging.info("Finished storing %s locally." % photofilename)

if __name__ == "__main__":

logging.info("Starting...")

for photo in get_list_of_pictures():
grab_pic(photo)

logging.info("All done!")

Approaching jQuery popup callback hell

I wrote this post on the jQuery mailing list and nobody replied, so I’m pasting it here. I could really use some advice.

I’m using a modal dialogs and I love them, but I haven’t found a really elegant way to handle actions in the dialog window that require changes to the parent page.

Here’s an example. I have a monthly calendar page that lists employee names on the days they are supposed to work. Clicking on an employee name opens a “shift detail” page in a modal dialog.

That shift detail page has more information like the specific tasks planned for the day, the start time and stop time of the shift, etc. From this shift detail screen, I can remove this particular employee from the schedule by submitting an AJAX POST from this popup.

After I remove the employee, I need to update both the popup window and the original page that hosted the link to the popup window. Right now I do this by adding a callback that fires when the AJAX POST succeeds. That callback then updates both pages. The callback is named “after_remove_employee”.

This system gets really nasty when I use the “shift detail” popup on different screens. For example, in addition to the monthly view, I also have a weekly view with more information. So after an employee is removed from the schedule on the weekly view, I need to do some different things in the callback.

Right now, the way I handle this is that I define the same callback twice. I define “var after_remove_employee = function (data) {…} on the weekly view to do what it needs there, and then I define it differently on the monthly view.

I’ve simplified the problem to help explain it. In reality, I have lots of different popups on lots of different pages, and in each popup, there are many different possible actions.

I’m sure I’m not the only one that’s been in this scenario. What is an elegant solution?

I’m thinking about using custom events. So, the callback after a successful AJAX POST would just fire an “employee removed” event, and everybody subscribed would get a reference to the event object and do whatever they want.

However, I’ve never used JS events before, and I don’t know if this is even possible.

Please, any feedback is welcome.