Category Archives: Programming

virtualenvwrapper postactivate and screen is a wonderful combination

Posted on by

I switched to virtualenvwrapper from plain old virtualenv a while back because I found out that virtualenvwrapper has a postactivate hook I can define. Now when I do

$ workon pitz

to start my work day, I get a customized work environment exactly the way I want it. In addition to activating a virtualenv named pitz, the postactivate script changes me into the top of my pitz checkout directory and then it starts up or reattaches a screen session customized just for pitz.

Setting this up was trivially easy. Here’s what my postactivate script looks like:
$ cat ~/.virtualenvs/pitz/bin/postactivate
cd /home/matt/projects/pitz
screen -S pitz -c ~/.screenrc-pitz -d -R

And here is that .screenrc-pitz file:
$ cat ~/.screenrc-pitz
# Matt's homemade .screenrc.

# Draw that blue and red bar across the bottom of the terminal, with the cute
# stuff like the window names and system CPU load.
hardstatus on
hardstatus alwayslastline
hardstatus string "%{.bW}%-w%{.rW}%n %t%{-}%+w %=%{..G} %H %l %{..Y} %m/%d %C%a "

# Let me scroll back a thousand lines or so.
defscrollback 1024

# terminfo and termcap for nice 256 color terminal
# allow bold colors - necessary for some reason
attrcolor b ".I"

# tell screen how to set colors. AB = background, AF=foreground
termcapinfo mlterm 'Co#256:AB=\E[48;5;%dm:AF=\E[38;5;%dm'

# erase background with current bg color
defbce "on"

# Automatically start up a bunch of windows.
screen -t "shell" 0 pitz-shell
screen -t "code" 1
screen -t "tests" 2
chdir docs
screen -t "docs" 3

Everything from the top down to the comment # Automatically start up a bunch of windows. just tweaks how screen works. The last part creates a bunch of named windows. In the first wiindow, it doesn’t start a bash prompt. Instead it starts up the pitz-shell.

One tiny detail that may not be obvious is that if I do $ workon pitz in one terminal, and then maybe I ssh in and rerun $ workon pitz again later, I won’t cause a second screen to start up. Instead, I’ll remote-detach the original session and reconnect to it.

In practice, all my remote servers have highly customized screen sessions already waiting for me. I just need to log in and resume them, and I have all the interesting log files, database connections, and monitoring scripts already up. For example, I know my apache log is open in a pager in window 5. Window 2 has a psql session running. Window 1 has a supervisorctl client running.

This setup shaves off precious seconds when I’m trying to figure stuff out during an emergency.

This stuff isn’t rocket science, but I hope it helps somebody out.

Python niceties

Posted on by

nicety (plural niceties)

  1. A small detail that is nice or polite to have but isn’t necessary.

Make a comma-separated string

>>> x = ['a', 'b', 'c']
>>> ', '.join(x)
'a, b, c'

Isn’t that prettier than some big loop?

If you want to join a list of elements that aren’t strings, you’ll have to add a step:>>> x = [1, 2, 3]
>>> ', '.join(map(str, x))
'1, 2, 3'

I’m using map to convert every element to a string first.

Choose a random element from a sequence

>>> from random import choice
>>> choice(x) # doctest: +SKIP
1

The doctest: +SKIP text is just there because the element returned is randomly chosen, and so I can’t predict what the value will be.

Use keyword parameters in your larger string templates

Well-named keywords make big string templates MUCH easier to deal with:>>> tmpl = """
... Dear %(display_name)s,
...
... Greetings from %(origin_country)s! I have millions of
... %(currency_type)s in my bank account and I can send it all to you if
... you first deposit %(amount)s %(currency_type)s into my account
... first.
...
... Sincerely,
... %(alias)s
... %(origin_country)s
... """

Seeing the key names makes it much easier to remember what gets interpolated. Also, it makes it easier to use the same values more than once.

Usually I pass in a dictionary to the template, but sometimes I do this instead:>>> display_name = 'Matt'
>>> origin_country = 'Nigeria',
>>> currency_type = 'US dollars'
>>> amount = '250.00'
>>> s = tmpl % locals()

python contextmanagers, subprocesses, and an ascii spinner

Posted on by

I have a command line script that takes too long to run. I profiled the script and the problem is in some fairly complex third-party code. So instead of fixing that stuff, I’m going to distract users with a pretty ascii spinner.

Run this script to see the spinner in action (you’ll have to kill the process afterward):

$ cat spin_forever.py
# vim: set expandtab ts=4 sw=4 filetype=python:

import sys, time

def draw_ascii_spinner(delay=0.2):
for char in '/-\\\\|': # there should be a backslash in here.
sys.stdout.write(char)
sys.stdout.flush()
time.sleep(delay)
sys.stdout.write('\\\\r') # this should be backslash r.

while "forever":
draw_ascii_spinner()

Next I made a context manager (this is the name of the things you use with the new python “with” keyword) that uses the subprocess module to fork off another process to run the spinner:

$ cat distraction.py
# vim: set expandtab ts=4 sw=4 filetype=python:

import contextlib, subprocess, time

@contextlib.contextmanager
def spinning_distraction():
p = subprocess.Popen(['python', 'spin_forever.py'])
yield
p.terminate()

Finally I added my new context manager to the code that takes so long to run:

def main():

with spinning_distraction():
time.sleep(3) # pretend this is real work here.

if __name__ == '__main__':
main()

If you’re following along at home, you can run python distraction.py and you’ll see the spinner go until the main application finishes sleeping.

The main thing I don’t like is how I’m using an external file named spin_forever.py. I don’t want to worry about the path to where spin_forever.py lives. I would prefer to use some function defined in a module that I can import.

My favorite software quality metric is the income statement

Posted on by

I’ve been working on an open-source project for the better part of a year. It is very complex, very fancy, and has no users except for me. But I love it. It sharpens my skills and gives me a chance to discover puzzles and solve them in the most elegant solution possible.

But in the business world, this approach is downright silly. Nobody writes blank checks to vendors. Nothing in nature works like this either.

When a shark chases down and eats an unlucky spring breaker, the shark burns some calories during the swim, and then gets some calories back during the meal.

On average, the shark has to extract at least many calories while eating as it burned during the chase.

So on to the income statement. The shark makes a good metaphor for a software business. The hunt is the process of acquiring customers. The calories are the firm’s revenues and expenses. A lot of the quality of a software team can be measured in the income statement.

The perfect product is written once and it solves all problems and never needs any updates or extensions or bug fixes. The worst product has to be rewritten from scratch for every new customer and requires lots of bug fixes. Really well-written software falls somewhere between. It supports extensions, but they are quick and safe to do. Bugs can be quickly patched.

It wasn’t obvious to me initially, but this aspect is easy to measure in the cost of goods sold. If a firm lands a deal for $60k, but burns 160 hours rushing some tweaks through, then those hours drive up the cost of goods sold and drive down the firm’s margins.

You can look at the firm’s margins over time and watch whether the app is getting better or worse. Sure, there are mitigating factors, but in the long run, you can’t maintain an attractive income statement and a shitty code base.

Is this pylint error message valid or bogus?

Posted on by

The vast majority of pythoners use None, but I prefer descriptive strings for default values, like this:
from datetime import datetime, timedelta

def midnight_next_day(initial_time="use today's date"):

if initial_time == "use today's date":
initial_time = datetime.now()

return initial_time.date() + timedelta(days=1)

Pylint doesn’t like it though:

$ pylint -e f.py
No config file found, using default configuration
************* Module f
E: 10:midnight_next_day: Instance of 'str' has no 'date' member (but some types could not be inferred)

I changed from using a string as the default to None, and then pylint
didn’t mind:

$ cat f.py
from datetime import datetime
def midnight_next_day(initial_time=None):

if initial_time is None:
initial_time = datetime.now()

return initial_time.date() + timedelta(days=1)

$ pylint -e f.py
No config file found, using default configuration

Checking where a variable is None is probably much faster than doing string comparison, but I find the string more informative.

Looking for comments. I already posted this to comp.lang.python here and got some good feedback.

Which code layout do you like more?

Posted on by

I can lay out the same function in two ways. Which one is better?

One return statement and a mutable temporary variable


def f(a, b):
temp = g(a)
if b:
temp = h(temp)
return temp

Two return statements and everything uses single assignment


def f(a, b):
if b:
return h(g(a)
else:
return g(a)

I prefer the second approach. Ever since I learned how prolog requires single assignment, I avoid updating variables. I’m not adamant about it, but in general, I try not to. It’s just a style thing though. I’ve met a lot of programmers (older ones, usually) that like only having one return statement in a function. I don’t know why they like that, but I’ve heard it from numerous people.

Don’t negotiate on your estimates

Posted on by

My dad made a remark once that if you want to be good in sales, you can’t just believe that the glass is half full. You have to say with a straight face that a half-full glass is better than a completely full glass.

Anyhow, my shop has generally good relationships between the technical team and the sales team. There’s one thing that drives me nuts though: when we discuss priority and they try to negotiate with me on my estimates.

It goes a little like this:

me: I saw your note about creating a new permission-level between employee and administrator, where you want to let the shift supervisor do admin work on just their direct reports. In other words, two different shift supervisors will see two different lists of employees even though their permissions are identical. I figure that’s an 8-point project.

Billy Mays: No way! You just need an entity code to track what department that supervisor is attached to and then only show the people in that department.

Me: Yeah.

the shamwow guy: I don’t see how that is a big change. It’s just one extra column in the database. Just add that entity code into your queries.

Me: It is straightforward. I’m not confused about how to do it. It’s going to take a long time. There are lot of places that I need to alter to use this new approach.

Ron Popeil: are we using an object-oriented approach in the code? I hear you say that you’re going to have to change a lot of places in the code. At ${A MUCH BIGGER SOFTWARE COMPANY}, we used single objects so that we didn’t have to change a lot to alter behavior.

The minute when I start justifying my estimate or explaining why I have to change a lot of things, I’ve lost. They go into “close the deal” mode. There is a primal instinct at work here that outclasses higher brain functions. If sales people didn’t have a highly evolved chase instinct, they would suck at sales. When they see me trying to explain why it’s going to take that long, they will just nag nag nag until I give in.

And that’s what we need them for — that’s how sales works.

So I have found that the way out of this situation IS NOT to go technical and start explaining how code works. They’re NEVER going to say gee, now that you gave me a quick tutorial in web application design, and then you explained how you’ll need to rewrite the permissions decorators to use a new state factory, I guess your estimate makes sense.

Instead, this approach works OK for me:

Me: I think it is a great idea and it won’t require a lot of research and development, but it’s going to take a while. So based on that estimate, should we put into the queue or not?

I don’t address the remark about OOD and I don’t explain why. From a salesman’s perspective, you don’t close deals by acknowledging why the prospect doesn’t need what you’re selling. You close deals by overcoming objections. The salesman’s natural instinct is to see an obstacle and wear it down. So the key is not to feed that. Instead, stonewall them. Over time, they’ll respect you more because of it.

Here’s the bottom line: sales people are not like us. And that is OK. We need them. And we can work together as long as we all follow some rules. Don’t take away their bowl while they eat and never justify anything.

Some notes on nosetests and coverage

Posted on by

I use Nose to run my automated tests. It can report on code coverage like this:

$ nosetests --quiet --with-coverage --cover-package pitz
Name Stmts Exec Cover Missing
------------------------------------------------------------
pitz 29 29 100%
pitz.bag 108 107 99% 150
pitz.cmdline 50 12 24% 23-54, 62-79, 92-93, 96, 109-114, 119-130
pitz.entity 105 105 100%
pitz.exceptions 1 1 100%
pitz.junkyard 0 0 100%
pitz.junkyard.ditzloader 22 15 68% 31-37, 45-47
pitz.pitzfiles 0 0 100%
pitz.project 52 52 100%
pitz.projecttypes 0 0 100%
pitz.projecttypes.agilepitz 55 54 98% 66
pitz.projecttypes.simplepitz 66 61 92% 84-90
------------------------------------------------------------
TOTAL 488 436 89%
----------------------------------------------------------------------
Ran 69 tests in 6.350s

OK (SKIP=6)

Most of the uncovered code is in the cmdline module, which does a lot of work with the filesystem and starting up IPython, and I’m having trouble writing tests there. You could help, you know 🙂.

I’m keenly aware that running with coverage makes tests much slower. Normally, my pitz tests run in about half a second:


$ nosetests --quiet
----------------------------------------------------------------------
Ran 69 tests in 0.504s

OK (SKIP=6)

Fortunately, I don”t have to rerun all your tests to see what lines are uncovered. I can query the .coverage file created by nose afterward to get details. It is really easy to get coverage for just one module:

$ coverage -r -m pitz/cmdline.py
Name Stmts Exec Cover Missing
--------------------------------------------
pitz/cmdline 50 12 24% 23-54, 62-79, 92-93, 96, 109-114, 119-130

And getting coverage on multiple modules is straightforward, but kind of tedious:

$ coverage -r -m pitz/cmdline.py pitz/__init__.py pitz/entity.py
Name Stmts Exec Cover Missing
---------------------------------------------
pitz/__init__ 29 29 100%
pitz/cmdline 50 12 24% 23-54, 62-79, 92-93, 96, 109-114, 119-130
pitz/entity 105 105 100%
---------------------------------------------
TOTAL 184 146 79%

I don’t know of an elegant solution to do what nosetests does, where it shows me all coverage for a package. Running coverage -r without any module lists dumps out everything, which is never what I want:

$ coverage -r
Name Stmts Exec Cover
--------------------------------------------------------------------------------------------------------------------------------------
/home/matt/checkouts/myfiles/bin/__init__ 0 0 100%
/home/matt/virtualenvs/pitz/lib/python2.6/site-packages/Jinja2-2.1.1-py2.6-linux-i686.egg/jinja2/__init__ 12 11 91%
/home/matt/virtualenvs/pitz/lib/python2.6/site-packages/Jinja2-2.1.1-py2.6-linux-i686.egg/jinja2/bccache 111 36 32%
Traceback (most recent call last):
KeyboardInterrupt

A while back on twitter I posted that I wanted my editor to read my .coverage file and highlight the lines that aren’t covered. I don’t know anything about how to control syntax highlighting, but I run this little deal from within vim to get the same facts:

:! coverage -r -m %

My edit buffer disappears and I see this instead:

Name Stmts Exec Cover Missing
---------------------------------------
cmdline 50 12 24% 23-54, 62-79, 92-93, 96, 109-114, 119-130

Press ENTER or type command to continue

In vim, :! is how you run a command in a shell. Incidentally, it is also possible to pull the results from the command into vim and write data from vim to the command, but that’s not what I want right now.

Vim replaces the % symbol with the file I’m currently editing. Of course, this command only works when I start vim in the same directory as the .coverage file. If I ain’t in that directory, then I have to specify how to find the .coverage file by setting the environmental variable COVERAGE_FILE like this:

:! COVERAGE_FILE=../.coverage coverage -r -m %

Setting it that way means it doesn’t last beyond that one shell. If I want vim to remember the setting, I could set COVERAGE_FILE when I start vim like this:

$ COVERAGE_FILE=../.coverage vi cmdline.py

Or I could export it like this:

$ export COVERAGE_FILE=../.coverage
$ vi cmdline.py

In summary, coverage is a neat tool, but it is silly to think that 100% test coverage guarantees anything. Coverage won’t warn you when your calculator returns 3 for 1 + 1.