A new pitz release (1.1.2)

I’m using semantic versioning now, so I bumped from 1.0.7 to 1.1.0 after adding a few tweaks to the command-line interface. Then I discovered a silly mistake in the setup.py file, and released 1.1.1. Then I realized I did the fix wrong, and then released 1.1.2 a few minutes later.

The –quick option

Normally, running pitz-add-task will prompt for a title and then open $EDITOR so I can write a description. After that, pitz will ask for choices for milestone, owner, status, estimate, and tags.

Sometimes I want to make a quick task without getting prompted for all this stuff.

I already had a --no-description option that would tell pitz-add-task to not open $EDITOR for a description. And I already had a --use-defaults option to just choose the default values for milestone, owner, status, estimate, and tags.

But when I just want to make a quick to-do task as a placeholder, writing out all this stuff:$ pitz-add-task --no-description --use-defaults -t "Fix fibityfoo"

is kind of a drag. So I made a --quick option (also available as -q) that does the same thing as --no-description --use-defaults.

The -1 alias for –one-line-view

This is the typical way that a to-do list looks:$ pitz-my-todo -n 3
slice from To-do list for matt

(3 task entities, ordered by ['milestone', 'status', 'pscore'])

Write new CLI scripts (witz!) to talk to pitz-webapp 9f1c76
matt | paused | difficult | 1.0 | 1
webapp, CLI
Starting up and loading all the pitz data at the beginning of ever...

Experiment with different task summarized views 0f6fee
matt | unstarted | straightforward | 1.0 | 1
Right now, the summarized view of a task looks a little like this:...

Add more supported URLs to pitz-webapp 295b5f
matt | unstarted | straightforward | 1.0 | 0
I want to allow these actions through the webapp: * Insert a ne...

Incidentally, notice the -n 3 option limits the output to the first three tasks.

Tasks also have a one-line view:$ pitz-my-todo -n 3 --one-line-view
slice from To-do list for matt

(3 task entities, ordered by ['milestone', 'status', 'pscore'])

Write new CLI scripts (witz!) to talk to pitz-webapp 9f1c76
Experiment with different task summarized views 0f6fee
Add more supported URLs to pitz-webapp 295b5f

Typing out --one-line-view is tedious, so now, -1 is an alias that works as well:$ pitz-my-todo -n 3 -1
slice from To-do list for matt

(3 task entities, ordered by ['milestone', 'status', 'pscore'])

Write new CLI scripts (witz!) to talk to pitz-webapp 9f1c76
Experiment with different task summarized views 0f6fee
Add more supported URLs to pitz-webapp 295b5f

pitz has a CLI

I’ve exposed lots and lots of pitz functionality as command-line scripts. Here’s the list so far:

$ pitz-help
pitz-abandon-task Abandon a task
pitz-add-task Walks through the setup of a new Task.
pitz-components All components in the project
pitz-estimates All estimates in the project
pitz-everything No description
pitz-finish-task Finish a task
pitz-milestones No description
pitz-my-tasks List my tasks
pitz-people All people in the project
pitz-prioritize-above Put one task in front of another task
pitz-prioritize-below Put one task behind another task
pitz-recent-activity 10 recent activities
pitz-setup No description
pitz-shell Start an ipython session after loading in a ...
pitz-show Show detailed view of one entity
pitz-start-task Begin a task
pitz-statuses All statuses in the project
pitz-tasks All tasks in the project
pitz-todo List every unstarted and started task in the...
pitz-unassign-task Take this task off somebody's list of stuff ...

Help me rewrite some repetitive scripts

I have about a dozen functions (that are run as scripts) that have very similar sections interleaved with specialized code. I copied two of the scripts below. Here’s the first:
def pitz_estimate_task():

p = optparse.OptionParser()

p.add_option('--version', action='store_true',
help='Print the version and exit')

# This script requires these arguments.
p.set_usage("%prog task [estimate]")
options, args = p.parse_args()

if options.version:

# This is unique to this script.
if not args:

# And now we're back to boring generic stuff.
pitzdir = Project.find_pitzdir(options.pitzdir)
proj = Project.from_pitzdir(pitzdir)

# This section is specific to this script.
t = proj[args[0]]

if len(args) == 2:
est = proj[args[1]]

est = Estimate.choose_from_already_instantiated()

t['estimate'] = est
# That was the last thing that was specific to just this script.

# Save the project (generic).

That script does some “generic” stuff to build an object p, then adds on some extra tweaks to p, and uses p to build an options object and an args object.

Then the script does some generic stuff to build a proj object based on the data in the options.pitzdir object, and does some various method calls on the proj object.

And here’s another script:
def pitz_attach_file():

p = optparse.OptionParser()

p.add_option('--version', action='store_true',
help='Print the version and exit')

# Notice this line is different than the one in pitz_estimate_task.
p.set_usage("%prog entity file-to-attach")
options, args = p.parse_args()

if options.version:

# This section is different too.
if len(args) != 2:

# Back to the generic code to build the project.
pitzdir = Project.find_pitzdir(options.pitzdir)
proj = Project.from_pitzdir(pitzdir)

# Some interesting stuff that is specific just for this script.
e, filepath = proj[args[0]], args[1]

# Save the project. (Generic).

So, the pattern in every script is: generic code, specific code, generic code, specific code, generic code. And each step depends on the previous step.

I know I could do stuff like wrap all the generic stuff into functions, but I’m not really a fan of that approach. I’m looking for an interesting way to reduce all repetition, but keep the legibility. I’m thinking some nested context managers or decorators might be the way to go. I like to hear ideas from other people, so, please, let me hear them.

By the way, all this code is from the command-line module of pitz, available here. That’s where you can see all the different variations on the same theme.

Need help with data files and setup.py

I’m working on a package that includes some files that are meant to be copied and edited by people using the package.

My project is named “pitz” and it is a bugtracker. Instead of using a config file to set the options for a project, I want to use python files.

When somebody installs pitz, I want to save some .py files somewhere so that when they run my pitz-setup script, I can go find those .py files and copy them into their working directory.

I have two questions:

  1. Do I need to write my setup.py file to specify that the .py files in particular directory need to be treated like data, not code? For example, I don’t want the installer to hide those files inside an egg.
  2. How can I find those .py files later and copy them?

Here’s my setup.py so far:

from setuptools import setup, find_packages
version = '0.1'
description="Python to-do tracker inspired by ditz (ditz.rubyforge.org)",

ditz (http://ditz.rubyforge.org) is the best distributed ticketing
system that I know of. There's a few things I want to change, so I
started pitz.""",

author='Matt Wilson',
packages=find_packages(exclude=['ez_setup', 'examples', 'tests']),



# 'PyYAML',
# 'sphinx',
# 'nose',
# 'jinja2',
# -*- Extra requirements: -*-

# I know about the much fancier entry points, but I prefer this
# solution. Why does everything have to be zany?
scripts = ['scripts/pitz-shell'],

test_suite = 'nose.collector',

When I run python setup.py install, I do get those .sample files copied, but they get copied into a folder way inside of my pitz install:

$ cd ~/virtualenvs/scratch/lib/
$ find -type f -name '*.sample'

I don’t know how I can write a script to copy those tracpitz.py.sample files out. Maybe I can ask pitz what its version is, and then build a tring and use os.path.join, but that doesn’t look like any fun at all.

So, what should I do instead?

How to see the to-do list for pitz

pitz is (among other things) a to-do list tracker like trac or bugzilla or version one.

I’m storing the list of stuff to do for pitz in the pitz source code. Here’s how to see the unfinished stuff in pitz.

Get a copy of the code

$ git clone git://github.com/mw44118/pitz.git
Initialized empty Git repository in /home/matt/pitz/.git/
remote: Counting objects: 621, done.
remote: Compressing objects: 100% (604/604), done.
remote: Total 621 (delta 383), reused 0 (delta 0)
Receiving objects: 100% (621/621), 98.52 KiB | 135 KiB/s, done.
Resolving deltas: 100% (383/383), done.

Install it, probably in a virtualenv

$ source ~/virtualenvs/pitz/bin/activate
$ python setup.py develop

Fire up pitz-shell

I have written one command-line tool so far: pitz-shell. Use it to start a python interpreter loaded with any pitz project. Here’s how to start a session for pitz itself:

$ pitz-shell pitz/pitzfiles/project-99c58812-5c1c-4fec-874c-c998933ba88b.yaml
/home/matt/virtualenvs/pitz/lib/python2.6/site-packages/ipython-0.9.1-py2.6.egg/IPython/Magic.py:38: DeprecationWarning: the sets module is deprecated
from sets import Set

pitz-shell imports a bunch of classes and makes an object named p (p stands for project). p has all the information about the project described in the yaml file passed in as an argument to pitz-shell. The __repr__ method on p gives some summarized data:

In [1]: p

p.todo is a property that just returns a bag of unfinished tasks for the project:

In [4]: p.todo

You can print any bag to see all the contents of the bag, and p.todo is no different:

In [5]: print(p.todo)
Stuff to do

(23 task entities)

0: Add support for something like 'ditz grep' (unknown status)
1: Update entities by loading a CSV file (unknown status)
2: Figure out why some tasks are not converting pointers to objects (unknown status)
3: Support intersection, union, and other set operations on bags (unknown status)
4: Demonstrate really simple tasks and priorities workflow (unknown status)
5: Support a .pitz config file with all pitz scripts (unknown status)
6: Add a todo property on project (or maybe bag) (unknown status)
7: Write code to use strings as keys (unknown status)
8: Prompt to save work at the end of an interactive pitz session (unknown status)
9: Make it possible to support a filter like attribute!=value (unknown status)
10: Write code to support sorting by anything (unknown status)
11: Support hooks (unknown status)
12: Write an attributes property on a bag that lists count of each attribute in any entities (unknown status)
13: Allow two bags to be compared for equality by using their entities (unknown status)
14: Make it easy to list each employee's tasks (unknown status)
15: Support a $PITZDIR env var to tell where yaml files live (unknown status)
16: Demonstrate release -< iteration -< user story -< task workflow. (unknown status) 17: Load new entities from a CSV file (unknown status) 18: Support grep on entities (unknown status) 19: write data to yaml in order (unknown status) 20: Support entity subclasses like releases, iterations, user stories, and tasks (unknown status) 21: A bag should dump to a single CSV file (unknown status) 22: Support using substring of name as name (unknown status)

That's how you see the to-do list for pitz!

In a future post, I'll show how to make new tasks and how to update tasks.

I also need to explain how Pitz lets you come up with whatever wacky workflow you want. When you set up a pitz project, you can use the classes I came up with, or subclass Entity into your own weird types. In a future post, I'll show I'm using pitz to model an agile development system using releases, iterations, checkpoints, user stories, tasks, and people.

pitz data model outline

I just finished a whole bunch of documentation on the pitz data model. You can read it here or just read all the stuff I copied below:

There are two classes in pitz: entities and bags. Everything else are subclasses of these two.


Making them

Every entity is an object like a dictionary. You can make an entity like this:

>>> from pitz import Entity
>>> e = Entity(title="example entity",
... creator="Matt",
... importance="not very")

You can also load an entity from a yaml file, but I’ll explain that later.

You can look up a value for any attribute like this:

>>> e['title']
'example entity'
>>> e.keys() #doctest: +NORMALIZE_WHITESPACE
['name', 'creator', 'importance', 'title', 'modified_time',
'created_time', 'type']
>>> e['type']

Viewing them

Entities have a summarized view useful when you want to see a list of entities, and a detailed view that shows all the boring detail:

>>> e.summarized_view
'example entity (entity)'

>>> print(e.detailed_view) #doctest: +SKIP
example entity (entity)



not very

example entity

2009-04-04 07:47:09.456068

2009-04-04 07:47:09.456068


Notice how our entity has some attributes we never set, like name, type, created_time, and modified_time. I make these in the __init__ method of the entity class.

By the way, you can ignore the #doctest: +SKIP comment. That is there so the doctests will skip trying to running this example, which will generate unpredictable values.

Saving and loading them

Entities have an instance method named to_yaml_file and a from_yaml_file classmethod. Here’s how to use them:

>>> outfile = e.to_yaml_file('.') # Writes file to this directory.
>>> e2 = Entity.from_yaml_file(outfile)


Making them

While entities are based on dictionaries, bags are based on lists. You can give a bag instance a title, which is nice for remembering what it is you want it for. Bags make it easy to organize a bunch of entities.

>>> from pitz import Bag
>>> b = Bag(title="Stuff that is not very important")
>>> b.append(e)

Viewing them

Converting a bag to a string prints the summarized view of all the entities inside:

>>> print(b) #doctest: +SKIP
Stuff that is not very important

1 entity entities

0: example entity (entity)

That number 0 can be used to pull out the entity at that position, just like a regular boring old list:

>>> e == b[0]

Querying them

Bags have a matches_dict method that accepts a bunch of key-value pairs and then returns a new bag that contains all the entities in the first bag that match all those key-value pairs.

First, I’ll make a few more entities:

>>> e1 = Entity(title="example #1", creator="Matt",
... importance="Really important")
>>> e2 = Entity(title="example #2", creator="Matt",
... importance="not very")

Now I’ll make a new bag that has both of these new entities:

>>> b = Bag('Everything')
>>> b.append(e1)
>>> b.append(e2)
>>> print(b)

2 entity entities

0: example #1 (entity)
1: example #2 (entity)

Here is how to get a new bag with just the entities that have an importance attribute set to “not very”:

>>> not_very_important = b.matches_dict(importance="not very")
>>> len(not_very_important) == 1
>>> not_very_important[0] == e2

Since matches_dict is the most common method I call on a bag, I made the __call__ method on the Bag class run matches_dict. So that means this works just as well:

>>> not_very_important = b(importance="not very")

Saving and loading them

Bags can send all contained entities to yaml files with to_yaml_files, and bags can load a bunch of entities from yaml files with from_yaml_files.

Right now, there is no way for a bag to save itself to yaml.

The Special Project Bag

After I finished bags and entities, I thought I was done, but then I ran into a few frustrations:

  • When I made a bunch of entities, but didn’t append them all into one bag, then I couldn’t run filters across all of them.
  • At the end of a session, it wasn’t easy for me to make sure that all of the entities got saved out to yaml.
  • I couldn’t figure out an elegant way to store one entity as a value for another entity’s attribute.

So I made a “special” Bag subclass called Project. The idea here is that every entity should be a member of the project bag. Also, every entity should have a reference back to the project.

Using a project is easy. Just pass it in as the first argument when you make an entity. Imagine I want to link some tasks to Matt and some other tasks to Lindsey. First I make a project:

>>> from pitz import Project
>>> weekend_chores = Project(title="Weekend chores")

Now I make the rest of the entities:

>>> matt = Entity(weekend_chores, title="Matt")
>>> lindsey = Entity(weekend_chores, title="Lindsey")
>>> t1 = Entity(weekend_chores, title="Mow the yard", assigned_to=matt)
>>> t2 = Entity(weekend_chores, title="Buy some groceries",
... assigned_to=lindsey)

Now it is easy to get tasks for matt:

>>> chores_for_matt = weekend_chores(assigned_to=matt)
>>> mow_the_yard = chores_for_matt[0]
>>> mow_the_yard['assigned_to'] == matt


There’s a problem in that last example: when I send this mow_the_yard entity out to a YAML file, what will I store as the value for the “assigned_to” attribute?

In SQL, this is what foreign keys are good for. In my chores table, I would store a reference to a particular row in the people table.

I wanted the same functionality in pitz, so I came up with pointers. This is dry stuff, so here’s an example:

>>> class Chore(Entity):
... pointers = dict(assigned_to='person')
>>> class Person(Entity):
... pass
>>> matt = Person(weekend_chores, title="Matt")
>>> lindsey = Person(weekend_chores, title="Lindsey")
>>> ch1 = Chore(weekend_chores, title="Mow the yard", assigned_to=matt)
>>> ch2 = Chore(weekend_chores, title="Buy some groceries",
... assigned_to=lindsey)

Not much is different, but instead of matt, lindsey, and the various chores all being entities, they’re now subclasses. But here’s the advantage of defining pointers on Chore:

>>> ch1['assigned_to'] >>> matt['name'] # doctest: +SKIP
>>> ch1.replace_objects_with_pointers()
>>> ch1['assigned_to'] # doctest: +SKIP

First of all, notice how I printed out the name attribute on matt.

After running the replace_objects_with_pointers method, I don’t have a reference to the matt object. Instead, I have matt’s name now.

Now I can send this data out to a yaml file. And when I load it back in from yaml, I can then reverse this action, and go look up an entity with the same name:

>>> mn = matt.name
>>> matt == weekend_chores.by_name(mn)

In practice, I convert all the entities to pointers, then write out the yaml files, then convert all the pointers back into objects automatically.

That’s the end of the data model documentation. I hope that shines enough light so that it is obvious if pitz would be useful to you or not.

I’m working on a separate article where I show some real-world workflows modeled in pitz, but that will be next week’s post.

How to load ditz issues into python

Ditz is a fantastic distributed bug tracking system written in Ruby.

Here’s some code that you can use to load some ditz issues into a python interpreter. You need to install my pitz project first though.

>>> from pitz.junkyard.ditzloader import *
>>> from glob import glob
>>> issue_file_path = glob('../.ditz/issue-*.yaml')[0]
>>> import yaml
>>> issue = yaml.load(open(issue_file_path))
>>> issue.title
'Distribute reports by email'
>>> print issue.desc
Somehow allow people to sign up for report subscriptions.

So when new reports come out, they get updated. Maybe I can use
RSS feeds to hold the reports.
>>> print issue.log_events
[[datetime.datetime(2008, 9, 2, 17, 47, 44, 549355), 'Matthew Wilson ', 'created', ''], [datetime.datetime(2008, 9, 2, 19, 18, 3, 286902), 'Matthew Wilson ', 'assigned to release 3.5.1 from unassigned', ''], [datetime.datetime(2008, 9, 4, 18, 27, 19, 571991), 'Matthew Wilson ', 'unassigned from release 3.5.1', '']]

That above just showed how to read the ditz issue. I’m having trouble updating the issue and then saving it in a format that ditz can still read. Updating the ditz issue and saving it out again is easy:

>>> issue.title = 'Distribute reports by email or RSS'
>>> open(issue_file_path, 'w').write(yaml.dump(issue))

But when I try to load it with ditz, I get this error:

$ ditz show 1209b
/home/matt/checkouts/ditz/lib/ditz/model-objects.rb:124:in `sort_by': comparison of String with Time failed (ArgumentError)
from /home/matt/checkouts/ditz/lib/ditz/model-objects.rb:124:in `assign_issue_names!'
from /home/matt/checkouts/ditz/lib/ditz/model-objects.rb:51:in `issues='
from /home/matt/checkouts/ditz/lib/ditz/file-storage.rb:21:in `load'
from /home/matt/checkouts/ditz/bin/ditz:165

I compared my dumped file to the original ruby file and found that python wrote dates out like this:

>>> print yaml.dump(issue.creation_time)
2008-09-02 17:47:43.268059

But in the ruby yaml files, the dates look like this:

$ grep creation_time ../../../.ditz/issue-1209b17b64335383a710ccadf10b74c3401dbcb2.yaml
creation_time: 2008-09-02 17:47:43.268059 Z

That trailing Z seems important.

Also, python and ruby seem to write out lists differently. Here’s how python dumped a list of lists:

>>> print yaml.dump(issue.log_events)
- [!!timestamp '2008-09-02 17:47:44.549355', Matthew Wilson , created,
- [!!timestamp '2008-09-02 19:18:03.286902', Matthew Wilson , assigned
to release 3.5.1 from unassigned, '']
- [!!timestamp '2008-09-04 18:27:19.571991', Matthew Wilson , unassigned
from release 3.5.1, '']

But the same data dumped by ruby looks like:

- - 2008-09-02 17:47:44.549355 Z
- Matthew Wilson
- created
- ""
- - 2008-09-02 19:18:03.286902 Z
- Matthew Wilson
- assigned to release 3.5.1 from unassigned
- ""
- - 2008-09-04 18:27:19.571991 Z
- Matthew Wilson
- unassigned from release 3.5.1
- ""

So, there’s clearly some more work for me (or you, this is an open-source project) to do.

Some progress on pitz

Pitz is my open-source, distributed, plain-text, command-line, very flexible issue tracker. I just started working on it and I’m looking for feedback and contributors.

This post describes a little of what I’ve gotten done so far. I haven’t written any interface yet other than from within an interactive python session. Deal with it.

Getting started

You have to make a bag to keep all the tasks and then you make tasks with any keywords you can dream up.

>>> from pitz import Bag, Task
>>> b = Bag('.pitz')
>>> b.append(Task(title='Wash the dishes', creator='Matt', importance='Not very'))
>>> b.append(Task(title='Clean the cat box', creator='Matt', importance='Not very'))

You can add tasks to the bag by using the append method on the bag, or by passing in the bag as the first argument to the task:

>>> Task(b, title='Set new high score on Sushi-go-round', creator='Matt', importance='critical')

Printing a bag really prints the title of each task in the bag:

>>> print(b)
Clean the cat box
Set new high score on Sushi-go-round
Wash the dishes

Running queries

I’m proud of this one. This is the main reason I’m working on ditz. Bag instances have a feature called matching_pairs that can filter the tasks down to a smaller new bag.

You can filter by a single pair like this:

>>> critical_tasks = b.matching_pairs([('importance', 'critical')])
>>> print(critical_tasks)
Set new high score on Sushi-go-round

Or you can filter by multiple pairs and the filtered tasks must satisfy ALL the pairs.

>>> print(b.matching_pairs([('creator', 'Matt'), ('importance', 'Not very')]))
Clean the cat box
Wash the dishes

Dumping and loading

Use the bag that holds all the tasks to quickly write all the tasks out to your hard drive like this:

>>> b.to_yaml_files()
['.pitz/task-07e1af97-0ac6-4904-9187-0c2fd61692b6.yaml', '.pitz/task-6a7af07c-d0fb-4a77-9347-8dc78ef490fe.yaml', '.pitz/task-5ce725dc-c1db-4eca-a74c-55cd0e910786.yaml']

The returned stuff is a list of files that pitz just wrote.

Loading from the hard drive is pretty simple too. Just tell the bag where to load from.

>>> b2 = Bag('.pitz')
>>> print(b2)
Clean the cat box
Set new high score on Sushi-go-round
Wash the dishes

Task details

Printing a task by itself gives all the details on the task.

>>> t = Task.from_yaml_file ('.pitz/task-4d9c1db2-fef3-4b50-8095-b2339384e118.yaml')
>>> print(t)
Do the 2008 taxes

type: Task
name: task-4d9c1db2-fef3-4b50-8095-b2339384e118
title: Do the 2008 taxes
created date: 2009-03-01 22:29:58.242512
modified date: 2009-03-01 22:29:58.242512
creator: Matt
last modified by: Matt

Do the 2008 taxes

Other stuff

You can use bags as iterators to go through the tasks one-by-one:

for task in b:

Also notice that tasks are really just subclassed dictionaries (UserDict, actually) with some extra methods bolted on.

My new ticket tracking system is now vaporware!

I set up http://pitz.tplus1.com to host my pitz project, which is a python implementation of ditz.

Instead of just banging out code, I decided to write the documentation and the list of supported features FIRST.

Once I have my feature set established, then I’ll write the tests for those features, and finally, I’ll write the code.

I’m looking for feedback on the feature set. What would an ideal bugtracker look like?