RIP Natchie

We adopted her in 2001 when Lindsey and I lived in Washington DC. Before that, Natchie lived with an economist I worked with, and I used to take care of Natch and another cat and a golden retriever occasionally.

I still remember one time when Lindsey and I were watching TV while I was pet-sitting. Natchie walked by the TV, looked at us, meowed, and then kept on walking. Was really endearing.

Natch loved affection more than any cat I’ve known. She would softly head-butt people to get them to pet her. She would hassle other animals to pet her. She loved to curl up next to the golden retriever and he would lick her.

Natch got her name from an Ethiopian phrase “Letenatchie!” That’s a drinking toast. Natch was an Absynnian, and the breed was supposedly from the area that is now known as Ethiopia.

It’s a gloomy day at the Wilson house today. We’ll miss her.

How to load ditz issues into python

Ditz is a fantastic distributed bug tracking system written in Ruby.

Here’s some code that you can use to load some ditz issues into a python interpreter. You need to install my pitz project first though.


>>> from pitz.junkyard.ditzloader import *
>>> from glob import glob
>>> issue_file_path = glob('../.ditz/issue-*.yaml')[0]
>>> import yaml
>>> issue = yaml.load(open(issue_file_path))
>>> issue.title
'Distribute reports by email'
>>> print issue.desc
Somehow allow people to sign up for report subscriptions.


So when new reports come out, they get updated. Maybe I can use
RSS feeds to hold the reports.
>>> print issue.log_events
[[datetime.datetime(2008, 9, 2, 17, 47, 44, 549355), 'Matthew Wilson ', 'created', ''], [datetime.datetime(2008, 9, 2, 19, 18, 3, 286902), 'Matthew Wilson ', 'assigned to release 3.5.1 from unassigned', ''], [datetime.datetime(2008, 9, 4, 18, 27, 19, 571991), 'Matthew Wilson ', 'unassigned from release 3.5.1', '']]

That above just showed how to read the ditz issue. I’m having trouble updating the issue and then saving it in a format that ditz can still read. Updating the ditz issue and saving it out again is easy:


>>> issue.title = 'Distribute reports by email or RSS'
>>> open(issue_file_path, 'w').write(yaml.dump(issue))

But when I try to load it with ditz, I get this error:

$ ditz show 1209b
/home/matt/checkouts/ditz/lib/ditz/model-objects.rb:124:in `sort_by': comparison of String with Time failed (ArgumentError)
from /home/matt/checkouts/ditz/lib/ditz/model-objects.rb:124:in `assign_issue_names!'
from /home/matt/checkouts/ditz/lib/ditz/model-objects.rb:51:in `issues='
from /home/matt/checkouts/ditz/lib/ditz/file-storage.rb:21:in `load'
from /home/matt/checkouts/ditz/bin/ditz:165

I compared my dumped file to the original ruby file and found that python wrote dates out like this:

>>> print yaml.dump(issue.creation_time)
2008-09-02 17:47:43.268059

But in the ruby yaml files, the dates look like this:

$ grep creation_time ../../../.ditz/issue-1209b17b64335383a710ccadf10b74c3401dbcb2.yaml
creation_time: 2008-09-02 17:47:43.268059 Z

That trailing Z seems important.

Also, python and ruby seem to write out lists differently. Here’s how python dumped a list of lists:

>>> print yaml.dump(issue.log_events)
- [!!timestamp '2008-09-02 17:47:44.549355', Matthew Wilson , created,
'']
- [!!timestamp '2008-09-02 19:18:03.286902', Matthew Wilson , assigned
to release 3.5.1 from unassigned, '']
- [!!timestamp '2008-09-04 18:27:19.571991', Matthew Wilson , unassigned
from release 3.5.1, '']

But the same data dumped by ruby looks like:

log_events:
- - 2008-09-02 17:47:44.549355 Z
- Matthew Wilson
- created
- ""
- - 2008-09-02 19:18:03.286902 Z
- Matthew Wilson
- assigned to release 3.5.1 from unassigned
- ""
- - 2008-09-04 18:27:19.571991 Z
- Matthew Wilson
- unassigned from release 3.5.1
- ""

So, there’s clearly some more work for me (or you, this is an open-source project) to do.

Some progress on pitz

Pitz is my open-source, distributed, plain-text, command-line, very flexible issue tracker. I just started working on it and I’m looking for feedback and contributors.

This post describes a little of what I’ve gotten done so far. I haven’t written any interface yet other than from within an interactive python session. Deal with it.

Getting started

You have to make a bag to keep all the tasks and then you make tasks with any keywords you can dream up.


>>> from pitz import Bag, Task
>>> b = Bag('.pitz')
>>> b.append(Task(title='Wash the dishes', creator='Matt', importance='Not very'))
>>> b.append(Task(title='Clean the cat box', creator='Matt', importance='Not very'))

You can add tasks to the bag by using the append method on the bag, or by passing in the bag as the first argument to the task:

>>> Task(b, title='Set new high score on Sushi-go-round', creator='Matt', importance='critical')

Printing a bag really prints the title of each task in the bag:


>>> print(b)
Clean the cat box
Set new high score on Sushi-go-round
Wash the dishes

Running queries

I’m proud of this one. This is the main reason I’m working on ditz. Bag instances have a feature called matching_pairs that can filter the tasks down to a smaller new bag.

You can filter by a single pair like this:


>>> critical_tasks = b.matching_pairs([('importance', 'critical')])
>>> print(critical_tasks)
Set new high score on Sushi-go-round

Or you can filter by multiple pairs and the filtered tasks must satisfy ALL the pairs.

>>> print(b.matching_pairs([('creator', 'Matt'), ('importance', 'Not very')]))
Clean the cat box
Wash the dishes

Dumping and loading

Use the bag that holds all the tasks to quickly write all the tasks out to your hard drive like this:


>>> b.to_yaml_files()
['.pitz/task-07e1af97-0ac6-4904-9187-0c2fd61692b6.yaml', '.pitz/task-6a7af07c-d0fb-4a77-9347-8dc78ef490fe.yaml', '.pitz/task-5ce725dc-c1db-4eca-a74c-55cd0e910786.yaml']

The returned stuff is a list of files that pitz just wrote.

Loading from the hard drive is pretty simple too. Just tell the bag where to load from.

>>> b2 = Bag('.pitz')
>>> print(b2)
Clean the cat box
Set new high score on Sushi-go-round
Wash the dishes

Task details

Printing a task by itself gives all the details on the task.


>>> t = Task.from_yaml_file ('.pitz/task-4d9c1db2-fef3-4b50-8095-b2339384e118.yaml')
>>> print(t)
Do the 2008 taxes
-----------------

type: Task
name: task-4d9c1db2-fef3-4b50-8095-b2339384e118
title: Do the 2008 taxes
created date: 2009-03-01 22:29:58.242512
modified date: 2009-03-01 22:29:58.242512
creator: Matt
last modified by: Matt

description:
Do the 2008 taxes

Other stuff

You can use bags as iterators to go through the tasks one-by-one:

for task in b:
print(t['title'])

Also notice that tasks are really just subclassed dictionaries (UserDict, actually) with some extra methods bolted on.

My new ticket tracking system is now vaporware!

I set up http://pitz.tplus1.com to host my pitz project, which is a python implementation of ditz.

Instead of just banging out code, I decided to write the documentation and the list of supported features FIRST.

Once I have my feature set established, then I’ll write the tests for those features, and finally, I’ll write the code.

I’m looking for feedback on the feature set. What would an ideal bugtracker look like?

My thoughts on roles in software teams

My biz is growing. We’re going from a state where I did everything all by myself to a “real” company. I’ve been working on a mental model to define what developers do.

I’ve boiled it down to three different kinds of work: engineering, consulting, and management. In any real shop, individuals have to be able to wear all three hats depending on circumstances.

Engineering

A great engineer builds real stuff out of blueprints.

The keyword for engineers is execution. They’re the ones that actually get shit done, and do it brilliantly.

Bad engineers

An engineer that doesn’t love building things is not going to be a good engineer. A bad engineer hacks shit together then hides from peer review.

Good engineers

A good engineer spends her free time reading manuals about obscure technologies. A good engineer gets physically ill when seeing bad design. A good engineer builds beautiful stuff and obsesses about skill mastery.

When a new problem comes up, a good engineer builds a tool that can be used to solve other problems.

Consulting

This is the task of figuring out the next big thing. This takes a lot more work than just talking to customers and writing down what they say.

Good consultants

A great consultant is a lot like an investigative journalist. They know how to keep asking questions until they know exactly what really needs to be built. Then they write down what they figured out from all those questions in plain language, so the engineer can start working.

Good consultants go beyond transcribing what the customer said and figure out what the customer really needs. Great consultants are able to synthesize product ideas from unordered, generalized customer complaints.

Managers

Good engineers and good consultants free the manager from doing anything but planning for sustainable growth. A manager’s single responsibility is to keep the system running over the long run. Managers analyze performance problems and figure out what to do; usually the decision is a choice between training somebody or firing that somebody.

A manager makes sure that her team can handle a loss of a few engineers or consultants as people move around, get new jobs, get hit by buses, or win the lottery.

Good managers

A good manager is really the mom of the organization. She makes sure everyone is happy, working on interesting projects, and generally, reaching their potential. Good managers make sure that as people inevitably leave the team, the team can still function at high capacity.

git cherry is neat

Sometimes I’ll patch a production bug in my production branch and then I will forget to merge that commit into the development branch. This is how I can check for that:

$ git checkout production_branch
$ git cherry dev_branch

This will spit out a list of commits that are in production_branch but not in dev_branch. It will not return any commits made to dev_branch but not in production_branch. It is not the same as a diff of the two branches either.

I got hacked

So, I’ve been ignoring all the messages to upgrade from WordPress 2.5. And then I got some phishing php code embedded into my server.

Then I rebooted, and the whole filesystem was gone.

So I lost the last few posts. Maybe I can rebuild them. Maybe I can’t. In either scenario, I probably won’t.

Posted in web

Now I have two human children

At 5 PM, Lindsey started feeling contractions, at 6:30 PM we were driving to MetroHealth hospital, and by 8:11 PM, Oliver was born. Lindsey and O. are both recovering very well. He shows a lot of Wilson family traits: really long toes, disdain toward waking up, and psychokinesis.

Lots of pictures forthcoming, but probably on flickr or Facebook or something else.

Ditz versus bugs everywhere

A few months ago, I sketched out a ticket-tracking system that would be married with my source code. Then some commenters told me about bugs everywhere (be) and ditz.

I’ve looked at both, but I’ve been using ditz full-time while just watching be. Anyhow, here’s a few comparisons:

setting up a project

Here’s what it looks like when you set up a project in ditz:

$ ditz init
I wasn't able to find a configuration file ./.ditz-config.
We'll set it up right now.
Your name (enter for Matthew Wilson):
Your email address (enter for [email protected]):
Directory to store issues state in (enter for .ditz):
Use your text editor for multi-line input when possible (y/n)? y
Paginate output (always/never/auto)? auto
Project name (enter for scratch):
Issues can be tracked across the project as a whole, or the project can be
split into components, and issues tracked separately for each component.
Track issues separately for different components? (y/n): y

Current components:
None!

(A)dd component, (r)emove component, or (d)one: a
Component name: documentation

... snip ...

(A)dd component, (r)emove component, or (d)one: d
Ok, .ditz directory created successfully.

And here’s how you can create a single issue.

$ ditz add
Title: Write something justifying yet another web framework
Is this a (b)ugfix, a (f)eature, or a (t)ask? t
Choose a component:
1) scratch
2) documentation
3) model code
4) controller code
5) view code
Component (1--5): 2
Issue creator (enter for Matthew Wilson ):
Added issue documentation-1 (e8a4a43f78ee83300cc0372a13375d9534b97abb).

You can’t tell, but when I punched in the title, ditz opened my $EDITOR and I wrote a longer description in there.

Now the same thing in be:

$ be set-root
Guessing id 'matt '
No revision control detected.
Directory initialized.

$ be new 'Write something justifying yet another web framework'
Guessing id 'matt '
Guessing id 'matt '
Created bug with ID 4d4

Not quite the same experience!

Here’s what a ditz issue looks like:

$ ditz show documentation-1
Issue documentation-1
---------------------
Title: Write something justifying yet another web framework
Description: Why not just polish any of the ones already out there?
Type: task
Status: unstarted
Creator: Matthew Wilson
Age: four minutes
Release:
References:
Identifier: e8a4a43f78ee83300cc0372a13375d9534b97abb

Event log:
- created (matt, four minutes ago)

And in be:

$ be show 4d4
Guessing id 'matt '
ID : 4d4e6a17-2097-42bb-a3cd-3c17566ecce8
Short name : 4d4
Severity : minor
Status : open
Assigned :
Target :
Creator : matt
Created : Mon, 22 Dec 2008 20:25 (Tue, 23 Dec 2008 01:25:04 +0000)
Write something justifying yet another web framework

Ditz issues have titles, long descriptions, types (feature, bugfix, or task), releases (optionally) and links to components (also optionally). There are ditz plugins to add support for assigning issues to people.

be has most of the same concepts, just with different names.

data serialization and storage

ditz makes a .ditz directory at the top of a project and be makes a .be directory in the top of the project.

Inside the .ditz folder, there’s one project.yaml file that lists releases (groupings of issues) and components (also groupings of issues, but cross-cutting). Then each issue lives in its own yaml file, and they look like this:

$ cat .ditz/issue-ac3177b3bf8c6757625977ef27279c1fe05df662.yaml
--- !ditz.rubyforge.org,2008-03-06/issue
title: Write some "WHY?" documentation
desc: Justify the existence of this project.
type: :task
component: documentation
release:
reporter: Matthew Wilson
status: :unstarted
disposition:
creation_time: 2008-12-23 00:59:05.840956 Z
references: []

id: ac3177b3bf8c6757625977ef27279c1fe05df662
log_events:
- - 2008-12-23 00:59:05.841349 Z
- Matthew Wilson
- created
- ""
- - 2008-12-23 01:08:58.605955 Z
- Matthew Wilson
- commented
- |-
Yeah, if you're gonna build another web framework, this needs to be
really good.

Meanwhile, be is fairly similar, but bugs get whole directories to themselves. be uses what seems to be a home-made plain-text format for storing bugs:

$ cat .be/bugs/4da8ee85-9353-4a92-a654-8510bb8be0d0/values

creator=matt

severity=minor

status=open

summary=Write some "WHY?" documentation

time=Tue, 23 Dec 2008 01:12:13 +0000

There’s actually much more whitespace than that. I replaced the eight blank lines between each line of text with just two blank lines.

While ditz stores the comments inside the issue’s yaml file, be makes a directory under the issue’s directory, and then stores the text of the comments in one file and the information about who said it in a separate file.

The community

The ditz mailing list is really active with people debating ideas for new features. The be mailing list is now showing some signs of life after looking dead in August.

What ditz has that be lacks

ditz can make really pretty HTML pages for all the issues for a project. example.

yaml was a really good choice. yaml makes it easy to deserialize to higher objects than just crappy boring primitive types like arrays. Instead, you can hop all the way to your own weird home made objects by specifying a tag. Then all the stuff in the yaml file gets passed in to your object.

Ditz has lots and lots of commands that are only on the be roadmap. You can search your issues with regular expressions with ditz grep, you can claim issues for yourself, you can group issues by releases and components, etc, etc, etc.

The ditz issue data model can be extended with plugins. Like I mentioned earlier, one plugin makes it possible for people to claim issues as assigned to them.

What I like about be

It’s written in python. I hate to feed the python snobbery monster, but there are certain python niceties that I don’t like doing without. In particular, ipython is just too awesome. When I read the ditz code, I spent most of my time navigating the code to get to the part that I cared about that. With ipython, I don’t have that problem. I just hit foo?? and immediately see the source code.

And ruby’s documentation is not what I’ve grown accustomed to with python. For comparison:

I think the Python docs have more explanatory text in just the table of contents.

In addition, ditz uses a lot homemade code: there’s a homemade option parser library (trollop), a homemade hack on the way ruby stores data files so that all the HTML templates are available, and all sorts of gymnastic FP tricks to get a lot of shit done in a very small number of lines. That’s cool, but as a yellow-belt in Ruby, it is really @#$ing hard to make any contributions to this project. Here’s some code that I find a little difficult to read:

def operation method, desc, *args_spec, &options_blk
@operations ||= {}
@operations[method] = { :desc => desc, :args_spec => args_spec,
:options_blk => options_blk }
end

operation :stop, "Stop work on an issue", :started_issue do
opt :comment, "Specify a comment", :short => 'm', :type => String
opt :no_comment, "Skip asking for a comment", :default => false
end

def stop project, config, opts, issue
puts "Stopping work on issue #{issue.name}: #{issue.title}."
issue.stop_work config.user, get_comment(opts)
puts "Recorded work stop for #{issue.name}."
end

After tracing through a few hundred lines of stuff like that, I usually get discouraged and just write a feature request rather than a patch.

In summary

I like ditz. I like reading nearly inscrutable Ruby code to see how wacky people solve problems. My experience with ditz so far has been about an A minus, which is pretty good!

Why I’m going to write my own

There have been a few times where the lack of a proper database system has bit me. Like when I renamed a release, I had to do some searching and replacing in lots and lots of files. Also, regenerating my HTML views is taking almost two minutes now that I have so many issues. Also, certain operations, like moving a handful of issues from one release to another, or searching for intersections of issue subsets, are trickier than what they should be.

Besides all that, I’m fascinated by couchdb, and I think this would be a good use.

I think my system is going to use a local couchdb server that loads in all the issues from local yaml files into the server on startup. Then after lots of work updating, I’ll write out all the issues back into yaml. So, when you update your checkout of your code, you’ll need to restart or reload your couchdb server. Then you can use the couchdb server to work with the system, and then at the end, re-serialize the data back out to JSON, and then to yaml.

ditz and be are sort of like old-school CGI web apps where each user action has to start up some the framework, do the action, then tear down. My system will instead keep all the issue data in memory and require explicit startups and shutdowns.