Dreaming in Code

Scott Rosenberg

Rosenberg is one of the co-founders of the online magazine Salon.com, a magazine I’ve been reading on and off since 2000. After having a bad experience with internal software development, he became interested in how, after more than 50 years experience, we still find, in the words of Donald Knuth, that ‘software is hard.’

In the interests of disclosure, I will point out that I got a copy of this book for free from the author after he offered copies to bloggers if they agreed to mention the book. So, this is me (gratefully, happily) fulfilling my end of the deal.

This book is a story, a story of a collection of great developers attempting to write an amazing piece of software. It is not a retrospective story either. It begins shortly after the project has started, but unfortunately couldn’t follow the project to completion: the project still isn’t done yet.

The Open Source Applications Foundation (OSAF) was formed by Mitch Kapor (of Lotus 1-2-3 fame) to build a new form of personal information management software: Chandler. Dreaming in Code tells the story of how they tried and (so far) failed. With many informative diversions into the theory of software engineering in an effort to discover why software seems to so consistently be delivered late (if at all) and buggy (if it even gets close to meeting the original vision.)

Rosenberg does an impressinve amount of research into the theory. No appropriately read professional software engineer will find any revelations here: this is all stuff you should already know. But Rosenberg is not claiming to invent or discover anything new. In fact, he goes out of his way to disclaim that there is any original research contained in this book. He is a journalist, and as a journalist he has produced a detailed literature survey. The summary in two well-written chapters is useful even for experienced software engineers. I’m sure non-software engineers will find this all very interesting; assuming they are interested in how software gets written.

For software engineers, there is something very interesting here. The internal mechanics of a a team building a piece of software is a very secret thing. Companies are secretive and companies and open source projects want to protect their reputations. Most software engineers only ever see how a project they are working on proceeds, and then they’re too close, plus there’s no nice summary of what happened. Dreaming in Code is something very valuable to our field: an accurate story of the human side of a software development project. With both the clarity of distance and the accuracy of events recorded at the time they happened.

It was simply astounding how familiar this story was. OSAF and Chandler get some things spectacularly wrong, but then in other cases they do things very right. It’s easy to point at the things they stuffed up and claim that you would never make those mistakes, but it’s a little too easy to ignore the things that were done right. The good decisions end up forgotten and never noticed.

So what did I think that got wrong? Firstly, and by far the biggest: Analysis Paralysis. This is a mistake that I’ve seen projects make over and over again. In fact, I’d go as far as to say that it’s more common to see this affliction than not. Projects just can’t seem to make a decision, stick to it and then start building. The fear of the future locks them solid: what if we make the wrong decision? What if someone blames me for the wrong decision? In the end, ikf the decision wsas so egregiously wrong that it can be traced back to just one person, then everyone else around at the same time is just as culpable for allowing that decision to happen. Everyone, please just get in the habit of making decisions. Rely on those around you to spot a bad direction: that’s what they’re there for.

Second big mistake for Chandler? People. No surprises there, it’s the big and obvious disaster. If you believe that software is hard and you care about your software then clearly you should only work with the best. That’s easy to say, but pretty hard to achieve. And in Chandler’s case the people problem exhibited in a couple of interesting ways. Before I talk about the people problems that I saw Chandler as having I need to say that any attempt to judge is based solely on what is described in this book. I (obviously) didn’t work on this project, I don’t know the people, there’s only so much I can say. Having said that, I’d also like to say that people problems kill most projects and if we hope to advance we need to get over this fear of talking about the problems with people. Maybe then we can find some solutions. Or maybe that’s just an overly analytical geek talking. Why can’t everyone just be a nice reducable puzzle, dammit?

There was a permanent employee of the OSAF who was hired quite early and ended up in quite a senior technical position, and this employee was unfortunately the very soul of Analysis Paralysis. Any story about some interminable technical discussion has this particular employee at its heart. He was extraordinarily conservative and wary of making decisions, but ultimately many of the technical decisions came down to him. After a few of these stories, I was left shouting ‘Do something about him!’ at the book. And I’ve seen precisely this problem face-to-face too many times to ignore. Once again people, make decisions! You probably won’t get it too far wrong. In fact, in this case, the inability to make a decision led to one of their few definite cases of over-engineering.

The second people problem: OSAF planned to run themselves as an open source project that encouraged volunteers. Predictably enough, the only people who could volunteer for any extended period tended to be ex-Apple and -Netscape employees who had already made their fortune and no longer needed to work for an income. From the outside these volunteers presented an interesting problem. They were all brilliant and had done amazing work in the past, but they didn’t see this project as any kind of meal ticket. There was no drive for them to get this project finished and out the door. In the end, they come across as people partially on the side-lines, commenting endlessly but never really pushing the project forward. Instead, they were offering endless advice on how things could be done better. Software projects are always cursed with people like this; encouraging volunteers just seems to guarantee it.

But back to the book, I think I’ve already made clear that I really enjoyed it. I also think the opportunity to see inside another project is infinitely valuable for software engineers and software engineering. Aside from all that, I will say that sometimes the brief newspaper style of the book was a little irritating. On occasion I felt a topic or anecdote could have done with some more depth before moving on.

Software engineers will get a lot out of this; non-software engineers who care about how software (upon which, our civilisation is built) is written will get even more out of this. Oh, and as a professional programmer and wannabe-amateur writer-slash-blogger, software is definitely written - then organically edited into shape.

To finish on a positive note, something Chandler got right? Python. Choosing to write their software in an expressive high level language was clearly a win. There is no question now that Python is fast enough for desktop software, and that’s really the only doubt. Hopefully Chandler can be used as an example of choosing better languages.

If you choose to read this, or not, in the end people! Make a decision!