Earlier this week I stumbled upon the unix-jun72 project which is a restoration of the 1st edition of UNIX based on a scanned printout of the original source.  Unfortunately not all the source is available, but some userland binaries and the C compiler have been recovered from some tapes and there is enough functionality to be able to boot to a minimal and usable system.

The first step is to download and build the most excellent Simh simulator so that we can simulate a PDP-11.   Fortunately the disk images are already pre-built and downloadable, so one just has to download these and point the PDP-11 simulator at a config file and the system almost instantly boots.  The full instructions are provided in great detail here.

I am rather fond of the PDP-11. It was the first mini computer I worked on back in 1987 when I was a very junior programmer at Prosig.  The machine was running RSX-11M Plus rather than UNIX but even so it gave me the appreciation of what one can do on a very small multi-user machine. At the time I was maintaining, building and testing DATS signal processing tools written in Fortran 77, and some of the work involved tinkering with task builder scripts to try and cram code into the very limited memory.

So in those days size really mattered.  Small was considered beautiful and this mindset was instilled into me in my formative years.  So I was very pleased to find the unix-jun72 project which allows me to relive those PDP-11 days and also experience UNIX-1.

Since some of the original source code still exists, one can browse the source of the kernel and some of the core utilities.  It is amazing to see the functionality that is available in very small binaries.

Let's  fire up the simulator and see the size of the 'cat' command:

 PDP-11 simulator V3.9-0  
 Disabling CR  
 Disabling XQ  
 RF: buffering file in memory  
 TC0: 16b format, buffering file in memory  
 :login: root  
 root  
 # ls -al /bin/cat  
 total  1  
  50 sxrwr- 1 bin   134 Jan 1 00:00:00 cat  

 And how does that compare to a GNU cat on a 64 bit Ubuntu laptop?

 ls -al /bin/cat  
 -rwxr-xr-x 1 root root 47912 Nov 28 12:48 /bin/cat  

134 bytes versus 47912 bytes. That is quite a difference!  Admittedly we are comparing apples vs pears here, so obviously this is an unfair comparison, but it does illustrate how we've "progressed" since the early UNIX-1 days.   Honestly I'm glad we can now write userland tools and the kernel in C rather than assembler and not have to worry about size constraints quite so much.

I'm very glad that this project exists to preserve the UNIX heritage.  History has a lot to teach us.  Browsing the early code is an education; it allows us to appreciate the memory constraints that shaped the design and implementation of UNIX.  To be able to run this code in a simulator and tinker with a running system adds to the experience.

We've moved on a long way in 40 or so years, machines are incredibly powerful, and memory and disk is cheap.  But let us not forget the "small is beautiful" ideal.  There is something very appealing about tools that do just enough to be very useful and yet are not bloated with unnecessary memory hogging feature creep.

Read more

It’s been a decade ago when I did my first steps with contributing to Free Software, about seven years when I joined Debian, and about 6 with Canonical and Ubuntu. Time for some reflection what I have done over these years!

Distribution Packaging and Maintenance

My first sponsored Debian upload ever was cracklib2, which seriously needed some love and was looking for a new maintainer. So in that upload I managed to close all outstanding bugs. Thanks to my mentor Martin Godisch about providing a lot of guidance for this!

Since then I’ve maintained various packages, where the most popular ones are certainly the free database server “PostgreSQL” (see next section) and the e-book management software “Calibre”.

“Maintaining” by and large means “making it really easy to get and use this software”. This decomposes to:

• Packaging it in a way that a simple apt-get install makes the software work out of the box (as far as possible)
• Provide a default configuration/customizations so that it integrates and plays well with the rest of the system; this includes the paths and permissions for log files, log rotation, debconf, configuration file standards, etc.
• Be the front line for bug reports from users, sort, answer, and de-duplicate them, and either fix them myself, or forward useful bugs to upstream.
• Providing security updates for stable releases
• To some extent, help with the development of the software; this gets mostly driven by user demand, and of course my personal interests.

In August 2004 I got employed by Canonical to work full time on Ubuntu, which pretty much turned a hobby into a profession. I never regretted this in the past years, it’s an awesome job to do!

In principle I’m doing the same thing in Ubuntu as well: Bring stuff from developers to the people out there. Except with a different focus, in Ubuntu my daily bread and butter is the GNOME desktop and stuff around (and immediately below) it. And even though after a long day of bug triaging and debugging I feel a bit low-hearted (“50.000 bugs away from perfection”), when I take a step back and see how much the usage of Free Software in the world has grown since 2004, I am very proud of being part of Ubuntu, which certainly has its fair contribution and share in this growth. So what seemed like a crazy idea from Mark back in 2004 actually has made a remarkable progress.

In the beginning of Ubuntu I mostly sent back patches to the Debian bug tracker, but this evolved quite a bit on both sides: These days I try to keep “my” packages in sync and commit stuff directly to Debian, which works very well with e. g. the pkg-utopia team, which is responsible for HAL, udisks, upower, PolicyKit, and related packages. At this point I want to thank Michael Biebl for being such an awesome guy on the Debian side! Also, it seems that Debian has moved a fair bit away from the strong “Big Maintainer Lock” towards team based maintenance, so these days it is easier than ever to commit stuff directly to Debian for a lot of packages, without much fuss.

PostgreSQL

I have done a handful of changes to PostgreSQL, but these mostly concerned easy packaging and crash fixes, nothing out of the ordinary. I’m not really a PostgreSQL upstream developer.

The thing I am really proud of is the postgresql-common package, which is a very nice example what a distro can provide on top of upstream: If you install the upstream tarball, you have to manually care for creating clusters, providing a sensible configuration for them, set up SSL, set up log rotation, etc. With postgresql-common, this is all done automatically. The biggest feature it provides is a robust and automatic way of upgrading between major releases with the pg_upgradecluster tool, which takes care of a dozen corner cases and the nontrivial process of dumping the old cluster and reloading the new one. Also, you can effortlessy run several instances of the same version in parallel, so that you can e. g. have a production and a development instance, or try the new 9.0 RC1 while still running the 8.4 production one. (more details)

Crash Reporting with Apport

This has been a pet peeve of mine pretty much from day 1. Back in the old days, crashes in software were a pain to track down: many crashes are hard to reproduce, it takes ages to get useful information from bug reporters, and a lot of data cannot be recovered any more when you try to reproduce and analyze a crash after the fact.

With the growing demand for QA from both Canonical and Ubuntu, in 2006 I finally got some time to start Apport, which would make all this a lot easier: It intercepts crashes as they happen, collects the data that we as a developer need, and makes it very easy for the user to submit them as a bug report. This is accompanied by a backend service (called “retracers”) which would reprocess the bug reports by taking the core dump, reproducing a chroot with the packages and versions that the reporter had, installing the debugging symbols, and re-running gdb, to produce a fully symbolic stack trace.

See this bug report for how this looks like. Since then, tons of crashes were fixed, way more than we could ever have done “the old way” with asking users to rebuild with “-g -O0″, running gdb, etc.

By today, Apport has grown quite a bit: rich bug reports, per-package hooks, automatic duplication of crashes, interactive GUI elements in hooks, etc.

Plumbing Development

Handling hotpluggable hardware has always interested me, since the day when I got my first USB stick and it was ridiculously hard (from an user perspective) to use it:

    $ su -
    # mount -t vfat -o uid=1000 /dev/sda1 /mnt

My first go at this was to write pmount which would allow normal users to mount hotpluggable storage without root privileges and worrying about mount paths and options, and then integrate it into GNOME and HAL. Personally I abandoned it years ago, but it seems other people still use it, so I’m glad that Vincent Fourmond took over the maintenance now.

Since then, the entire stack evolved quite a bit: HAL grew to something useful and rather secure, and finally into some monstrous unmaintainable beast, which is why it was declared dead in 2008, and replaced with the “U” stack: udev, udisks, upower, etc. I enjoy hacking on that a lot, and since it’s part of my Desktop Team Tech Lead/Developer role in Canonical, I can spend some company time on it. so far I worked on bug fixes, small new features, and writing a rather comprehensive test suite for udisks (see my udisks commits so far). I also did a fair share of porting stuff away from HAL to the new stack, including some permanent commitments like maintaining the keymaps in udev.

GNOME

Before I started with Canonical I was never much of a GUI person: I was fully content with fvwm and a few xterms around it. But as an Ubuntu developer I do dogfooding, and thus I switched to GNOME for my day to day work. It didn’t take long before I really fell in love with it!

Similar to my Debian packages, my upstream involvement with GNOME is mostly integration and bug fixing. As already explained, we get a looooot of bug reports, so my focus is mostly on bug fixing. To date, I sent 93 patches to bugzilla. Since January 2010 I became a committer, so that it’s easier for me to get patches upstream.

Debugging problems and fixing bugs is a pretty tedious task, but I still enjoy the rewarding nice feeling when you finally tracked down something and can close a bug with 50 duplicates, and you have made people’s life a little bit easier from now on.

Read more