Things just working

I have a Macbook with a bcm4331 wireless chip that has not been supported in Linux. The driver was added to kernel 3.2. I was anxious to test this when I upgraded to precise.

After the update there was no net connection. The network indicator said “Missing firmware”. So I scoured the net and found the steps necessary to extract the firmware file to the correct directory.

I typed the command and pressed enter. That exact second my network indicator started blinking and a few seconds later it had connected.

Without any configuration, kernel module unloading/loading or “refresh state” button prodding.

It just worked. Automatically. As it should. And even before it worked it gave a sensible and correct error message.

To whoever coded this functionality: I salute you.

More uses for btrfs snapshots

I played around with btrfs snapshots and discovered two new interesting uses for them. The first one deals with unreliable operations. Suppose you want to update a largish SVN checkout but your net connection is slightly flaky. The reason can be anything, bad wires, overloaded server, electrical outages, and so on.

If SVN is interrupted mid-transfer, it will most likely leave your checkout in a non-consistent state that can’t be fixed even with ‘svn cleanup’. The common wisdom on the Internet is that the way to fix this is to delete or rename the erroneous directory and do a ‘svn update’, which will either work or not. With btrfs snapshots you can just do a snapshot of your source tree before the update. If it fails, just nuke the broken directory and restore your snapshot. Then try again. If it works, just get rid of the snapshot dir.

What you essentially gain are atomic operations on non-atomic tasks (such as svn update). This has been possible before with ‘cp -r’ or similar hacks, but they are slow. Btrfs snapshots can be done in the blink of an eye and they don’t take extra disk space.

The other use case is erroneous state preservation. Suppose you hack on your stuff and encounter a crashing bug in your tools (such as bzr or git). You file a bug on it and then get back to doing your own thing. A day or two later you get a reply on your bug report saying “what is the output of command X”. Since you don’t have the given directory tree state around any more, you can’t run the command.

But if you snapshot your broken tree and store it somewhere safe, you can run any analysis scripts on it any time in the future. Even possibly destructive ones, because you can always run the analysis scripts in a fresh snapshot. Earlier these things were not feasible because making copies took time and possibly lots of space. With snapshots they don’t.

Fun stuff with btrfs

I work on, among other things, Chromium. It uses SVN as its revision control system. There are several drawbacks to this, which are well known (no offline commits etc). They are made worse by Chromium’s enormous size. An ‘svn update’ can easily take over an hour.

Recently I looked into using btrfs’s features to make things easier. I found that with very little effort you can make things much more workable.

First you create a btrfs subvolume.

btrfs subvolume create chromium_upstream

Then you check out Chromium to this directory using the guidelines given in their wiki. Now you have a pristine upstream SVN checkout. Then build it once. No development is done in this directory. Instead we create a new directory for our work.

btrfs subvolume snapshot chromium_upstream chromium_feature_x

And roughly three seconds later you have a fresh copy of the entire source tree and the corresponding build tree. Any changes you make to individual files in the new directory won’t cause a total rebuild (which also takes hours). You can hack with complete peace of mind knowing that in the event of failure you can start over with two simple commands.

sudo btrfs subvolume delete chromium_feature_x
btrfs subvolume snapshot chromium_upstream chromium_feature_x

Chromium upstream changes quite rapidly, so keeping up with it with SVN can be tricky. But btrfs makes it easier.

cd chromium_upstream
gclient sync # Roughly analogous to svn update.
cd ..
btrfs subvolume snapshot chromium_upstream chromium_feature_x_v2
cd chromium_feature_x/src && svn diff > ../../thingy.patch && cd ../..
cd chromium_feature_x_v2/src && patch -p0 < ../../thingy.patch && cd ../..
sudo btrfs subvolume delete chromium_feature_

This approach can be taken with any tree of files: images, even multi-gigabyte video files. Thanks to btrfs’s design, multiple copies of these files take roughly the same amount of disk space as only one copy. It’s kind of like having backup/restore and revision control built into your file system.

The four stages of command entry

Almost immediately after the first computers were invented, people wanted them to do as they command. This process has gone through four distinct phases.

The command line

This was the original way. The user types his command in its entirety and presses enter. The computer then parses it and does what it is told. There was no indication on whether the written command was correct or not. The only way to test it was to execute it.

Command completion

An improvement to writing the correct command. The user types in a few letters from the start of the desired command or file name and presses tab. If there is only one choice that begins with those letters, the system autofills the rest. Modern autocompletion systems can fill in command line arguments, host names and so on.

Live preview

This is perhaps best known from IDEs. When the user types some letters, the IDE presents all choices that correspond to those letters in a pop up window below the cursor. The user can then select one of them or keep writing. Internet search sites also do this.

Live preview with error correction

One thing in common with all the previous approaches is that the input must be perfect. If you search for Firefox but accidentally type in “ifrefox”, the systems return zero matches. Error correcting systems try to find what the user wants even if the input contains errors. This is a relatively new approach, with examples including Unity’s new HUD and Google’s search (though the live preview does not seem to do error correction).

The future

What is the next phase in command entry? I really have no idea, but I’m looking forward to seeing it.

Complexity kills

The biggest source of developer headache is complexity. Specifically unexpected complexity. The kind that pops out of nowhere from the simplest of settings and makes you rip your hair out.

As an example, here is a partial and simplified state machine for what should happen when using a laptop’s trackpad.

If you have an idea of what should happen in the states marked “WTF?”, do send me email.

What is worse than having a problem?

The only thing worse than having a problem is having a poor solution to a problem.

Why?

Because that prevents a good solution from being worked out. The usual symptom of this is having a complicated and brittle Rube Goldberg machine to do something that really should be a just simpler. It’s just that nobody bothers to do the Right Thing, because the solution that we have almost kinda, sorta works most of the time so there’s nothing to worry about, really.

Some examples include the following:

  • X used to come with a configurator application that would examine your hardware and print a conf file, which you could then copy over (or merge with) the existing conf file. Nowadays X does the probing automatically.
  • X clipboard was a complete clusterf*ck, but since middle button paste mostly worked it was not seen as an issue.
  • The world is filled with shell script fragments with the description “I needed this for something long ago, but I don’t remember the reason any more and am afraid to remove it”.
  • Floppies (remember those?) could be ejected without unmounting them causing corruption and other fun.

How can you tell when you have hit one of these issues? One sign is that you get one of the following responses:

  • “Oh, that’s a bit unfortunate. But if you do [complicate series of steps] it should work.”
  • “You have to do X before you do Y. Otherwise it just gets confused.”
  • “It does not do X, but you can do almost the same with [complicated series of steps] though watch out for [long list of exceptions].”
  • “Of course it will fail [silently] if you don’t have X. What else could it do?”
  • “You ran it with incorrect parameters. Just delete all your configuration files [even the hidden ones] and start over.”

If you ever find yourself in the situation of getting this kind of advice, or, even worse, giving it out to other people, please consider just spending some effort to fixing the issue properly. You will be loved and adored if you do.

Rooting basic infrastructure for fun and profit

You know how we laugh at users of some other OS’s for running random binary files they get from the Internet.

Well we do it as well. And instead of doing it on our personal machines, we do it on those servers that run our most critical infrastructure?

Here is a simple step by step plan that you can use to take over all Linux distributions’ master servers.

  1. Create a free software project. It can be anything at all.
  2. Have it included in the distros you care about.
  3. Create/buy a local exploit trojan.
  4. Create a new minor release of your project.
  5. Put your trojan inside the generated configure script
  6. Boom! You have now rooted the build machines (with signing keys etc) of every single distro.

Why does this exploit work? Because configure is essentially an uninspectable blob of binary code. No-one is going to audit that code and the default packaging scripts use configure scripts blindly if they exist.

Trojans in configure scripts have been found in the wild.

So not only are the Autotools a horrible build system, they are also a massive security hole. By design.

Post scriptum: A simple fix to this is to always generate the configure script yourself rather than using the one that comes with the tarball. But then you lose the main advantage of Autotools: that you don’t need special software installed on the build machine.

Accessing files, how hard can it be?

If there were no resource files, programming would be easy. You could just directly access any thing you need by its variable name. Unfortunately in the real world, you sometimes need to load stuff from files. This seemingly simple task is in fact very complicated.

When confronted with this problem for the first time the developer will usually do something like this:

fp = fopen("datadir/datafile.dat", "r");

Which kinda works, sometimes. Unfortunately this is just wrong on so many levels. It assumes that the program binary is run in the root of your source dir. If it is not, the path “datadir/datafile.dat” is invalid. (You are of course keeping your build directory completely separate from your source dir, right?) After diving into LSB specs and config.h autogeneration, the fearless programmer might come up with something like this:

fp = fopen(INSTALL_PREFIX "/datadir/datafile.dat", "r");

Which works. Sometimes. The main downside being that you have to run “make install” always before running the binary. Otherwise the data files in INSTALL_PREFIX may be stale and cause you endless debugging headaches. It also does not work if the binary is installed to a different directory than given in INSTALL_PREFIX. The platforms that can do this are Windows, OSX and, yes, Linux (though, to be honest, no-one really does that).

Usually the next step is to change the binary to take a command line argument specifying where its data files are. Then a wrapper script is created that determines where the binary currently lies, constructs the argument and starts the binary.

This also works. Sometimes. Unfortunately there is no portable scripting language that works on all major platforms so there need to be several scripts. Also, what happens if you want to run the binary under gdb? You can’t run the script under gdb, and the binary itself won’t work without the script. The only choice is to code custom support for gdb in the script itself. Simply invoking gdb reliably is hard. The commands to run are completely different depending on whether you are using Libtool or not and have installed the binary or not. If you want to run it under Valgrind, it needs custom support as well. The wrapper script will balloon into a rat’s nest of hacks and ifs very, very quickly.

Before going further, let’s list all the different requirements for file access. The binary would need to access its own files:

  • in the source tree when the binary is compiled in-source
  • in the source tree when the binary is compiled out-of-source
  • in the install directory when installed
  • in a custom, user specified directory (overriding all other options)
  • without wrapper scripts
  • in Unix, OSX and Windows

There are many ways to do this. A recommended exercise is to try to think up your own solution before going further.

The approach I have used is based on an environment variable, say MYPROG_PREFIX. Then we get something like this in pseudocode:

open_datafile(file_name) {
  if(envvar_set("MYPROG_PREFIX"))
    return fopen(envvar_value("MYPROG_PREFIX") + file_name);
  return platform_specific_open(file_name);
}

// One of these is #ifdeffed to platform_specific_open.

open_datafile_unix(file_name) {
  return fopen(INSTALL_PREFIX + file_name);
}

open_datafile_windows(file_name) {
  // Win apps usually lump stuff in one dir and
  // cwd is changed to that on startup.
  // I think. It's been a long time since I did win32.
  return fopen(file_name);
}

open_datafile_osx(file_name) {
  // This is for accessing files in bundles.
  // Unfortunately I don't know how to do that,
  // so this function is empty.
}

During development, the environment variable is set to point to the source dir. This is simple to do in any IDE. Vi users need to do their own thing, but they are used to it by now. ;-) The end user does not have this variable set, so their apps will load from the install directory.

The one remaining issue is Unix where the binary is relocated to somewhere else than the install location. A simple approach that comes to mind is to dynamically query the location of the current binary, and then just do CURRENT_BINARY_DIR + “../share/mystuff/datafile.dat”.

Unfortunately Posix does not provide a portable way to ask where the currently executing binary is. For added difficulty, suppose that your installed thing is not a binary but a shared library. It may lie in a completely different prefix than the binary that uses it and thus the app binary’s location is useless. I seem to recall that the Autopackage people had code to work around this but their website seems to be down ATM so I can’t link to it.

The Ubuntu Advantageā„¢

As a touch developer people often ask me what makes our touch stack better than the rest. As exhibit A I present this image of one of our competitor’s products.

This was found in Orlando’s Hard Rock Cafe.

On marketing and perception, part 3/3

Note: nothing written here should be seen as an endorsement or anything by Canonical or any other party. This is just me being a comedian speculating on things.

By now we have seen that in the world of marketing black is white and outside is downside or something to that effect. Let’s apply our newly found knowledge to a real world issue. If we were to design a new “image” for Ubuntu using the guidelines given, what would it look like.

First we need to determine what Ubuntu is. It is an operating system. Therefore we must not ever mention that fact. Or the fact that it is scalable, has high performance or any other attribute that can be quantified.

Then we need to determine what it is not. Reading through Internet postings we find that due to Ubuntu’s Unix heritage there are problems with non-working hardware, having to use the command line, compiling applications from source to use them and so on. Whether or not these accusations are true is irrelevant. They simply tell us that according to valued Internet posters such as mr Trolly McTrollenstein Ubuntu is user-hostile.

What is the opposite of hostile? There are several choices, but let’s go with cozy.

For a visual look we’re going to use a cheap trick: upturned palms. This is an age-old technique to look sincere as used by used car salesmen, politicians and other people whose job it is to make you trust them even if you really should not. Putting it all together we get something like this.

Ubuntu

The Coziest Computer Experience in the World

Now all that is needed is that a few million people keep repeating this mantra consistently to change reality as we know it.