Archive for March, 2016

David Henningsson

Daily driver builds with DKMS and Launchpad

So; assume that you have some new hardware that works for the most part, but you have some problems with your built-in sound card. The problem has been fixed upstream, but if you start using that particular upstream kernel only, you will lose Ubuntu kernel security updates. In some cases, bug fixes will come to Ubuntu kernels too – after some time – but in other cases these fixes won’t, for a variety of reasons.

You want to run a standard Ubuntu kernel, except for your sound driver (or some other driver), which you want to be something different. This actually happens quite often when our team enables hardware that isn’t yet on the market, and therefore lack full support in already released kernels.


To the rescue comes DKMS (short for Dynamic kernel module support), which installs the source of the actual driver on the machine, and, whenever the Ubuntu kernel is upgraded, automatically recompiles the driver to fit the new kernel. The compiled modules are installed into the right directory for them to be used at next boot. We’ve used this tool for several years, and found it to be incredibly useful.

Launchpad automation

Launchpad got a feature called recipes, which combines one or more bzr branches and automatically makes a source package whenever one of the source packages change. The source package is then uploaded to a ppa, which builds a binary package from the source package.

What is then the result of all this well-oiled machinery? That every day, you have the latest sound driver which is ready for you to install and use to see if it fixes your sound issues – and because it’s packaged as a normal Debian package, uninstallation is easy in case it does not work. We have had this up and running for the Intel HDA driver for several years now, and it’s been useful for both Canonical and the Ubuntu community.


That’s the birds-eye overview. In practice, things are a bit more complicated. Get ready for the mandatory boxes-and-arrows picture:


Preparing for import

Our main source is the master branch of sound.git, maintained by Takashi Iwai. However, Launchpad does not yet support git in recipe builds, therefore, a machine somewhere in the cloud runs a preparation script. This script checks the git branch for updates every hour and if there is one, starts with filtering out the “sound” directory (this is a simple optimization, because kernel trees are huge). The result is added to a bzr branch.

Actually this cloud machine does one more thing, but it’s more of a bonus: it runs some hda-emu based testing. Hda-emu is a tool for emulating an HD-audio codec, and takes alsa-info as input. So, we contributed a lot of alsa-infos from machines Canonical enable to the upstream hda-emu repository, along with some scripts to run some emulation tests on all of them. So, in case something breaks, we get an early warning, before the code reaches more people. The most common case for the test to break however is not an actual bug, but that the hda-emu tool needs updating to handle changes in the kernel driver. Therefore, the script is not stopped when this happens, it just puts a warning message in the bzr commit log.

The cloud machine runs a bzr server, which Launchpad then checks a few times per day for updates, and imports changes into a Launchpad hosted mirror branch.

Making a DKMS package

As our launchpad recipe detects that the bzr branch has changed, it re-runs the recipe. The recipe is quite simple – it only copies files from different branches into some directory, creates a source package out of the result, and uploads that package to a PPA. That’s where we combine the upstream source with our DKMS configuration. There is some scripting involved to e g figure out the names of the built kernel modules – if you’re making your own DKMS package, it will probably be easier to write that file by hand.

Unfortunately, compiling a new driver on an older kernel can be challenging, when the driver starts relying on features only present in the new kernel. Therefore, we regularly need to manually add patches to the new driver to make it compile on the older kernel.

Launchpad build

This part is just like any other build on a Launchpad PPA – it takes a source package and builds a binary package. This is where the backport patches actually get applied to the source. Please note that even though this is a binary package, what’s inside the package is not compiled binaries, it’s the source code for the driver. This is because the final compilation occurs on the end user machine.

(A funny quirk: when DKMS is invoked, it creates a .deb file by itself, but for some reason Launchpad wouldn’t accept this .deb file. I never really figured out why, but instead worked around it by manually unpacking DKMS’s .deb, then repacking it again using low-level dpkg-gencontrol and dpkg-deb tools.)

The binary package is then published in the PPA, downloaded/copied by the end user to his/her machine, and installed just like any other Debian package.

On the end user machine

The final step; where the driver source is combined with a standard Ubuntu kernel, is done on the end user’s machine. DKMS itself installs triggers on the end user machine that will be called every time a new kernel is installed, upgraded or removed.

On installation of a new kernel, DKMS will verify that the relevant kernel header package is also installed, then use these headers to recompile all installed DKMS binary packages against the new kernel. The resulting files are copied into /lib/modules/<kernel>/updates/dkms. On installation of a new DKMS binary package, the default is to recompile the new package against the latest kernel and the currently running kernel.

DKMS also runs depmod to ensure the kernel will pick up the newly compiled modules.

Final remarks

There are some caveats which might be worth mentioning.

First, if you combine the regular Ubuntu kernel (with security updates) with a DKMS driver, you will get security updates for the entire kernel except that specific driver, so in theory, you could be left with a security issue if the vulnerability is in the specific driver you use DKMS for. However, in practice the vast majority of security bugs are userspace facing code, rather than deep down in hardware specific drivers.

Second, on every Ubuntu kernel released there is a potential risk for breakage, e g, if the DKMS driver calls a function in the kernel and that function changes its signature, then the DKMS driver will fail to compile and install on the new kernel. Or even worse, the function changes behavior without changing signature, so that the DKMS driver will compile just fine, but break in some way when the driver runs. All I can say about that is that, to my knowledge, if this can happen then it happens very rarely – I’ve never seen it cause any problems in practice.