Canonical Voices

What James Westby talks about

Project Cambria

David, it's interesting that you posted about that, as it's something I've been toying with for the last couple of years. For the last few months I've been (very) slowly experimenting in my free time with an approach that I think works well, and I think it's time to tell more people about it and to ask for contributions.

Opportunistic programmers are useful to cater for here, as Debian/Ubuntu development isn't trivial, and so we are simplifying something existing, which means that it will still be powerful, which is also important. I'm not only interested in improving the experience for the opportunistic programmer though, why should they get all the cool stuff? I'm interested in producing something that I can use for doing Ubuntu development too (though not every last detail).

The project I am talking about has been christened "cambria" and is now available on Launchpad. It's a library that aims to provide great APIs for working with packages throughout the lifecycle, including things like Bazaar, PPAs, local builds, testing, lintian, etc. It should be pleasurable to use and also allow you to build tools on top that are also pleasurable. It should also allow for easy extension in to different GUI toolkits and for command-line tools, though I've only been working with GTK so far.

In addition, there is a gedit plugin that allows you to perform common tasks from within gedit. I chose gedit as it has a pleasant Python API for plugins, isn't so complicated that it takes much learning, and will already be installed on most Ubuntu desktop systems. As I said though, the libarary allows you to implement in anything you like (that can use a python library.)

I've put together some mockups that suggest some of the things that I would like to do:

A mockup of an inteface for building packages within gedit. There is a button to build the active package, and a box that shows the output of the build.


A mockup of an inteface for jumping to work on packages already downloaded in gedit. There is a list of packages that have previously been worked on, and the user can choose any to open a dialog of the contents of that package to choose a file to edit from within.

Package list

A mockup of an inteface for downloading the source of packages within gedit. The main point conveyed is that the user should be asked what they intend to work on (bug fix, merge, etc.) so that the tools can do some of the work for them, and wizards and the like can be used to do the rest.


The RATIONALE file includes some more reasons for the project:

Project cambria is about wrapping the existing tools for Debian/Ubuntu development to allow a more task-based workflow. Depending on the task the developer is doing there may be several things that must be done, but they must currently work each one out individually. We have documentation to help with this, but it's much simpler if your tools can take care of it for you.

Project cambria aims to make Ubuntu development easier to get started with. There are several ways that it will help. Providing a task-based workflow where you are prompted for the information that is needed to complete the task, and other things are done automatically, or defaults chosen helps as it means you can concentrate on completing the task, rather than learning about all the possible changes you could make and deciding which applies.

Project cambria aims to make Ubuntu development easier for everyone by automating common tasks, and alleviating some of the tool tax that we pay. It won't just be a beginner tool, but will provide tools and APIs that experienced developers can use, or can build upon to build tools that suit them.

Project cambria will help to take people from novice to experienced developer by providing documentation that allows you to learn about the issues related to your current task. This provides an easier way in to the documentation than a large individual document (but it can still be read that way if you like).

Project cambria will make Ubuntu development more pleasurable by focusing on the user experience. It will aim to pull together disparate interfaces in to a single pleasing one. Where it needs to defer to a different interface it should provide the user with an explanation of what they will be seeing to lessen the jarring effect.

I'm keen for others to contribute, there is some information about this in the project's CONTRIBUTING file. I'm looking for all sorts of contributions from all kinds of people and keen to help you get started if you aren't confident with the type of contribution you would like to make.

There's a mailing list as part of the ~cambria team on Launchpad and IRC channel if you are interested in discussing it more.

Read more

You may well have heard about it (on this blog especially), but though I spend lots of my time involved with it and talking to people about it, there may be some people who aren't entirely sure what we are doing with the Ubuntu Distributed Development initiative, or what we are trying to achieve. To try and help this I wrote up an overview of what we are doing.

If this project interests you and you would like to help, or just observe, then you can subscribe to the mailing list. There's lots of fun projects that you could take on: there's far more that is possible and would be hugely useful to Ubuntu developers than we can currently work on. If you want to work on something then feel free to talk to me about it and we can see if there is something that would suit you.

Without further ado...

The aim

The TL;DR version:

  1. Version Control rocks.
  2. Distributed version control rocks even more.
  3. Bazaar rocks particularly well.
  4. Let's use Bazaar for Ubuntu.

Or, if you prefer a more verbose version...

Ubuntu is a global project with many people contributing to the development of it in many ways. In particular development/packaging involves many people working on packages, and much of this requires more than one person to work on the change that it is being made, for e.g.

  1. Working on the problem together
  2. Sponsoring
  3. Other review


These things usually require the code to be passed backwards and forwards, and in particular, merged. In addition, we sometimes have to do things like merge the patch in the bug with a later version of the Ubuntu package. In fact, Ubuntu is a derivative of Debian, and we expend a huge effort every cycle merging the two.

Distributed version control systems have to be good at merging, it's a fundamental property. We currently do without, but we have tools such as MoM that use version control techniques to help us with some of the merging. We could carry on in this fashion, or we could move to use a distributed version control system and make use of its features, and gain a lot of other things in the process.

Tasks such as viewing history, and annotating to find who made a particular change and why, also become much easier than when you have to download and unpack lots of tarballs.

This isn't to say that there aren't costs to the transition, and tools and processes we currently use that don't currently have an obvious analogue in the bzr world. That just means we have to identify those things and put the work in to provide an alternative, or to port, where it makes sense.

The aim is therefore to help make Ubuntu developers more productive, and enable us to increase the number of developers, by making use of modern technologies, in particular Bazaar, though there are several other things that are also being used to do this.

What it isn't

This isn't a project to overhaul all the Ubuntu development tools. While there are many things I would like to fix about some of our tools (see some of the things that Barry had to get his head around in the "First Impressions" thread), that can go ahead without having to tie it in to this project. I hope that when me make some common tasks easier, it will focus attention on others that are still overly complex, and encourage people to work on those too.

We are not replacing the entire stack. We are building upon the lower layers, and replacing some of the higher ones. We aim for compatibility where possible, and not breaking existing workflows until it makes sense.

The plan

You can read the original overall specification for this work at

It is rather dry and lacking in commentary, and also a little out of date as we drill down in to each of the phases. Therefore I'll say a little more about the plan here.

The plan is to work from the end of the Ubuntu developers, converting the things that we work most directly with first. This should give the biggest impact. We will then work to pull in other things that improve the system.

This means that we start by making all packages available in bzr, and make it possible to use bzr to do packaging tasks. In addition to this we are working with the LP developers to make it possible for Soyuz to build a source package from the branch, so that you don't have to leave bzr to make a change to a package. This work is underway.

After that we make all of Debian available in bzr in the same way. This allows us to merge from Debian directly in bzr. At a first cut, this just allows us to replace MoM, but in fact allows for more than that. Have a conflict? You have much more information available as to why the changes were made, which should help when deciding what to do.

The next step after that is to also bring the Vcs-* branches in to the history. These are the branches used by the Debian maintainer, and so allow you to work directly with the Debian maintainer without switching out of the system that you have learnt.

In a similar way we then want to pull in the upstream branches themselves. Again, this will allow you to work closely with upstream, without having to step out of the normal workflow you know.

The last point deserves some more explanation. The idea is that you will be able to grab a package as you normally do, work on a patch, and then when you are happy run a command or three that does something like the following:

  • Merges your change in to the tip of upstream, allowing you to resolve any conflicts.
  • Provide a cover letter for the change (seeded with the changelog entry and/or commit message(s).
  • Send the change off to upstream in their preferred format and location (LP merge proposal, patch in the bugtracker, mailing list etc.)

As you can imagine, there are a fair number of prerequisites that we need to complete before we can get to that stage, but I think of that as the goal. This will smooth some of the difficulties that arise in packaging from having to deal with a variety of upstreams. Finding the upstream VCS, working out their preferred form and location for submission, rebasing your change on their tip etc. I hope this will make Ubuntu developers more efficient, make forwarding changes easier to do and do well, and save new contributors from having to learn too many things at once.

Where we are now

We currently have all of Ubuntu imported (give or take), you can

bzr branch lp:ubuntu/<source package name>

which is great in itself for many people.

We also have all of Debian imported, and similarly available with

bzr branch lp:debian/<source package name>

which naturally allows

bzr merge lp:debian/<source package name>

so you can make use of that right now.

We are also currently looking at the sponsorship process around bzr branches, and once we have that cracked it will be much easier for upstream developers who know bzr to submit a bugfix, and that's a large constituency.

In addition, this means that a new contributor can start without having to learn debdiff etc., we can pass code around without having to merge two diffs and the like.

This is great in itself, but we are still some way from the final goal.

We are currently working on the VCS-* branches, to make them mergeable, but their are a number of prerequisites.

In addition the Launchpad team are also working on making it possible to build from a branch.

Where we can go

As I said, building on top of bzr makes a number of things easier.

For instance, once LP can build from branches, we could have a MoM-a-like that very cheaply tries to merge from Debian every time there is an upload there, and if it succeeds build the package. This could then tell you not only if there were any conflicts in the merge, but any build failures, even before you download the code.

In addition, we are currently talking a lot about Daily Builds, building the latest code every day (or commit, week, whatever). There are a number of things this brings. It doesn't strictly require version control, but as it's basically a merging problem having everything in Bazaar makes it much easier to do. We have a system now built on "recipes" that we are working to add to LP.

Parts of the work

There are a number of parts to the work, and you will see these and others being discussed on the list:

  • bzr (obviously), which we sometimes need to change to make this work possible, either bug fixes, or sometimes new features.
  • bzr-builddeb, which is a bzr plugin that knows how to go from branch to package and vice-versa.
  • bzr-builder, the bzr plugin that implements "recipes."
  • Launchpad, which hosts the branches, provides the merge prosals, and will allow building from branches and daily builds.
  • The bzr importer, this is the process that mirrors the Ubuntu and Debian archives in to bzr and pushes the branches to LP.

and probably others that I have forgotten right now.

Read more

Commit access is no more

Many projects that I work on, or follow the development of, and granted there may be a large selection bias here, are showing some of the same tendencies. Combined these indicate to me that we need to change the way we look at becoming a trusted member of the project.

The obvious change here is the move to distributed version control. I'm obviously a fan of this change, and for many reasons. One of those is the democratisation of the tools. There is no longer a special set of people that gets to use the best tools, with everyone else having to make do. Now you get to use the same tools whether you were the founder of the project, or someone working on your first change. That's extremely beneficial as it means that we don't partition our efforts to improve the tools we use. It also means that new contributors have an easier time getting started, as they get to use better tools. These two influences combine as well: a long time contributor can describe how they achieve something, and the new contributor can directly apply it, as they use the same tools.

This change does mean that getting "commit access" isn't about getting the ability to commit anymore; everyone can commit anytime to their own branch. Some projects, e.g. Bazaar, don't even hand out "commit access" in the literal sense, the project blessed code is handled by a robot, you just get the ability to have the robot merge a branch.

While it is true that getting "commit access" was never really about the tools, it was and is about being trusted to shepherd the shared code, a lot of projects still don't treat it that way. Once a developer gets "commit access" they just start committing every half-cooked patch they have to trunk. The full use of distributed version control, with many branches, just emphasises the shared code aspect. Anyone is free to create a branch with their half-baked idea and see if anyone else likes it. The "blessed" branch is just that, one that the project as a whole decides they will collaborate on.

This leads to my second change, code review. This is something that I also deeply believe in; it is vital to quality, and a point at which open source software becomes a massive advantage, so something we should exploit to the full. I see it used increasingly in many projects, and many moving up jml's code review "ladder" towards pre-merge review of every change. There seems to be increasing acceptance that code review is valuable, or at least that it is something a good project does.

Depending on the project the relationship of code review and "commit access" can vary, but at the least, someone with "commit access" can make their code review count. Some projects will not allow even those with "commit access" to act unilaterally, requiring multiple reviews, and some may even relegate the concept, working off votes from whoever is interested in the change.

At the very least, most projects will have code review when a new contributor wishes to make a change. This typically means that when you are granted "commit access" you are able or expected to review other people's code, even though you may never have done so before. Some projects also require every contribution to be reviewed, meaning that "commit access" doesn't grant you the ability to do as you wish, it instead just puts the onus on you to review the code of others as well as write your own.

As code review becomes more prevalent we need to re-examine what we see as "commit access," and how people show that they are ready for it. It may be that the concept becomes "trusted reviewer" or similar, but at the least code review will be a large part of it. Therefore I feel that we shouldn't just be looking at a person's code contributions, but also their code review contributions. Code review is a skill, some people are very good at it, some people are very very bad at it. You can improve with practice and teaching, and you can set bad examples for others if you are not careful. We will have to make sure that review runs through the blood of a project, everyone reviews the code of everyone else, and the reviews are reviewed.

The final change that I see as related is that of empowering non-code contributors. More and more projects are valuing these contributors, and one important part of doing that is trusting them with responsibility. It may be that sometimes trusting them means giving them "commit access", if they are working on improving the inline help for instance. Yes, it may be that distributed version control and code review mean that they do not have to do this, but those arguments could be made for code contributors too.

This leads me to another, and perhaps the most important, aspect of the "commit access" idea: trust. The fundamental, though sometimes unspoken, measure we use to gauge if someone should get "commit access" is whether we believe them to be trustworthy. Do we trust them to introduce code without review? Do we trust them to review other people's changes? Do we trust them to change only those areas they are experts in, or to speak to someone else if they are not? This is the rule we should be applying when making this decision, and we should be sure to be aware that this is what we are doing. There will often be other considerations as well, but this decision will always factor.

These ideas are not new, and the influences described here did not create them. However the confluence of them, and the changes that will likely happen in our projects over the next few years, mean that we must be sure to confront them. We must discard the "commit access" idea as many projects have seen it, and come up with new responsibilities that better reflect the tasks people are doing, the new ways projects operate, and that reward the interactions that make our projects stronger.

Read more

One of the new things that is going to be in karmic is that the kerneloops daemon will be installed and running by default. This tool, created by Arjan van de Ven, watches the kernel logs for problems. It has a companion service, which aggregates reports of these problems, and can sort by kernel version and the like. This allows kernel developers to spot the most commonly encountered problems, areas of the code which are prone to bugs etc. When the kerneloops daemon catches a problem it allows you to send the problem to

We however, are not using the applet that comes with kerneloops to do this, we are making use of the brilliant Apport. There are a couple of reasons for this. We also want to make it easy for you to report these issues as bugs to Launchpad, and we don't want to prompt you with two different interfaces to do that.

The changes mean that if your machine has a kernel issue you will get an apport prompt as usual. As well as asking if you would like to report the problem to Launchpad like it does for other crashes it will ask if you would also like to report it to Passing the information through apport means that it can also be used on servers as well without running X.

Hopefully you will never see this improvement, but it's now going to be there for when those bugs do creep in.

Read more

I've just implemented the most requested feature in bzr-builder (Hi Ted), command support.

Sometimes you need to run a particular command to prepare a branch of your project for packaging (e.g. autoreconf). I think this should generally go in your build target, but not everyone agrees, and sometimes there is just no other way.

Therefore I added a new instruction to bzr-builder recipes, "run". If you put

run some command here

in your recipe then it will run "some command here" at that point when assembling.

Note that running commands that require arbitrary network access is still to be discouraged, as you don't know in what environment someone may assemble the recipe. I'd also advise against using commands unless you really need them, but that's obviously your call.

Read more

I recently gave a talk to some fellow Canonical employees about where we are with the "Distributed Development" project. For that I made a screencast showing some of the Launchpad Codehosting features that you can now use for Ubuntu. Thanks to the Launchpad team for making this happen. We're still ironing out the remaining kinks that make it a pain to use, and getting all the packages imported, but it's possible to use them now.

The screencast has no audio unfortunately, but you can watch it and try and guess what I was saying. There's documentation available on the wiki as well.

One of the things the video shows is how to request someone review your change, i.e. how to get a change sponsored in to Ubuntu. I'm keen to have people test this, as it's not something I do very often now that I am a core-dev. Therefore if you want to help test then propose a merge and set the appropriate sponsor team as the reviewer, and I will prioritise it and you can give me feedback in return.

Note that a bug in Launchpad means that I won't get a notification when they are created, so feel free to drop me a line via email or on IRC until that bug is fixed next month. I'll continue to poll the lists though, so nothing will get dropped.

Read more