Canonical Voices

Colin Watson

October already! As the leaves start to turn red here in the northern hemisphere, here’s a brief summary of what we did in September.


  • BugTask:+addcomment’s title doesn’t duplicate the bug number (#1323808)
  • The duplicates portlet of bug tasks no longer links to invisible private bugs and now uses the correct sprite for each bug (#1443418, #1465880)
  • Show hidden bug comments to their owners (#1391394)


  • Line numbers in merge proposal preview diffs are now unselectable (#1483925)
  • Branch revision listings link more sensibly to merged branches and merge proposals (#711647)
  • Git repositories and refs have more sensible breadcrumb links (#1466271)
  • Source package recipe builds now include the distribution series in their titles (#1491336)
  • Allow archive owners to cancel their recipe builds, and make recipes use the normal cancellation infrastructure (#624630)
  • Make it possible to retry superseded builds, since there are situations where they can become unsuperseded (#444030)
  • The index page for Git refs is more useful
  • Fix crashes on various +activereviews pages when there are no active reviews (#1499744, #1501134)
  • Precache branch permissions in Branch.landing_candidates, fixing API timeouts (#1500576)
  • Add webhook support for Bazaar branches, currently enabled on qastaging though not yet on production


  • Team membership notifications now honour the “Include filtering information in email footers” setting, and have rationale information in the headers and footer (#296889, #508897)
  • There is a new X-Launchpad-Message-For mail header (or Launchpad-Message-For in expanded footers) giving just the name of the person or team directly subscribed to the notification (#1493844)
  • Treat “me” in person-or-team contexts in mail handlers as the current user (#340397)

We now consider the “Gmail filtering improvements” work complete.  Let us know if there are further categories of mail sent by Launchpad that you’re finding difficult to filter using Gmail.


  • Distribution index pages hide links to disabled features (#80315, #257627)
  • Set consistent colour for the “Opinion” bug task status (#648645)
  • Blueprints on milestone pages have icons again and are sortable (#1354387)
  • Old user-to-user email database references no longer prevent changing team privacy (#1498497)

Snap packages

  • Snap packages now have edit, administer, and delete views
  • Branches, Git repositories and refs, people, teams, and products now have snap package listing views
  • Members of ~launchpad-snap-builders can create snap packages in the web UI based on Bazaar or Git branches
  • Snap package owners can request builds in the web UI

Soyuz (package building)

  • Stop showing the confusing .dsc component on source package index pages (#521722)
  • Export archive deletion on the API (#814633)
  • Allow overriding the build version for live filesystem builds (#1496074)
  • Archive owners can change the “build debug symbols” and “publish debug symbols” settings on their own archives, rather than needing to ask an admin
  • The publisher no longer sometimes tries to update Contents files for immutable suites such as the release pocket of stable series, which could cause checksums in Release to get out of sync (#1448270)
  • Publishing PPAs now creates clearsigned InRelease files to improve reliability of updates (#804252); we intend to add this to the primary Ubuntu archive as well once one last piece of mirroring infrastructure is made ready for it
  • Archive owners can enable or disable unrestricted architectures on their own archives, rather than needing to ask an admin; we will unrestrict further architectures once they have adequate virtualised build infrastructure available
  • Enabling a test rebuild archive no longer times out (#1500973)

Read more
Colin Watson

Here’s a summary of what the Launchpad team got up to in August.


  • Webhook support for Git repositories is almost finished, and only needs a bit more web UI work (#1474071)
  • The summary of merge proposal pages now includes a link to the merged revision, if any (#892259)
  • Viewing individual comments on Git-based merge proposals no longer OOPSes (#1485907)

Mail notifications

Our internal stakeholders in Canonical recently asked us to work on improving the ability to filter Launchpad mail using Gmail.  The core of this was the “Include filtering information in email footers” setting that we added recently, but we knew there was some more to do.  Launchpad’s mail notification code includes some of the oldest and least consistent code in our tree, and so improving this has entailed paying off quite a bit of technical debt along the way.

  • Bug notifications and package upload notifications now honour the “Include filtering information in email footers” setting (#1474071)
  • Bug notifications now log an OOPS rather than crashing if the SMTP server rejects an individual message (#314420, #916939)
  • Recipe build notifications now include an X-Launchpad-Archive header (#776160)
  • Question notification rationales are now more consistent, including team annotations for subscribers (#968578)
  • Package upload notifications now include X-Launchpad-Message-Rationale and X-Launchpad-Notification-Type headers, and have more specific footers (#117155, #127917)

Package build infrastructure

  • Launchpad now supports building source packages that use Debian’s new build profiles syntax, currently only with no profiles activated
  • Launchpad can now build snap packages (#1476405), with some limitations; this is currently only available to a group of alpha testers, so let us know if you’re interested
  • Builders can now access Launchpad’s Git hosting (HTTPS only) in the same way that they can access its Bazaar hosting
  • All amd64/i386 builds now take place in ScalingStack, and the corresponding bare-metal builders have been detached pending decommissioning; some of the newer of those machines will be used to further expand ScalingStack capacity
  • We have a new ScalingStack region including POWER8-based ppc64el builders, which is currently undergoing production testing; this will replace the existing POWER7-based builders in a few weeks, and also provide virtualised build capacity for ppc64el PPAs
  • We’ve fixed a race condition that sometimes caused a user’s first PPA to be published unsigned for a while (#374395)


  • The project release file upload limit is now 1 GiB rather than 200 MiB (#1479441)
  • We spent some more time supporting translations for the overlay PPA used for current Ubuntu phone images, copying a number of existing translations into place from before the point when they were redirected automatically
  • Your user index page now has a “Change password” link (#1471961)
  • Bug attachments are no longer incorrectly hidden when displaying only some bug comments (#1105543)

Read more
Colin Watson

Here’s a summary of what the Launchpad team got up to in July.


  • We fixed a regression in the wrapping layout of side-by-side diffs on (#1436483)
  • Various code pages now have meta tags to redirect “go get" to the appropriate Bazaar or Git URL, allowing the removal of special-casing from the “go" tool (#1465467)
  • Merge proposal diffs including mention of binary patches no longer crash the new-and-improved code review comment mail logic (#1471426), and we fixed some line-counting bugs in that logic as well (#1472045)
  • Links to the Git code browsing interface now use shorter URL forms

We’ve also made a fair amount of progress on adding support for triggering webhooks from Launchpad (#342729), which will initially be hooked up for pushes to Git repositories.  The basic code model, webservice API, and job retry logic are all in place now, but we need to sort out a few more things including web UI and locking down the proxy configuration before we make it available for general use.  We’ll post a dedicated article about this once the feature becomes available.

Mail notifications

We posted recently about improved filtering options (#1474071).  In the process of doing so, we cleaned up several older problems with the mails we send:

  • Notifications for a bug’s initial message no longer include a References header, which confuses some versions of some mail clients (#320034)
  • Package upload notifications no longer attempt to transliterate non-ASCII characters in package maintainer names into ASCII equivalents; they now use RFC2047 encoding instead (#362957)
  • Notifications about duplicate bugs now include an X-Launchpad-Bug-Duplicate header (#363995)
  • Package build failure notifications now include a “You are receiving this email because …” rationale (#410893)

Package build infrastructure

  • The sbuild upgrade last month introduced some regressions in our handling of package builds that need to wait for dependencies (e.g. #1468755), and it’s taken a few goes to get this right; this is somewhat improved now, and the next builder deployment will fix all the currently-known bugs in this area
  • In the same area, we’ve made some progress on adding minimal support for Debian’s new build profiles syntax, applying fixes to upload processing and dependency-wait analysis, although this should still be considered bleeding-edge and unlikely to work from end to end
  • We’ve been working on adding support for building snap packages (#1476405), but there’s still more to do here; we should be able to make this available to some alpha testers around mid-August


  • We’ve arranged to redirect translations for the overlay PPA used for current Ubuntu phone images to the ubuntu-rtm/15.04 series so that they can be translated effectively (#1463723); we’re still working on copying translations into place from before this fix
  • Projects and project groups no longer have separately-editable “display name” and “title” fields, which were very similar in purpose; they now just have display names (#1853, #4449)
  • Cancelled live file system builds are sorted to the end of the build history, rather than the start (#1424672)

Read more
Colin Watson

Users of some email clients, particularly Gmail, have long had a problem filtering mail from Launchpad effectively.  We put lots of useful information into our message headers so that heavy users of Launchpad can automatically filter email into different folders.  Unfortunately, Gmail and some other clients do not support filtering mail on arbitrary headers, only on message bodies and on certain pre-defined headers such as Subject.  Figuring out what to do about this has been tricky.  Space in the Subject line is at a premium – many clients will only show a certain number of characters at the start, and so inserting filtering tags at the start would crowd out other useful information, so we don’t want to do that; and in general we want to avoid burdening one group of users with workarounds for the benefit of another group because that doesn’t scale very well, so we had to approach this with some care.

As of our most recent code update, you’ll find a new setting on your “Change your personal details” page:

Screenshot of email configuration options

If you check “Include filtering information in email footers”, Launchpad will duplicate some information from message headers into the signature part (below the dash-dash-space line) of message bodies: any “X-Launchpad-Something: value” header will turn into a “Launchpad-Something: value” line in the footer.  Since it’s below the signature marker, it should be relatively unobtrusive, but is still searchable.  You can search or filter for these in Gmail by putting the key/value pair in double quotes, like this:

Screenshot of Gmail filter dialog with

At the moment this only works for emails related to Bazaar branches, Git repositories, merge proposals, and build failures.  We intend to extend this to a few other categories soon, particularly bug mail and package upload notifications.  If you particularly need this feature to work for some other category of email sent by Launchpad, please file a bug to let us know.

Read more
Colin Watson

It’s been a while since we posted much regularly on this team blog, not least because for a while Launchpad was running more or less in maintenance mode.  That’s no longer the case and we’re back to the point where we can do feature development work again, as exemplified by our recent addition of Git code hosting support.

Lots of other things have been happening in the Launchpad world lately, though, and the half-way point in the year seems like a good time to start talking about them.  I’m going to try to do this a bit more regularly, aiming for about once a month when we also update our internal stakeholders.  This post covers roughly the last three months.


Of course, you don’t get to release a major new feature and then have everything be perfect.  In fact, we’d released it pretty much as soon as we had a minimum viable product, so we knew there was plenty more to do.

Now that the basics work reasonably well, we’ve been focusing on tying off loose ends, and on getting to the point where Launchpad itself can be self-hosted in Git: this is partly because most of the things we need to do for that are relevant to many other projects considering a migration, partly because that ensures that any problems with it will affect Launchpad developers directly, and partly because this will allow us to trim a very large amount of database cruft associated with the representation of Launchpad’s own branches.

Here’s a brief changelog since our initial public announcement:

  • Leaving the “Target reference path” field empty when proposing a Git-based merge proposal no longer crashes (#1451068)
  • Git-based merge proposals can be deleted
  • Pushing to private repositories no longer hangs (#1451107)
  • The backend now stores reflogs so that we can help users recover from mistakes
  • Pushing a new default repository for a project now sets the repository owner to the project owner rather than to the person performing the push
  • The project +sharing view now handles Git repositories
  • Branches of a project default repository are now shown on the project’s code page, although we expect to be making significant further improvements in this area soon
  • Individual Git commits under “Recent Commits” now link directly to a full view of the corresponding commit
  • Merge proposal listings now cover Git (#1453020)
  • Branches in private repositories now show a privacy banner (#1457553)
  • Repositories can now be deleted (#1456583)
  • Private repositories are now browseable and show a privacy banner in the browsing interface
  • The URLs for Git-based merge proposals have moved to be directly under the source repository rather than under the source branch, so that historical MPs can still be accessed after deleting the branch (#1456589)
  • Launchpad now sends mail for Git-based merge proposals
  • There are various new Git repository listing views, chiefly for projects, packages, and people
  • Repositories now have a complete edit form allowing you to rename them, move them between targets, and set their default branch (“HEAD”) (#1456625)
  • We’ve exported a few more bits and pieces on the webservice API to support Git merge robots
  • We’ve brought various bits of automation for Git-based merge proposals up to par with their Bazaar equivalents: when a Git branch is updated, Launchpad now automatically updates the preview diffs for any MPs where that is the source branch, and detects whether any MPs where that is the target branch can now be marked as Merged
  • The Code settings view for projects now allows you to set the project’s default Git repository and to choose whether the default revision system for the project is Bazaar or Git

We’ve started on support for webhooks, and hope to be able to tell you more about this soon. Git-to-Git repository mirroring and native Git recipes are also frequent requests that we know are important to people, and we hope to get to those soon.

Other code review changes

Many of the changes needed for Git touched on our code review infrastructure, but we’ve been making some other improvements there too.  Launchpad has supported “inline comments” on merge proposals for a year or so now, allowing you to comment on parts of a diff without having to manually quote it; but the feature hadn’t been quite finished and lots of people had nits to pick with it.  We’ve made a couple of improvements here lately:

  • Preview diffs in merge proposals now have keyboard navigation to move between files, diff hunks, and inline comments
  • Links to diffs now include a per-file diffstat under an expander triangle
  • The mails sent when a reviewer adds inline comments now only include relevant parts of the diff context rather than the entire diff, making them much easier to read and making it harder to miss comments (#1334577)

We threw in a couple of bonus features here as well.  You can now see side-by-side diffs for merge proposals as well as unified diffs (#917502): there’s a link at the top of each preview diff allowing you to switch it to side-by-side mode (we may add a user preference for this later).  And resubmitting a merge proposal now automatically preserves the commit message (#676769).

Package build infrastructure

When we first added support for building debug symbol packages (ddebs) in Ubuntu, we handled the build and publication side of things with a temporary hack involving a custom hack to our sbuild fork that stashed the ddebs on the builder for a while, and a job that periodically fetched them to As is the way of temporary hacks, it proceeded to stay that way for eight and a half years. It mostly worked, but was rather fragile, contributed to tying us to an old sbuild, wasn’t very extensible to PPAs, and couldn’t work with virtualised builders where all the builder’s state is reset before every build.

A couple of years ago we put together the Launchpad changes required to store ddebs straightforwardly in Launchpad instead, but weren’t able to deploy this at the time due to needing better librarian infrastructure. We finally got all this cleared out in April and deployed the new ddeb publishing mechanism. The main user-visible effect of this is that isn’t liable to lose packages any more, and ddebs are also available from Launchpad build pages.

Having done this, we were able to upgrade from our 2004-era fork of sbuild to a modern version, which fixed a number of long-standing bugs.

All architectures are now optional for PPAs (#1244868), so, for example, a PPA that only needs to support armhf can save builder cycles by disabling amd64 and i386. At the moment this can only be done by Launchpad administrators, so ask a question on Launchpad if you think this would be worth doing for any of your PPAs.

We rewrote the way that we install build-dependencies for source package recipes. This allows recipes to handle the “:native” qualifier in build-dependencies which is used in some places as part of supporting cross-building.

The next step in this modernisation programme will be to move all builds into ScalingStack and decommission the corresponding bare-metal builders, which we’ll be starting soon and working on an architecture-by-architecture basis as that cloud gains support for them. In the short term, this will give us better build capacity on the migrated architectures; in the longer term, it will let us support PPA builds on more architectures, and erase the cumbersome distinction between “non-virtualised” and “virtualised” builders. More news on this as it happens.


We upgraded to significantly more modern database servers earlier this year, which meant that a lot of hitherto difficult timeout problems suddenly either disappeared or became tractable. There are still some bad queries and a few mysterious problems, but as a general trend our timeout rates are very significantly down from where they were six months ago. We have some more general plans in this area, and will continue to spot-fix bad pages as they show up and as time permits.


We have lots more to do across the whole application, but still have a rather limited number of developers. If any of this kind of thing sounds interesting and you’d like to help, you can!

Read more
William Grant

Today we’re announcing early support for hosting Git repositories directly on Launchpad, as well as or instead of Bazaar branches. This has been by far the single most commonly requested feature from Launchpad code hosting for a long time; we’ve been working hard on it for several months now, and we’re very happy to be able to release it for general use.

This is distinct from the facility to import code from Git (and some other systems) into Bazaar that Launchpad has included for many years. Code imports are useful to aggregate information from all over the free software ecosystem in a unified way, which has always been one of the primary goals of Launchpad, and in the future we may add the facility to import code into Git as well. However, what we’re releasing today is native support: you can use git push to upload code to Launchpad, and your users and collaborators can use git clone to download it, in the same kind of way that you can with any Git server.

Our support is still in its early stages, and we still have several features to add to bring it up to parity with Bazaar hosting in Launchpad, as well as generally making it easier and more pleasant to use. We’ve released it before it’s completely polished because many people are clamouring to be able to use it and we’re ready to let you all do so. From here on in, we’ll be adding features, applying polish, and fixing bugs using Launchpad’s normal iterative deployment process: changes will be rolled out to production once they’re ready, so you’ll see the UI gradually improving over time.

What’s supported?

You can:

  • push Git repositories to Launchpad over SSH
  • clone repositories over git://, SSH, or HTTPS
  • see summary information on repositories and the branches they contain in the Launchpad web UI
  • follow links from the Launchpad web UI to a full-featured code browser (cgit)
  • push and clone private repositories, if you have a commercial subscription to Launchpad
  • propose merges from one branch to another, including in a different repository, provided that they are against the same project or package

What will be supported later?

Launchpad’s Bazaar support has grown many features over the years, and it will take some time to bring our Git support up to full parity with it and beyond. Git repositories use a somewhat different model from Bazaar branches, which we’ve had to account for in many places, and some facilities will require complete reimplementation before we can support them with Git.

Here’s an incomplete list of some of the features we hope to add:

  • useful subscriptions (currently only attribute change notifications work, which are not usually very interesting in themselves)
  • RSS feeds
  • mirroring
  • webhooks
  • an integrated code browser

See our help page for more known issues and instructions on using Launchpad with Git.

Helping out

This is a new service, and we welcome your feedback: you can ask questions in #launchpad on freenode IRC, on our launchpad-users mailing list, or on Launchpad Answers, and if you find a bug then please tell us about that too.

Launchpad is free software, licensed under the GNU AGPLv3. We’d be very happy to mentor people who want to help out with parts of this service, or to build things on top of it using our published API. Some preliminary documentation on this is on our developer wiki, and you can always contact us for help.

Read more
Colin Watson

Launchpad build farm improvements

We’ve made a number of improvements to the Launchpad build farm in the last month, with the aim of improving its performance and robustness.  This sort of work is usually invisible to users except when something goes wrong, so we thought it would be worth taking some time to give you a summary.  Some of this work was on the Launchpad software itself, while some was on the hardware.

(To understand some of the rest of this post, it’s useful to be aware of the distinction between virtualised and devirtualised builders in Launchpad.  Virtualised builders are used for most PPAs: they build untrusted code in a Xen guest which is initialised from scratch at the start of each build, and are only available for i386, amd64, and a small number of ARM builds by way of user-mode QEMU.  Devirtualised builders run on ordinary hardware with less strict containment, and are used for Ubuntu distribution builds and a few specialised PPAs.)

ARM builders have been a headache for some time.  For our devirtualised builders, we were using a farm of PandaBoards, having previously used BeagleBoards and Babbage boards.  These largely did the job, but they’re really a development board rather than server-class hardware, and it showed in places: disk performance wasn’t up to our needs and we saw build failures due to data corruption much more frequently than we were comfortable with.  We recently installed a cluster of Calxeda Highbank nodes, which have been performing much more reliably.

It has long been possible to cancel builds on virtualised builders: this is easy because we can just reset the guest.  However, it was never possible to cancel builds on devirtualised builders: killing the top-level build process isn’t sufficient for builds that are stuck in various creative ways, and you need to make sure to go round and repeatedly kill all processes in the build chroot until they’ve all gone away.  We’ve now hooked this up properly, and it is possible for build daemon maintainers to cancel builds on devirtualised builders without operator assistance, which should eliminate situations where we need urgent builds to jump the queue but can’t because all builders are occupied by long-running builds.  (People with upload privileges can currently cancel builds too, which is intended mainly to allow cancelling your own builds; please don’t abuse this or we may need to tighten up the permissions.)  As a bonus, cancelling a build no longer loses the build log.

Finally, we have been putting quite a bit of work into build farm reliability.  A few problems have led to excessively long queues on virtual builders:

  • Builders hung for some time when they should have timed out, due to a recent change in su; this is now fixed in the affected Ubuntu series.
  • Xen guests often fail to restore for one reason or another, and when this happened builders would fail in ways that required an operator to fix.  We had been dealing with this by having our operators do semi-automatic builder fixing runs a few times a day, but in recent months the frequency of failures has been difficult to keep up with in this way, especially at the weekend.  Some of this is probably related to our current use of a rather old version of Xen, but the builder management code in Launchpad could also handle this much better by trying to reset the guest again in the same way that we do at the start of each build.  As of this morning’s code deployment, we now do this, and the build farm seems to be holding up much more robustly.

This should make things better for everyone, but we aren’t planning to stop here.  We’re intending to convert the virtual builders to an OpenStack deployment, which should allow us to scale them much more flexibly.  We plan to take advantage of more reliable build cancellation to automatically cancel in-progress builds that have been superseded by new source uploads, so that we don’t spend resources on builds that will be rejected on upload.  And we plan to move Ubuntu live file system building into Launchpad so that we can consolidate those two build farms and make better use of our available hardware.

Read more
Deryck Hodge

Today, the Private Projects and Private Blueprints features on Launchpad are leaving beta. These features are available now for use by all Launchpad users. Private Blueprints was started as part of the Private Projects work, with the end goal in mind of truly private projects on Launchpad. Private Projects was described in its beta announcement like this:

When creating a new project on Launchpad, beta testers will have the option to create “Proprietary” or “Embargoed” projects. Embargoed exists for projects that intend to start private but later be revealed publicly. All other private projects should be proprietary. Milestones and series are proprietary or embargoed based on the project setting. To make them public, you will need to make the project itself public.

When you create a proprietary or embargoed project on Launchpad, all of the sharing policies for your project will be set correctly for you. This means that if you start your project as a proprietary project, your branches, bugs, and blueprints will be created proprietary by default. Answers and translations are not available for private projects.

A commercial subscription is required to use private projects, but any user who creates a proprietary or embargoed project on Launchpad will receive a 30 day trial commercial subscription. Launchpad users with existing commercial subscriptions can convert a public project to proprietary or embargoed by changing the information type in the project’s settings. You may have some work to do on your project before you can transition to a private information type — for example, disable answers if you have that app enabled for your project — but Launchpad will block the change and tell you what needs to happen before you can switch to a private information type.

Users should be aware, though, that if your project has been listed on Launchpad publicly until now, then search engines know of its existence already. If you want a proprietary project that no one can learn of its name, you should create a new project on Launchpad. Transitioning a public project to private allows you to keep your series and milestones private going forward, but users may have already been able to discover the existence of the project since it was public already.

We are happy to make truly private projects available for all users on Launchpad. If you run into any issues, please file a bug against Launchpad or ask for help in #launchpad on Freenode.

Read more
Laura czajkowski

Launchpad has been a key tool used in developing Novacut. I use Launchpad for code hosting, bug tracking, daily builds, and more. For almost two years I’ve been doing monthly stable releases on Launchpad, and Novacut now spans six separate Launchpad projects. To say the least, I’ve learned a lot about Launchpad in the process.

I don’t think Novacut could be where it is today without Launchpad, so I want to pass on some of what I’ve learned the past two years. Here are my five essential Launchpad best practices:

1. Daily Builds

I’m always very thankful that early on Paul Hummer took the time to school me on using Source Package Recipes to do daily builds. This Launchpad service gives you automated package builds across multiple architectures, and multiple Ubuntu releases.

I don’t know how to emphasize this enough, but seriously, you need daily builds. As a point of reference, daily builds are the 3rd item in the famed Joel Test.

These builds are triggered simply by making commits to the appropriate bzr branch on Launchpad (usually your trunk branch). You’ll automatically get up to one build per 24-hour period, and you can manually trigger additional builds when needed.

You can include your debian/ packaging directory in your project source tree, or you can keep debian/ in a separate bzr branch. For the Novacut components, I’ve found it most helpful to keep debian/ in the source trees because it’s handy to be able to land a code change and its corresponding packaging change in a single merge. This works for us because we currently can use the exact same debian/ for all the Ubuntu versions we support. If that’s not true for your project, you’ll need multiple debian/ branches.

For reference, here’s the Novacut Source Package Recipe.

2. Unit Tests

You should run your unit tests during your package builds, and you should fail the build when any unit test fails. This is particularly important for daily builds, because this will prevent a package with broken unit tests from reaching your daily PPA.

The Launchpad build servers are strict and unforgiving environments, which is a good thing when it comes to unit tests. The build servers are also probably quite different from your local development environment. On countless occasions our daily builds have caught failures that only occur on i386 (my workstation is amd64), or only occur on an Ubuntu release other than the one I’m running, etc.

To run your unit tests during the package build, you’ll need to modify your debian/rules file as appropriate. If you’re using debhelper, add an override_dh_auto_test target.

You might also need to add additional packages to the Build-Depends section of your debian/control file, packages that are needed by the unit tests but are otherwise not needed by the build itself.

For reference, here’s the debian/rules file used to run the Dmedia unit tests (which is also a handy Python3 example).

3. Track Ubuntu+1

When a new Ubuntu version opens up for development, I immediately start doing daily builds on the development version, even though I don’t typically upgrade my own computers till around 4 months into the cycle.

I use daily builds on the development release as an early warning system. With no extra effort on my part, these builds give me a heads-up about code or packaging changes that might be needed to make Novacut work well on the next Ubuntu release.

To enable daily builds on the next Ubuntu version, just go to your Source Package Recipe, click on “Distribution series”, and check the box for the newest series. Now you’ll have daily builds on the newest Ubuntu version, in addition to all the versions you were already building for.

For example, I’m currently in the process of enabling daily builds for Raring, as you can see in the Microfiber Source Package Recipe. And I did indeed encounter a build failure on Raring, seemingly caused by a debhelper issue.

For the first month or so in a cycle, I don’t tend to worry much about build failures on the development version. But after the dust has settled a bit, I make sure to keep the builds in working order, and I even do monthly stable releases for the Ubuntu development version. Again, I do all this pro-actively even before I personally start running the newest Ubuntu version.

4. PPAs & Users

Whenever someone asks me why I use Launchpad instead of github, my short answer is always, “PPAs and users”.

Source Package Recipes give you much more than just a build, they give you daily packages that are easily consumable by your testing community and early adopters. This tight feedback loop prevents you from running too far ahead without getting a good reality check from your target users.

Keep in mind that for some products, the early adopters willing to install from a PPA might not be all that representative of your target user. So when it comes to making design decisions, you might need to politely ignore certain feedback from some of these early adopters. In my experience, this wont cause any hard feelings as long as you have clearly communicated who your target user is, and why.

For reference, you might look at the way we’ve defined the Novacut target user.

I recommend creating PPA names that are well-branded and easy to remember. First, create a Launchpad team with the same name as your product. In our case, we have a ~novacut team. Second, I recommend creating a daily and a stable PPA owned by the same team. In our case, that gives us two easy to remember PPAs:

Although none of our target users (professional video editors) currently use Ubuntu to do their job, I’ve been surprised by how many follow Novacut’s development via our stable PPA, and even our daily PPA. This has helped keep us on track, and has helped us build customer loyalty even before we have a finished product.

For me personally, this daily user engagement also makes the design and development process more enjoyable. It’s hard to empathize with an abstract persona; it’s easier to solve specific problems for specific people.

5. Use Apport

Till recently I didn’t realize that you can use Apport for automated crash reporting in unofficial packages delivered through a PPA.

We haven’t had Apport integration for that long, but it’s already provided us with dozens of highly valuable crash reports. Almost immediately some hardware specific issues came to light and were fixed, convincing me that a key benefit of Apport is knowing how your app might misbehave on a larger, more variable pool of hardware.

Apport also helped some rare bugs come to light. I thought Dmedia was basically crash-free, but those one-in-a-thousand bugs pop out quickly when thousands of people are running it. Most of these bugs would have eventually been found by one of our core devs, but the quicker a bug is discovered, the quicker and easier the bug is to fix.

For more info, check out this blog post and this screencast, where I covered our Apport integration in detail.

And for reference, see the merge proposal that added Apport integration in Novacut.

A big thank you to Jason DeRose for sharing how his project uses Launchpad on a daily basis.


Read more
Curtis Hovey

Launchpad’s bug and branch privacy features were replaced by information sharing that permits project maintainers to share kinds of confidential information with people at the project level. No one needs to manage bug and branch subscriptions to ensure trusted users have access to confidential information.

The Disclosure features

Disclosure is a super feature composed on many features that will allow commercial projects to work in private. Untrusted users cannot see the project’s data. Project maintainers can share their project with trusted users to reveal all or just some of the project’s data. The ultimate goal is to create private project in Launchpad, but that feature required several other features to be completed first. The Purple squad worked on Trusted Pickers, Privacy Transitions, Hardened Projects, Social Private Teams, and Sharing.

There was a lot of overlap between each feature the Purple squad worked on. Though we could start each feature independent of one another, we could only complete about 90% of each. When the Sharing UI changes entered beta, we were unblocked and fixes about most of the remaining issues, but fixing all the issues required all projects to switch to Sharing.   We did not consider Sharing, or any of the required features complete until we fixed all the bugs.

Disclosure facts

  • Planning started in June 2010 to replace the existing privacy mechanisms with something that would scale.
  • Early testing revealed that users did not trust Launchpad because the UI could not explain what was confidential, or what the consequences of a change would be — this needed to be fixed too.
  • 149 related bugs were identified in Launchpad.
  • Work started in June 2011 by the Purple squad.
  • Replacing the old privacy mechanisms and addressing the trust and information issues took 16 months.
  • About 45,000 lines code were added to support the features.
  • About 15% of the lines were for missing JavaScript test coverage.
  • More that 700 bugs were fixed in total.
  • About 5% of the fixed bugs were caused by the old non-scaling privacy mechanisms.
  • About 4% of the fixed bugs were caused by old JavaScript enhancements that broke features for non-JavaScript users.

Lessons learned

  • Misrepresentation of what is confidential, or what will be confidential or public is very important to users — more important than supporting private data.
  • Privacy/Sharing must be a first-class mechanism beneath all the mechanisms that work with confidential data.
    • Privacy was added on top of bugs, and it failed to scale to 100’s of bugs.
    • Privacy was added on top of branches, and it failed to scale to 1000’s of branches.
    • Filtering private items in code, or in database joins is not fast enough to work with 100,000’s of items.
  • Launchpad’s ReSTful object API is not suitable for working with large collections of objects like bugs or branches; a lighter, service-based approach was used to quickly work with large amounts of data.
  • Users need to work with confidential data via the API, using a text web browser from servers, using a browser with accessibility tools, as well as the common case of using a JavaScript enabled browser.
  • Lots of mock-ups and interactive tests will not predict all the interactions a user will have with real data; test with real code and data early to developer the final design.

Read more
Deryck Hodge

Private projects for beta testers

If you are part of Launchpad’s beta testers team, you can now start trialing private projects on Launchpad. The private projects feature builds on the great sharing work that Launchpad’s Purple Squad has done, allowing Launchpad users to create true private projects now. A commercial subscription is required to use private projects, but any user who creates a private project on Launchpad will receive a 30 day trial commercial subscription.

When creating a new project on Launchpad, beta testers will have the option to create “Proprietary” or “Embargoed” projects. Embargoed exists for projects that intend to start private but later be revealed publicly. All other private projects should be proprietary. Milestones and series are proprietary or embargoed based on the project setting. To make them public, you will need to make the project itself public.

Be warned, this is a large change to Launchpad and there are certainly bugs in our handling of privacy. You can check out our list of known issues, if you’d like. We, as the Launchpad Orange Squad, are committed to fixing all of those before we leave beta. So don’t worry, we’re still actively working on this feature. We did, however, want users to begin using this feature to get early feedback on the work. Don’t trial your super secret project with this feature just yet, but if you have something safe to try out private projects, now is a good time for beta testers to get going with the feature.

Enjoy private projects on Launchpad now, beta testers! And please file any bugs you find.

Read more
Laura czajkowski

Launchpad Workshop at UDS-R

After the success of the Launchpad clinics at the last UDS-Q we’ve decided to run some more! This time removed the sterile name of clinic and called them workshops.

If you want to get involved, scratch that itch, learn how to fix that irksome bug that has been bugging you’re not alone. Everyone probably has at least one that they’d like to see fixed.  The problem is now knowing how to fix them or maybe they don’t know how to set up the Launchpad development  environment, well lucky for you we have a lot of Launchpad developers at UDS-R and we’d like to help you help get bugs fixed!

The idea being if you have a bug you would like to fix, or pointed in the right direction  that we’ll be there to help you get on the road  to offer advice on every step of the Launchpad development process from Lines of code, to branch reviews to getting things done. We’ll have EC2 instances ready for you to develop on, so if you haven’t already gone through the process of setting up local Launchpad development on your machine, you don’t need to worry.

I have created a wiki page on which you should register if you’re going to be attending either of the clinics. Just list your name and the ID of the bug(s) you want to work on on that page. We’ll check the bugs out and get in touch with you if we think they’re too big to work on in the clinics – in which case we’ll try and work with you to get them fixed over a longer period. We’ve added the event to summit schedule, for Tuesday and Thursday of UDS so why not sign up and come along!

If you’ve never contributed before, Graham Binns has written a useful guide to contributing to  Launchpad.  He has also done up a screencast on fixing a bug in Launchpad.

Read more
Curtis Hovey

I have been analysing Launchpad’s critical bugs to track the Purple squad’s progress while on Launchpad maintenance duty. In January of 2011, the Cloud Engineering team né Launchpad Engineering team was reorganised into squads, where one or more squads would maintain Launchpad while other squads work on features. This change also aligned with a new found effort to enforce the zero-oops policy. The two maintenance squads had more than 332 critical bugs to close before we could consider adding features that the stakeholders and community wanted. By July 2011, the count dropped to its lowest point, 250 known critical bugs. Why did the count stop falling for fifteen months? Why is the count falling again?

Charting and analysing critical bugs

Chart of Launchpad's critical bugs since the formation of Launchpad squads and maintenance duties
The chart above needs some explanation to understand what is happening in Launchpad’s critical bugs over time. (You may want to open the image in a separate window to see everything in detail.) Each iteration is one week. The backlog represent the open critical bugs in launchpad at the start of the iteration. The future bugs are either bugs that are not discovered, not introduced, or reported and fixed within the iteration. The last group is crucial to understand the lines plotting the number of bugs fixed and added during the iteration. We strive to close critical bugs immediately. Most critical bugs are reported and fixed in a few days, so most bugs were not open long enough to be show up in the backlog. The number of bugs fixed must exceed the number added to make the backlog count fall. You can see that the maintenance squads have always been burning down the critical bugs, but if you are just watching the number of open bugs in Launchpad, you get the sense that the squads are running to just stand still.

I use the lp-milestone YUI widget to chart the bugs and analyse the our progress through the critical bugs. It allows me to summarise a set of bugs, or analyse a subset by bug tag.

Launchpad maintenance analysis -- driving critical bugs to zero

Though 22 bugs were fixed this past week, 14 were added, thus the critical count dropped by 8. The last eight iterations are used to calculate the average bugs closed and open per iteration. The relative velocity (velocity – flux) is used to estimate the remaining number of days to drive the count to zero. When the Purple squad started maintenance on September 10th of 2012, the estimated days of effort was more than 1,200. In just three weeks, the number has fallen dramatically. The principle reason the backlog of critical bugs has fallen is that the Purple squad is now giving those bugs their full attention, but that generalisation is unsatisfactory.

Why is the Purple squad so good at closing bugs in the critical backlog?

I do not know the answer to my question. The critical backlog reached its all-time low of 250 bugs with the release of the Purple squad’s maintenance work in July 2011. There was supposition that  Purple fixed the easy bugs, or that the fixes did not address the root cause, so another critical bug was opened. I disagree. The squad had no trouble finding easy bugs, and it too would have been fixing secondary bugs if the first fix was incomplete. I can tell you how the squad works on critical bugs, but not why it is successful.

I was surprised to see the Purple squad were still the top critical bug closers when it returned to maintenance after 15 months of feature work. How could that be?  The squad fixed a lot of old timeout and JavaScript bugs in the last few months through systemic changes — enough to significantly affect the statistics. About 600 critical bugs were closed while Purple squad were on feature work. The squad closed 210 of those bugs. 60 were regressions that were fixed within the iteration, so they never showed up in the backlog. 70 critical bugs were fixed because they blocked the feature, and 80 critical bugs were because Purple was the only squad awake when the issue was reported. The 4 other squads fixed an average of 98 bugs each when they were on maintenance. The Purple squad fixed more bugs then maintenance squads on average even when they were not officially doing maintenance work.  The data, charts, and analysis always includes the Purple squad.

I suspect the Purple squad has more familiarity with bugs in the critical backlog. They never stopped reading the critical bugs when they were on feature work. They saw opportunities to fix critical bugs while solving feature problems. I know some of the squad members are subscribed to all critical bugs and re-read them often. They triage and re-triage Launchpad bugs. This familiarity means that many bugs are ready to code — they know where the problem is and how to fix it before the work is assigned to them. They fixed many bugs in less than a day, often doing exactly what was suggested in the bug comments.

During the first week of their return to maintenance, about 30 critical bugs were discovered to be dupes of other bugs. Though this change does make the backlog count fall, it also revised all the data, so the chart is not showing these 30 bugs as at all now. The decline of backlog bugs does not include dupes. While the squad was familiar enough to find many bugs that they close in a single day, they were not so familiar as to have known that there were 30 duplicate bugs in the backlog when they started.

Most squad have only one person with DB access, but the Purple squad is blessed with 3 people who can test queries against production-level data. This could be a significant factor. It is nigh impossible to fix a timeout bug without proper database testing. Only 13 of the recent bugs closed were timeouts though. The access also helps plan proper fixes for other bugs as well, so maybe 20% of the fixed bugs can be attributed to database access.

Maybe the Purple squad are better maintenance engineers than other squads who work on maintenance. For 28 months, I was the leading bug closer working on Launchpad. I closed 3 times more bugs than the average Launchpad engineer. I am not a great engineer though. My “winning” streak came to a closed shortly after William Grant started working on Launchpad full time; he soundly trounced me over several months. Then he and I were put on the same squad and asked to fix critical bugs. Purple also had Jon Sackett, who was closing almost 2 times the number bugs than the other engineers. I don’t think I need to be humble on this matter. To use the vulgar, we rocked! Ian was the odd man on the Purple squad. He was the slowest bug closer, often going beyond our intended scope to fix an issue. Then Purple switched to feature work…Ian lept to the first rank while the rest of the squad struggled. Ian fixed almost double the number of Disclosure bugs than other squad members. The leading critical bug closer on the squad at the moment though is Steve Kowalik. This is his first time working on maintenance. His productivity has jumped since transitioning to maintenance.

I can only speculate as to why some engineers are better at maintenance, or can just close more bugs than others. A maintenance engineer must be familiar with the code and the rules that guide it. Feature engineers need to analyse issues and create new rules to guide code. I did not gradually become a leading bug closer, it happen in a single day when I realised while solving one issue that the code I was looking at was flawed, it certainly was causing a bug, I knew how to fix it, and with a few extra hours of extra effort, I could close two bugs in a single day. Closing bugs has always been easy since that moment.

I believe the Purple squad values certainty over severity and small scope over large scope when choosing which critical backlog bugs to fix. I created several charts that break the critical bugs into smaller categories. I suggested the squad burn down sub-categories of bugs like regressions, or 404s. The squad members are instead fixing bugs from the entire backlog. They are choosing bugs that they are certain they can fix in a few hours.  I think the squad has tacitly agreed to fix bugs that are less than a day of effort. When this group is exhausted, they will fix issues that require days of effort, but also fix as many bugs. The last bugs to be fixed will be those that require many days to fix a single bug. Fixing the bugs with the highest certainty reduced our churn through the critical bugs, there are fewer to triage, to dupe, to get ready to code.

The Purple squad avoids doing feature-level design and effort to fix critical bugs. Feature-level efforts entail more risk, more planning, and much more time. There is often no guarantee, low certainty, that a feature will fix the issue. A faster change with higher certainty can fix the issue, but leaves cruft in the code that the engineers do not like. Choosing to do feature-level fixes when a more certain fix is available indicates there is tension between the Launchpad users who have a “critical” issue that stops them from using Launchpad, and the engineers who have a “high” issue maintaining mediocre code. I contend it is easier to do feature-level work when you are not interrupted with maintenance issues. When the Purple squad does choose to do feature-level work to fix a critical, they have a list of the bugs they expect to fix, and they cut scope when fixing a single bug delays the fix of the others. The Launchpad Answers email subsystem was re-written when other options were not viable, there we about 20 leading timeouts represented by 5 specific bugs to justify 10 days of effort to fix them.

The Purple squad is not unique

Nothing that I have written explains why the Purple squad are better are closing critical bugs. All squads have roughly the same skills and make decisions like Purple. Maybe the issue is just a matter of degree. If the maintenance squad is not closing enough bugs to burn down the backlog, their time is consumed by triaging and duping new critical bug reports. Familiarity with Launchpad’s 1000’s of bugs is an advantage when triaging bugs and getting a bug ready to code. Being able to test queries yourself on a production-level database takes hours or days off the time needed to fix an issue. Familiarity with the code and the reasoning that guided it increases the certainty of success. The only domain that Purple is not comfortable working with is lp.translations; the squad is comfortable changing 90% of Launchpad’s code. There may be correlation between familiarity with code, and the facts that the squad members participated in the apocalypse that  re-organised the code base, and that some have a LoC credit count in the 1000’s.

Read more
Matthew Revell

On the 5th of October we’ll be ending our beta of Mercurial imports in Launchpad. On that day your existing Mercurial imports will cease and you won’t be able to create new ones.

This doesn’t affect Bazaar, Git, Subversion or CVS imports.

You’re probably wondering why. During the beta, we found that not many people wanted to import Mercurial branches into Launchpad. Today there are only around forty people using the facility. It’s also fair to say that our importer wasn’t of the quality we want for Launchpad.

So, with low demand for the feature we decided to focus engineering effort elsewhere rather than continue to maintain, or fix up, a less than satisfactory feature.

I’m sorry if you currently rely on Launchpad to import code from Mercurial into Bazaar. You can, though, still use the bzr-hg plugin locally.

Read more
Richard Harding

Network graph of the combo loaded JavaScript.

Updated network graph

Back in January a side project was started to update the JavaScript used in Launchpad. Launchpad has been using YUI 3.3.0 for a long time, very successfully, however recent advances in YUI 3.5 and higher have added some great tools for development that Launchpad can take advantage of. In order to facilitate easier upgrades our YUI library version Launchpad has been moved to using a combo loader for serving out JavaScript.

This means, that instead of a single launchpad.js file that can be upwards of 3MB in size, each request builds a list of JavaScript modules needed for the current page to work, and the combo loader only sends down those modules. This drastically cuts down on the download size of the JavaScript for users. These combo loaded JavaScript files are also cached for speedy serving to other users of Launchpad.

The combo loader also allows us to specify which YUI version to load via a tweak to the url. In this way we can easily test new version of YUI side by side with the current stable version as they come out. This allows Launchpad to keep with future YUI released much faster.

We’re excited that today Launchpad has moved from YUI 3.3.0 to 3.5.1 and is now served by the combo loader. This change provides a faster experience for users along with easier maintenance and new JavaScript library features for developers.

We’ve still got more to do though. YUI just released version 3.7 and we aim to push that into production faster than ever before. Please let us know how these changes work for you.

Launchpad also wants to thank the folks over at YUI for continuing the great work on a tool that Launchpad heavily depends on.

Read more
Laura czajkowski

Parallel testing is live

One of the projects the Launchpad Squads (yellow) have been working on has been the Parallel Testing during the last cycle, this has now been completed and is now in operation. WebOps have today finished setting up parallel testing in buildbot. Buildbot-poll has been updated to know about the new builders, and the developers have  confirmed that [testfix] and automatic stable merging etc. work fine. Nothing should have changed except that builds now take 35 minutes rather than 6.5 hours.

If something goes wrong, and may be helpful. The buildbot master is still praseodymium, but the slaves are new: sluagh for devel, and radande for db-devel. If you need packages upgraded on the slaves, poke WebOps as before.

If you would like to follow the discussion on this topic you’ll find more on the Launchpad development mailing list

Read more
Deryck Hodge

To go along with recent work to enable information sharing for bugs and branches, we are now enabling privacy for blueprints for beta testers. This means that blueprints now support some of the different information types that bugs and branches also support. For projects with a commercial subscription on Launchpad, this means blueprints can now be set to proprietary or embargoed. Project owners can also manage sharing for blueprints from their project’s sharing details page. For more on how sharing itself works, see Curtis’ blog post that announced that Information sharing is now in beta for everyone.

We have some minor fit-n-finish issues to complete, like nicer UI elements, and of those, we have one last known bug in progress — we know that blueprints don’t currently honor the sharing policy default when new blueprints are created. However, we thought it was worth getting this work to beta testers now to start getting feedback on this as we turn to finishing off the privacy work that is left to do.

Enjoy privacy for blueprints, beta testers! And please file bugs on any issues you find.

Read more
Laura czajkowski

Launchpad Builders update

We have recovered some of the affected builders, more will be coming back on later this week with the remainder to come in a few weeks when we have new hardware.  Until then launchpad builders will be at a reduced build capacity. l

We apologise for the inconvenience and we’re sorry for the disruption to your service.

Read more
Laura czajkowski

Reduced Builder Capacity

The Launchpad builders are currently operating at reduced build capacity. We are aware of the issue and have raised this with IS who are investigating the situation.  If you are having any issues with your builds please ask for help in #launchpad.

We apologise for the inconvenience and we’re sorry for the disruption to your service.

Read more
Curtis Hovey

Launchpad’s bug and branch privacy features are being replaced by information sharing that permits project maintainers to share kinds of information with people at the project level. No one needs to manage bug and branch subscriptions to ensure trusted users have access to confidential information.

Maintainers can share and unshare their project with people

Project maintainers and drivers can see the “Sharing” link on their project’s front page. The page lists every user and team that the project shares with. During the transition period of the beta, you might see many users with “Some” access to “Private Security” or “Private” user information. They have this access because they are subscribed to bugs and branches. Maintainers can unshare with users who do not need access to any confidential information, or just unshare a bug or branch with a user. Maintainers can share share with a team to give them full access to one or more kinds of confidential information.

I have prepared a video that demonstrates the features (my apologies for the flickering)

Commercial projects can set bug and branch policies

Projects with commercial subscriptions can also change bug and branch sharing policies to set the default information type of a bug or branch, and control what types they may be changed to. Maintainers can set policies that ensure that bugs and branches are proprietary, and only proprietary, to ensure confidential information is never disclosed.

Sharing can be managed using API scripts

I maintain many project which have a lot of private bugs and branches. The sharing page lists a lot of people, too many to read quickly. I know most work for my organisation, but I don’t even know everyone in my organisation. So I wrote a Launchpad API script that can be run by any project maintainer to share the project with a team, then unshare with the team members. The members still have access to the bugs and branches and their subscriptions still work, but they will lose access to my project when they leave the team. This arrangement makes it very easy to manage who has access to my projects. is run with the name of the team and a list of projects to share with it.

./ my-team project1 project2

Read more