Canonical Voices

Posts tagged with 'code'

Colin Watson

It’s been a while since we posted much regularly on this team blog, not least because for a while Launchpad was running more or less in maintenance mode.  That’s no longer the case and we’re back to the point where we can do feature development work again, as exemplified by our recent addition of Git code hosting support.

Lots of other things have been happening in the Launchpad world lately, though, and the half-way point in the year seems like a good time to start talking about them.  I’m going to try to do this a bit more regularly, aiming for about once a month when we also update our internal stakeholders.  This post covers roughly the last three months.

Git

Of course, you don’t get to release a major new feature and then have everything be perfect.  In fact, we’d released it pretty much as soon as we had a minimum viable product, so we knew there was plenty more to do.

Now that the basics work reasonably well, we’ve been focusing on tying off loose ends, and on getting to the point where Launchpad itself can be self-hosted in Git: this is partly because most of the things we need to do for that are relevant to many other projects considering a migration, partly because that ensures that any problems with it will affect Launchpad developers directly, and partly because this will allow us to trim a very large amount of database cruft associated with the representation of Launchpad’s own branches.

Here’s a brief changelog since our initial public announcement:

  • Leaving the “Target reference path” field empty when proposing a Git-based merge proposal no longer crashes (#1451068)
  • Git-based merge proposals can be deleted
  • Pushing to private repositories no longer hangs (#1451107)
  • The backend now stores reflogs so that we can help users recover from mistakes
  • Pushing a new default repository for a project now sets the repository owner to the project owner rather than to the person performing the push
  • The project +sharing view now handles Git repositories
  • Branches of a project default repository are now shown on the project’s code page, although we expect to be making significant further improvements in this area soon
  • Individual Git commits under “Recent Commits” now link directly to a full view of the corresponding commit
  • Merge proposal listings now cover Git (#1453020)
  • Branches in private repositories now show a privacy banner (#1457553)
  • Repositories can now be deleted (#1456583)
  • Private repositories are now browseable and show a privacy banner in the browsing interface
  • The URLs for Git-based merge proposals have moved to be directly under the source repository rather than under the source branch, so that historical MPs can still be accessed after deleting the branch (#1456589)
  • Launchpad now sends mail for Git-based merge proposals
  • There are various new Git repository listing views, chiefly for projects, packages, and people
  • Repositories now have a complete edit form allowing you to rename them, move them between targets, and set their default branch (“HEAD”) (#1456625)
  • We’ve exported a few more bits and pieces on the webservice API to support Git merge robots
  • We’ve brought various bits of automation for Git-based merge proposals up to par with their Bazaar equivalents: when a Git branch is updated, Launchpad now automatically updates the preview diffs for any MPs where that is the source branch, and detects whether any MPs where that is the target branch can now be marked as Merged
  • The Code settings view for projects now allows you to set the project’s default Git repository and to choose whether the default revision system for the project is Bazaar or Git

We’ve started on support for webhooks, and hope to be able to tell you more about this soon. Git-to-Git repository mirroring and native Git recipes are also frequent requests that we know are important to people, and we hope to get to those soon.

Other code review changes

Many of the changes needed for Git touched on our code review infrastructure, but we’ve been making some other improvements there too.  Launchpad has supported “inline comments” on merge proposals for a year or so now, allowing you to comment on parts of a diff without having to manually quote it; but the feature hadn’t been quite finished and lots of people had nits to pick with it.  We’ve made a couple of improvements here lately:

  • Preview diffs in merge proposals now have keyboard navigation to move between files, diff hunks, and inline comments
  • Links to diffs now include a per-file diffstat under an expander triangle
  • The mails sent when a reviewer adds inline comments now only include relevant parts of the diff context rather than the entire diff, making them much easier to read and making it harder to miss comments (#1334577)

We threw in a couple of bonus features here as well.  You can now see side-by-side diffs for merge proposals as well as unified diffs (#917502): there’s a link at the top of each preview diff allowing you to switch it to side-by-side mode (we may add a user preference for this later).  And resubmitting a merge proposal now automatically preserves the commit message (#676769).

Package build infrastructure

When we first added support for building debug symbol packages (ddebs) in Ubuntu, we handled the build and publication side of things with a temporary hack involving a custom hack to our sbuild fork that stashed the ddebs on the builder for a while, and a job that periodically fetched them to ddebs.ubuntu.com. As is the way of temporary hacks, it proceeded to stay that way for eight and a half years. It mostly worked, but was rather fragile, contributed to tying us to an old sbuild, wasn’t very extensible to PPAs, and couldn’t work with virtualised builders where all the builder’s state is reset before every build.

A couple of years ago we put together the Launchpad changes required to store ddebs straightforwardly in Launchpad instead, but weren’t able to deploy this at the time due to needing better librarian infrastructure. We finally got all this cleared out in April and deployed the new ddeb publishing mechanism. The main user-visible effect of this is that ddebs.ubuntu.com isn’t liable to lose packages any more, and ddebs are also available from Launchpad build pages.

Having done this, we were able to upgrade from our 2004-era fork of sbuild to a modern version, which fixed a number of long-standing bugs.

All architectures are now optional for PPAs (#1244868), so, for example, a PPA that only needs to support armhf can save builder cycles by disabling amd64 and i386. At the moment this can only be done by Launchpad administrators, so ask a question on Launchpad if you think this would be worth doing for any of your PPAs.

We rewrote the way that we install build-dependencies for source package recipes. This allows recipes to handle the “:native” qualifier in build-dependencies which is used in some places as part of supporting cross-building.

The next step in this modernisation programme will be to move all builds into ScalingStack and decommission the corresponding bare-metal builders, which we’ll be starting soon and working on an architecture-by-architecture basis as that cloud gains support for them. In the short term, this will give us better build capacity on the migrated architectures; in the longer term, it will let us support PPA builds on more architectures, and erase the cumbersome distinction between “non-virtualised” and “virtualised” builders. More news on this as it happens.

Performance

We upgraded to significantly more modern database servers earlier this year, which meant that a lot of hitherto difficult timeout problems suddenly either disappeared or became tractable. There are still some bad queries and a few mysterious problems, but as a general trend our timeout rates are very significantly down from where they were six months ago. We have some more general plans in this area, and will continue to spot-fix bad pages as they show up and as time permits.

Epilogue

We have lots more to do across the whole application, but still have a rather limited number of developers. If any of this kind of thing sounds interesting and you’d like to help, you can!

Read more
William Grant

Today we’re announcing early support for hosting Git repositories directly on Launchpad, as well as or instead of Bazaar branches. This has been by far the single most commonly requested feature from Launchpad code hosting for a long time; we’ve been working hard on it for several months now, and we’re very happy to be able to release it for general use.

This is distinct from the facility to import code from Git (and some other systems) into Bazaar that Launchpad has included for many years. Code imports are useful to aggregate information from all over the free software ecosystem in a unified way, which has always been one of the primary goals of Launchpad, and in the future we may add the facility to import code into Git as well. However, what we’re releasing today is native support: you can use git push to upload code to Launchpad, and your users and collaborators can use git clone to download it, in the same kind of way that you can with any Git server.

Our support is still in its early stages, and we still have several features to add to bring it up to parity with Bazaar hosting in Launchpad, as well as generally making it easier and more pleasant to use. We’ve released it before it’s completely polished because many people are clamouring to be able to use it and we’re ready to let you all do so. From here on in, we’ll be adding features, applying polish, and fixing bugs using Launchpad’s normal iterative deployment process: changes will be rolled out to production once they’re ready, so you’ll see the UI gradually improving over time.

What’s supported?

You can:

  • push Git repositories to Launchpad over SSH
  • clone repositories over git://, SSH, or HTTPS
  • see summary information on repositories and the branches they contain in the Launchpad web UI
  • follow links from the Launchpad web UI to a full-featured code browser (cgit)
  • push and clone private repositories, if you have a commercial subscription to Launchpad
  • propose merges from one branch to another, including in a different repository, provided that they are against the same project or package

What will be supported later?

Launchpad’s Bazaar support has grown many features over the years, and it will take some time to bring our Git support up to full parity with it and beyond. Git repositories use a somewhat different model from Bazaar branches, which we’ve had to account for in many places, and some facilities will require complete reimplementation before we can support them with Git.

Here’s an incomplete list of some of the features we hope to add:

  • useful subscriptions (currently only attribute change notifications work, which are not usually very interesting in themselves)
  • RSS feeds
  • mirroring
  • webhooks
  • an integrated code browser

See our help page for more known issues and instructions on using Launchpad with Git.

Helping out

This is a new service, and we welcome your feedback: you can ask questions in #launchpad on freenode IRC, on our launchpad-users mailing list, or on Launchpad Answers, and if you find a bug then please tell us about that too.

Launchpad is free software, licensed under the GNU AGPLv3. We’d be very happy to mentor people who want to help out with parts of this service, or to build things on top of it using our published API. Some preliminary documentation on this is on our developer wiki, and you can always contact us for help.

Read more
Matthew Revell

On the 5th of October we’ll be ending our beta of Mercurial imports in Launchpad. On that day your existing Mercurial imports will cease and you won’t be able to create new ones.

This doesn’t affect Bazaar, Git, Subversion or CVS imports.

You’re probably wondering why. During the beta, we found that not many people wanted to import Mercurial branches into Launchpad. Today there are only around forty people using the facility. It’s also fair to say that our importer wasn’t of the quality we want for Launchpad.

So, with low demand for the feature we decided to focus engineering effort elsewhere rather than continue to maintain, or fix up, a less than satisfactory feature.

I’m sorry if you currently rely on Launchpad to import code from Mercurial into Bazaar. You can, though, still use the bzr-hg plugin locally.

Read more
Curtis Hovey

We are reimagining the nature of privacy in Launchpad. The goal of the disclosure feature is to introduce true private projects, and we are reconciling the contradictory implementations of privacy in bugs and branches.

Launchpad will use policies instead of roles to govern who has access to a kind of privacy

We are implementing three kinds of policies, proprietary, embargoed security, and user data. The maintainer is the default member of these policies. The maintainer can share a kind or private data by adding a user or team to a policy.

For proprietary projects, the maintainer can add their organisational teams to the proprietary policy to share all the project information with the team members.

For Ubuntu, the maintainer will set the apport bot to be the only user in the user data policy; user data is only shared with a bot that removes personal data so that the bug can be made public. The Ubuntu security team will be the only users in the security policy.

Most maintainers will want to add project drivers to the policies if they use drivers. Bug supervisors can be added as well, though the team must be exclusive (moderated or restricted).

You can still subscribe a user or team to a private bug or branch and Launchpad will also permit the user to access it without sharing everything with the user. The existing behaviour will continue to work but it will be an exception to the normal rules.

Polices replace the bug-subscription-on-privacy-change rules. If you have every had to change the bug supervisor for a project with many private bugs, you can rejoice. You will not need to manually update the subscriptions to the private bugs to do what Launchpad implied would happen. Subscriptions are just about notification. You will not be notified of proprietary changes is proprietary information is not shared with you. Sharing kinds or information via policy means that many existing private bugs without subscribers will finally be visible to project members who can fix the issue.

Read more
Curtis Hovey

We are reimagining the nature of privacy in Launchpad. The goal of the disclosure feature is to introduce true private projects, and we are reconciling the contradictory implementations of privacy in bugs and branches.

We are adding a new kind of privacy called “Proprietary” which will work differently than the current forms of privacy.

The information in proprietary data is not shared between projects. The conversations, client, customer, partner, company, and organisation data are held in confidence. proprietary information is unlikely to every be made public.

Many projects currently have private bugs and branches because they contain proprietary information. We expect to change these bugs from generic private to proprietary. We know that private bugs and branches that belong to projects that have only a proprietary license are intended to be proprietary. We will not change bugs that are public, described as security, or are shared with another project.

This point is a subtle change from what I have spoken and written previously. We are not changing the current forms of privacy. We do not assume that all private things are proprietary. We are adding a new kind of privacy that cannot be shared with other projects to ensure the information is not disclosed.

Launchpad currently permits projects to have default private bugs and branches. These features exist for proprietary projects. We will change the APIs to clarify this. eg:

    project.private_bugs = True  => project.default_proprietary_bugs = True
    project.setBranchVisibilityTeamPolicy(FORBIDDEN) => project.default_proprietary_branches = True

Projects with commercial subscriptions will get the “proprietary” classification. Project contributors will be able to classify their bugs and branches as proprietary. The maintainers will be able to enable default proprietary bugs and branches.

Next part: Launchpad will use policies instead of roles to govern who has access to a kind of privacy.

Read more
Curtis Hovey

We are reimagining the nature of privacy in Launchpad. The goal of the disclosure feature is to introduce true private projects, and we are reconciling the contradictory implementations of privacy in bugs and branches.

We must change the UI to accommodate the a kind of privacy, and we must change some existing terms because to avoid confusion.

We currently have two checkboxes, Private and Security that create 4 combined states:

  • Public
  • Public Security
  • Private Security
  • Private something else

Most private bugs in Launchpad are private because they contain user data. You might think at first that something that is just private is proprietary. This is not the so. Ubuntu took advantage of defects in Launchpad’s conflation of subscription and access to address a kind of privacy we did not plan for. Most private bugs in Launchpad are owned by Ubuntu. They were created by the apport bug reporting process and may contain personal user data. These bugs cannot be made public until they are redacted or purged of user data. We reviewed a sample of private bugs that belong to public projects and discovered more than 90% were made private because they contained user data. Since project contributors cannot hide or edit bug comments, they chose to make the bug private to protect the user. Well done. Launchpad needs to clarify when something contains user data so that everyone knows that it cannot
be made public without removing the personal information.

Public and private security bugs represent two states in a workflow. The goal of every security bug is to be resolved, then made public so that users are informed. People who work on these issues do not use ”public” and “private”, they use “unembargoed” and “embargoed”.

Also, when I view something that is private, Launchpad needs to tell me why. The red privacy banner shown on Launchpad pages must tell me why something is private. Is it because the page contains user data, proprietary information, or an embargoed security issue? This informs me if the thing could become public.

When I want to change somethings visibility, I expect Launchpad to show me a choice that clearly states my options. Launchpad’s pickers currently shows me a term without an explanation, yet Launchpad’s code does contain the term’s definition. Instead of making me search help.launchpad.net (in vain), the picker must inform me. Given the risks of disclosing personal user data or proprietary information, I think an informative picker is essential. I expect to see something like this when I open the visibility picker for a bug:

Branches require a similar, if not identical way of describing their kind of information. I am not certain branches contain user data, but if one did, it would be clear that the branch should not be visible to everyone and should not be merged until the user data is removed.

Next post: We are adding a new kind of privacy called “Proprietary” which will work differently than the current forms of privacy.

Read more
Curtis Hovey

Launchpad is ending support for multi-tenancy for branches and bugs to ensure that projects can manage the disclosure of private information. This is a fundamental change to how launchpad permits communities to share projects. Very few users will be affected by this change, but several communities will need to change how they work with private bugs and branches.

Launchpad currently permits users to create private bugs or branches that cannot be seen by the project maintainers, or the project’s other communities. This feature permits communities and companies to develop features in secret until they are ready to share their work with the other communities. This exclusivity feature is difficult to use, difficult to maintain, and makes Launchpad slow. This feature also contradicts the project maintainer’s need to be informed and to manage the disclosure of confidential information. When multiple parties can control privacy on a project, important information is lost.

While discussing the proposed changes with Launchpad stakeholders, I was surprised that most believed that the project maintainers could see the private bugs they were reporting — they wanted to collaborate, but the subscription-as-access mechanism is faulty. There are thousands of private bugs reported in Launchpad that cannot be fixed because the person who can fix the issue is not subscribed.

A contributing reason to drop support for private branches on project you do not maintain is that the feature is fundamentally broken. Privacy is inherited from the base branch. If you cannot access the base branch your branch is stacked on, you are locked out. Project owners can, and accidentally do, lock out contributors. You cannot subscribe someone to review the security concerns in your branch if that user does not have access to the base branch. Project contributors must subscribe each other to their respective branches to collaborate on a fix or feature.

This change is a part of a super-feature called Disclosure. To ensure that confidential data is not accidentally disclosed, project maintainers will be able to view and change who has access to confidential project information. Maintainers can add users to security or proprietary policies to grant access to all the information in the respective policy. You will not need to subscribe users to individual bugs or branches, unless you want to grant an exception to a user to access one confidential piece of information.

Read more
Raphaël Badin

Edit 2011-11-15 08:18 UTC: The problem is now fixed and we’ve re-enabled the new menu.

Edit 2011-11-11 13:42 UTC: We’ve temporarily disabled the new menu while we fix some unfortunate side effect.

We’ve just deployed a new, simplified version of the branch menu displayed on the right hand side of personal code pages (e.g. personal page for the Launchpad team). It looks like this:

Old menu

New menu

Calculating the number of branches took way too much time for people/teams with a huge number of branches (e.g. https://code.launchpad.net/~ubuntu-branches), up to the point that they were getting timeouts.

The new design, along with optimisations we’ve made to the database queries, should improve performance for everyone.

Read more
Martin Pool

We’ve just upgraded Launchpad’s builder machines to Bazaar 2.4. Most importantly, this means that recipe builds of very large trees will work reliably, such as the daily builds of the Linaro ARM-optimized gcc. (This was bug 746822 in Launchpad).

We are going to do some further rollouts over the next week to improve supportability of recipe builds, support building non-native packages, handle muiltiarch package dependencies, improve the buildd deployment story etc.

Read more
Aaron Bentley

I recently posted about Initializing page JavaScript from the JSONCache. Now I’m pleased to announce that you can also get updated copies of the IJSONRequestCache, to make it easier to update your page.

Brad Crittenden and I started work on this at the Dublin Thunderdome, and it’s finally been deployed. What this means is that for basically any page on Launchpad, you can append /++model++ to the URL, to get a fresh copy of the IJSONRequestCache. With ++model++, a change will typically require only two roundtrips; one to make a change, and one to retrieve an updated model. Future work may reduce this to a single roundtrip.

Why ++model++, not ++cache++? Cache is a really poor name for what the IJSONRequestCache is. Rather than providing fast access to whatever data has been previously retrieved, it is a complete collection of all the relevant data.

In Launchpad, the IJSonRequestCache is associated with the view, so we’re trying to rebrand it as the “view model”. This may seem strange from an MVC (Model, View Controller) perspective, but MVC can be recursive. A view may use a model to render itself.

Read more
Matthew Revell

Source package recipes

A pint of ale

Here’s a quick pub quiz:

Question: How do you make packages for Ubuntu?

You can choose from the following answers:

  1. learn Debian packaging through hours of study and practice
  2. borrow existing packaging from elsewhere, throw a couple of Bazaar branches together and let Launchpad handle the rest
  3. Uruguay in both 1930 and 1950.

If you selected either of the first answers you’d be right.

Okay, so, if you want to do it for real — i.e. become an Ubuntu MOTU or otherwise create Debian-style packages from scratch — then you still need to go through the hard work.

However, for everyone else who really just needs to get something out there and working for, say, a group of beta testers, we now have Launchpad’s source package recipes.

How it works, in three steps

It’s almost ridiculously easy to set up a source package build:

  1. Choose a branch in Launchpad, whether hosted directly or imported.
  2. Write a short recipe that tells Launchpad which other branches to pull in, for example to provide packaging or make the code buildable.
  3. Paste your recipe into Launchpad.

And that’s it. Within a few minutes you can set up a daily build direct from your trunk or any other buildable branch in Launchpad.

Watch how it works in our screencast:

An example

Alvin Hall

Let’s say you’re the developer of a home finance application called Alvin. You track your project’s code using Git and host it on your own server. For the past couple of years Alvin has been packaged in the Ubuntu universe and your trunk has also been imported from Git to a Bazaar branch in Launchpad at lp:alvin.

Just as you’re approaching Alvin’s next release, you want to get some wider testing. In the past, you’ve published a nightly tarball and provided instructions on manual installation. That’s given you a handful of dedicated beta testers but you’re worried that you’re asking too much of people.

With Launchpad’s source package recipes, you write a short recipe that pulls in your trunk branch, adds the packaging from Alvin’s existing Ubuntu package and then builds an installable Ubuntu package in the PPA of your choice:


# bzr-builder format 0.3 deb-version 2.0beta+{revno}
lp:alvin
nest-part packaging lp:ubuntu/alvin debian debian

Paste the recipe into Launchpad and with a couple of clicks you have a daily build of your trunk, that’s published as an Ubuntu package in your PPA.

Now you can ask people to test the latest Alvin code by doing no more than adding your PPA to their system. Launchpad will build a new version of the package on each day it spots a change in your trunk (or the Ubuntu packaging). For your beta testers, any changes will show up just like any other Ubuntu update.

Simple as that!

Here’s a quick recap of how it works: you can take any buildable branch — whether hosted in directly Launchpad or imported from Git, Subversion, CVS or Bazaar hosted elsewhere — merge or nest other branches, add packaging and then leave it to Launchpad to create a daily build that it publishes in your chosen PPA.

Seeing it in action

List of daily builds in Launchpad

During the beta, people added a whole range of source package recipes, with a list of more than 350 daily builds as I write this.

Daily builds on Launchpad right now include Project Neon, who have around sixty recipes providing daily builds of KDE and Amarok. There are also daily builds of the Scribus DTP app, Audacity and the scriptable screen reader Gnome Orca.

Try it out

It’s easy to get your own source package recipes and daily builds up and running.

Start at our Getting Started guide and screencast.

I’ll leave the last word to Luke Benstead, who has been using source package recipes while developing a set of game libraries:

I’ve been using LP to develop some small open source game libraries. Because there are quite a few of them, packaging them all is a pain, so the package builds have worked out pretty well for them.

Now I get nightly builds delivered to a PPA, so I know that if I fix a bug it’s reflected to all my machines. And my recipes are only a single line so they’ve been really easy to use. I’m not really sure how they could be easier.

Images:
Beer photo by dearbarbie. CC-BY-SA.
Alvin Hall photo by Phil Guest. CC-BY-SA.

Read more
Julian Edwards

Continuing with the recent improvements to the build farm – Jelmer has made another massive one.

The last major scalability problem that we had was one where the whole farm was blocked when a single builder was ready to upload a build. In the case of large packages, like the kernel, the manager process could block for over a minute while it waited for the upload processor to unpack the package and verify its contents.

Jelmer’s work has decoupled the upload processing from the build farm manager process. What happens now is that the files collected from the builder are thrown into a staging queue area and then the manager process immediately continues with polling the rest of the builders, unblocked. A cron job will then process the builder upload queue at 1 minute intervals.

You can see the dramatic effect this has had on the overall queue for the PPA builders here:
Build Farm Queue Size

This is quite an incredible improvement as you can see! But we’re not stopping there, we’re currently doing a massive refactoring of the builder dispatching code so it’s all fully asynchronous. When this is all done we’re going to be in superb shape to support an increase in load that’s anticipated from the increasing number of people using the package recipes.

Read more
Matthew Revell

As I’ve said before, Launchpad is pretty big. Getting to know everything you can to do in Launchpad can take a while.

Of course, there are the user guide, the tour and the dev wiki even has a feature check-list. It’s still easy to miss things.

So, I’m turning the fifth day of each week into Feature Friday :)

I’m gonna kick off with something I use a lot and that, to be honest, is more a Bazaar feature, than a Launchpad feature:

bzr commit -m "Adds email functionality to the client, thereby obeying Zawinski's law." --fixes lp:1234

Adding --fixes lp:1234 to a commit tells Bazaar that the branch contains a fix for bug 1234 tracked in Launchpad.

The next time you push the branch to Launchpad, Launchpad will create a link between the branch and bug 1234.

Read more
Matthew Revell

Tim has made a couple of improvements to the way that comments are displayed on merge proposals.

The first is pretty simple but makes it far easier to see which merge proposals need attention. Up until now, if you gave your merge proposal a description it would show up as a comment. This made it harder to see which merge proposals had zero activity.

Now, your description no longer shows up as a comment. And you can edit your description in-line using the same Javascript editor as you’ll find on bug pages.

The other change affects merge proposals that supersede another merge proposal. The merge proposal page explicitly states that it supersedes a previous proposal.

Notice showing the merge proposal supersedes another

It also lists all the comments from that previous review, giving them a yellow background and a link to the superseded review so you can easily tell them apart from comments on the current review.

A comment from the superseded merge proposal

Read more