Canonical Voices

What How Bazaar talks about

Posts tagged with 'launchpad'

Tim Penhey (thumper)

Launchpad and stacked branches

As I'm sure most of you are aware, Launchpad hosts Bazaar branches. One early design decision that we had on Launchpad was that branches should be able to be renamed, moved between people and projects without limitation. This is one reason why each branch that was pushed to Launchpad had a complete history. We wanted to make sure that there weren't any problems where one person was blocking another pushing branches, or that people weren't able to get at revisions that they shouldn't be able to.

The Bazaar library, bzrlib, gives you awesome power to manipulate the internals giving you access to the repository of revisions through the branch. This can be a blessing and a curse, as if you have private revisions, then they can't be in a public repository.

Having a complete copy of the data for every branch became a severe limitation, especially for larger projects, of which Launchpad itself is one. A solution to this was a change in Bazaar itself that allowed a fallback repository which contained some of the revisions. This is what we call stacked branches. The repository for the branch on Launchpad has a fallback to another repository, which is linked to a different Launchpad branch. We ideally wanted all of this to be entirely transparent to the users of Launchpad. What it means is that when you are pushing a new branch to Launchpad, the bzr client asks for a stacked location. If there is a development focus branch specified for the project, this is then offered back to the client. The new branch then only adds revisions to its repository that don't exist in the development focus branch's repository. This makes for faster pushes, and smaller server side repositories.

The problem though was what do we specify the stacked on location to be? When we created the feature, we used absolute paths from the transport root. What the mean was that we stored the path aspect of the branch. For example, lp:wikkid gets translated to bzr+ssh://bazaar.launchpad.net/~wikkid/wikkid/trunk or http://bazaar.launchpad.net/~wikkid/wikkid/trunk depending on whether the bzr client knows your Launchpad id. The absolute path stored would be /~wikkid/wikkid/trunk. This information was then stored in the branch data on the file system.

The problem however was that the web interface allows you to rename branches. The actual branch itself on disk is referred to using a database id, which is hidden from the user using a virtual file system which has rewrite rules for http and at the bazaar transport level. However since the stacked on location refers to a full branch path, changing any part of that, whether it is the branch owner, branch name, or the project or package that the branch is for, would cause any branches stacked on that changed branch to break, bug 377519.

In order to fix this we had to change the location that the branch is stacked on to be independent of the branch path. The best solution here is to use the database id. I really didn't want to expose the user to this opaque id, but one opaque id is as good as another. Now when pushing branches to Launchpad, when it is creating a stacked branch you'll see a message like:

Created new stacked branch referring to /+branch-id/317141.

Existing branches still have their old branch paths saved for now. We'll run a migration script early next week to fix all these up, and hopefully we'll have seen the last of this bug.

Read more
Tim Penhey (thumper)

Launchpad code update

We've been very busy over the last couple of months with lots of changes that most people will never notice.

Reduced latency


Branches pushed to Launchpad are now immediately available over http to anonymous readers of the branch, which includes the loggerhead code browser.

Code review email changes


When proposing a branch for review the initial emails and subsequent comments will now come in order. Previously if someone commented before the script that generated the diff was run, the comment would be emailed out first, now it isn't.

Teams requested to review now get email


Everyone in the team that is requested to review will get email now. This is a blessing for those that want it, and almost a curse for those that aren't interested. Launchpad adds a number of email headers to help users with filtering of email. Here is an example from an email I received:


X-Launchpad-Message-Rationale: Reviewer @drizzle-developers
X-Launchpad-Notification-Type: code-review
X-Launchpad-Project: drizzle


Since it was a team that was requested to review, there is the @drizzle-developers added to the X-Launchpad-Message-Rationale. If I was personally asked to review, the header would just say Reviewer.

Build branch to archive


This was the original name of the feature, but it is more about recipes now. A recipe is a collection of instructions on how to build a source package. We are still testing this internally, but I'm hoping to get this enabled on edge very soon. This will be extended to add daily builds.

What does this really mean?

Lets say you want to have a daily build of a project, like gwibber. You would then create a recipe that uses trunk as a base branch, merge in the packaging info, and say "Please build this every day into my PPA". And Launchpad will.

Read more
Tim Penhey (thumper)

Trivial bugs

This is just a quick note really. One thing I've been trying to do more and more is to address simple bugs in a more timely manner.

I use the tag "trivial" to indicate to me that the bug is very simple to fix. By this I mean that I should be able to have the fix and the test all written in under an hour, and normally under 30 minutes.

Personally I'm (hopefully) fixing one trivial bug a day in addition to other work. This way the simple bugs get some attention, and I get the feeling of accomplishing something when other things are in the pipeline that take longer to get completed.

My scheduling of trivial bugs is somewhat arbitrary. Often the most recently commented on trivial bug will get my attention.

Read more
Tim Penhey (thumper)

Don't wait for perfection

Way back in July I was thinking of writing a post about the new branch listings in Launchpad. I was working on making branch listings for distributions and distroseries work, for example Package branches for Ubuntu and Packages branches for Ubuntu Lucid. But the code wasn't entirely optimised. Then as things happen, the optimisation got pushed back... and back. And finally when I did get the optimisation in, I didn't feel it was worthy of talking about.

I guess the thing to remember is: don't wait for perfection. Sure it wasn't perfect, but if more people were accessing the pages, the optimisation may have happened sooner.

One thing going on at the moment is more integration of the lazr-js widgets. The main merge proposal page now has in in-page multi-line editor for the commit message. Sure, it needs tweaking, but the main functionality is there. More ajaxy goodness is finding its way into Launchpad.

One of the things that I'm thinking about right now is splitting the concepts of code reviews and merge proposals. At the moment we almost use the term interchangeably, which does cause some confusion. I'd like to have the merge proposal reflect the meta-data and information around the intent to have work from one branch be landed or merged into another branch (normally the trunk branch), and the code review the conversation that goes on around the merge proposal. Merge proposals may have an associated code review, but right now, a code review must be associated with a merge proposal.

Associated with this, I'd like to extract some state information. Currently merge proposals have only one status, which really reflects two things. I'd like to break this out into two: review status; and merge status. Review status would be one of: work in progress; needs review; approved; rejected; superseded; and maybe withdrawn. Merge status would be one of: proposed; merged; queued; and merge failed. Queued relates to the merge queues which are currently partially implemented in the Launchpad codebase, and merge failed is the state that a proposal would be set to when a landing robot like PQM or Tarmac attempt to land the branch but it fails due to either merge conflicts or test failures.

My goal for the next six months it to write more often, talk about ideas more, and not wait for perfection.

Read more
Tim Penhey (thumper)

Breaking up work for review

It was Friday morning after three days of working on one feature. Last thing Thurday I counted the size of the change and it was over 1100 lines and I wasn't quite finished. I found myself in the situation that turns up periodically that I wanted to break up my work into cohesive reviewable chunks. Now it isn't a matter of taking commits x through y as chunk one, and so on, as the size grew organically as I changed what needed to be changed, and wrote what needed to be written without really stopping to think about the size of the change until it was done. However now it was done, I wanted to break it up.

Last time I did this, I used looms, but Aaron told me we could do it easily using his new Bazaar pipeline plugin. So I spent some time talking through with Aaron on how to do it, promising to write it up if it worked well. I must say that it was good. During the process we identified a number of enhancements to the plug in to make it even easier.

I'm going to show the progression we made, along with our thoughts. I have trimmed some of the output when I've decided that it doesn't add value.

The first thing I had to do was to get the pipeline plugin.

$ bzr branch lp:bzr-pipeline ~/.bazaar/plugins/pipeline


Unfortunately this seemed to clash slightly with the QBzr plugin. The were both trying to redefine merge. Personally I don't use QBzr and had probably just installed it to take a look, so I removed that plugin.

Caution: the pipeline plugin relies on switch so works with lightweight checkouts. This is how I work anyway, so I didn't have anything to do here, but if you work differently, YMMV.


The pipeline plugin is designed around having a set of branches one after (individual pipes) the other that perform a pipeline, clever eh? When you have the pipeline plugin, any branch is also considered a pipeline of one.

$ bzr show-pipeline
* nice-distribution-code-listing


What I was wanting to do was to break up this work into a number of distinct change sets, each that could be reviewed independently. We decided that the way to do this was to create a pipe before the current one, and bring changes in. This is done with the command add-pipe.

$ bzr add-pipe factory-tweaks --before nice-distribution-code-listing
$ bzr show-pipeline
factory-tweaks
* nice-distribution-code-listing


Right here we decided that there should be an easier way to add a pipe before the current pipe, as right now it needs a pipe name. A bug was filed to track this.

You can see from the show-pipeline command that the new pipe is before the current one. The pipeline plugin addes a number of branch aliases:

  • :first - the first pipe in the pipeline

  • :prev - the pipe before the current pipe

  • :next - the pipe after the current pipe

  • :last - the last pipe in the pipeline



Now to make the switch to the first pipe. Both :prev and :first refer to the same branch here, and I could have used either.

$ bzr switch :prev
... changed files shown
All changes applied successfully.
Now on revision 8747.


Now this pipe was added from the pipe after it, so it starts off with the same head revision. Not exactly the starting point I wanted, so we replaced the head of this branch with the last revision of the trunk branch that we had merged in.

$ bzr pull --overwrite -r submit:


The submit: alias refers to the submit branch. This is often trunk, and is in my project layout (specified using submit_branch in .bazaar/locations.conf).

Now the lower pipe was a copy of trunk. A good place to start adding changes I think. The next problem was how to get the changes from the following pipe into this one. Our first attempt was to merge in the following branch, shelve what we didn't want, throw away the actual merge, but keep the changed text, and commit.

$ bzr merge :next
$ bzr shelve
$ bzr stat
modified:
lib/lp/testing/factory.py
pending merge tips: (use -v to see all merge revisions)
Tim Penhey 2009-07-02 New view added.
$ bzr revert --forget-merges
$ bzr stat
modified:
lib/lp/testing/factory.py
$ bzr commit -m "More default args to factory methods and whitespace cleanup."


Now this seemed very convoluted. Why merge and then forget the merge? I seemed kinda icky, but it worked. The next thing to do is to merge these changes down the pipeline. This is done through another command pump.

$ bzr pump


This merges and commits the changes down the pipeline. If there are conflicts, it stops and leaves you in the conflicted pipe. This didn't occur here, nor did it occur for any of my other ones. Here you can see the commit message that pump used:

$ bzr switch :next
$ bzr log --line -r -1
8714: Tim Penhey 2009-07-03 [merge] Merged factory-tweaks into nice-distribution-code-listing.
$ bzr switch :prev


Now it was time to add the next pipe.

$ bzr add-pipe code-test-helpers
$ bzr show-pipeline
* factory-tweaks
code-test-helpers
nice-distribution-code-listing
$ bzr switch :next


This time, instead of merging in the changes, we shelved them in. The shelve command. The shelve command can apply changes from arbitrary revisions, and it also knows about files. The change that I wanted in this branch was a single added file, so I could tell shelve about that file.

$ bzr shelve -r branch::next lib/lp/code/tests/helpers.py
Selected changes:
+ lib/lp/code/tests/helpers.py
Changes shelved with id "2".


However the big problem with this is it all looks backwards. We are shelving from the future not the past. This really did my head in. Shelve would say "remove this file?" and by shelving it, it would add it in. It worked but made my head fuzzy. We filed a bug about this too. By adding a better way to take the changes, the command could do the reversal for you and provide you with a nicer question.

More of the same happened for the next few pipes, and I won't bore you with repeated commands.

On the whole, the pipeline plugin worked really well. I was able to break my work up into five hunks which could be reviewed easily. In the end I kept working on the branch that was my original, so all my original history remained intact. It would have been just as easy to add another pipe and take the remaining changes. This would have left me with five branches, each with one commit. This works well for the way we work as we have reviews based on branches. Each pipe could be pushed to Launchpad and a review initiated for it. With some more UI polish, I think pipelines will be even more awesome than I think they are now.

Read more
Tim Penhey (thumper)

launchpadlib updates for branches

Just a quick note to yet you know of a few changes to launchpadlib for branches. Mainly because I've removed a method that I know someone is using.

You used to be able to get to the branches for a project by saying my_project.branches, but I've removed this. It would have been nicer to deprecate it but we don't have a nice deprecation method right now for launchpadlib, and since it is still in beta, I didn't feel too bad.

The branches of a project was an attribute, now we have a getBranches method. The old attribute would give you all the branches of the project, including the merged and abandoned ones. The method defaults to give you the active branches, and allows you to pass in the statuses that you'd like to get.

Also with this change you can now get the branches for a project group, or the branches owned by a person using the same getBranches method call.

Project groups also grew the method getMergeProposals in the same way that the method was already available for people and projects.

Please file any bugs on the launchpad-code project on Launchpad.

Read more
Tim Penhey (thumper)

Code in Launchpad

Launchpad offers many things to developers, and open source software developers in particular. One of these things is the ability to host Bazaar branches. For those that have looked a little deeper, they will have noticed that there are four types of branches in Launchpad: Hosted; Mirrored; Remote; and Imported. Hmm, this isn't really what I was intending to talk about at all, but I'm going to go with the flow.

Hosted branches are those where Launchpad is the primary public location of the branch. Hosted branches are normally created by pushing a branch directly to Launchpad. Before you do that though, you need to have registered on Launchpad, and supplied an SSH key. This is how Launchpad knows who you are. There are two ways you can push a branch to Launchpad: one is via SFTP; and the other using the Bazaar smart server (bzr+ssh).

As an example I'm going to use my alias-command bzr branch. The complete SFTP location would be sftp://thumper@bazaar.launchpad.net/~thumper/bzr/alias-command, and the smart server one bzr+ssh://thumper@bazaar.launchpad.net/~thumper/bzr/alias-command. These are a bit unwieldy, so we extended the lp type urls for bzr to support writing if the launchpad plug-in knows who you are. In order for you to do this you use the lp-login command. bzr lp-login will tell you the username that is currently set. If you have not done this yet, you'll see a message like "No Launchpad user ID configured." I set mine by saying bzr lp-login thumper. This stores thumper as the launchpad_username in the bazaar.conf file. This also means I can use bzr push lp:~thumper/bzr/alias-command to push to my hosted Launchpad branch.

Mirrored branches allow you to have your branches stored publicly in some location that you control, and you let Launchpad know where this is. Launchpad will then update its copy of your branch every six hours. This is handy if you don't have an SSH key, or you have a slow network connection, or you just like having your branches available on your own server.

Remote branches are a bit different. Remote branches were sort of created out of necessity. Some people were registering mirrored branches with unreachable locations. Some of these were possibly by mistake, but quite a few were obviously inaccessible. But more strange is that those branches were linked to bugs or blueprints. There was obviously a desire to have branch meta-data there, but not actually allow Launchpad to get access to the branches. So we have remote branches. You cannot get a copy of a remote branch from Launchpad as Launchpad does not have a copy of it.

Imported branches are those branches where Launchpad get the code from either CVS or Subversion, and puts it into a Bazaar branch. I was really wanting to talk about this as I saw two projects recently where we are importing code that I didn't know about. One is my favourite music player, Amarok, and the other was MPlayer. Just out of curiosity I looked at both of these branches on Launchpad. The Amarok one has 12195 revisions as I'm writing this, and the last revision was 11 hours old, and MPlayer had even more revisions, at 26761. However that isn't even the cool bit. What is really nifty is you can go bzr branch lp:amarok or bzr branch lp:mplayer to get the code. Just to check I did just that, and got a copy of the amarok source. It was the first bit of C++ I had looked at in a long time (it used to be all I did).

Anyway, that was what I really wanted to say. Oh yeah, and bzr rocks.

Read more
Tim Penhey (thumper)

The Launchpad branch directory service

Recently Bazaar grew a branch directory service. This allows plug-in developers to define custom "protocols" that resolve the branch names into some other branch location.

The Launchpad plug-in defines a protocol "lp". Launchpad uses the other parts of the URL to relate to projects, series or individual's branches. The shortest valid URL for Launchpad is something like lp:do. You can also use an empty authority (or site part of a URL), so lp:///do is exactly the same, just longer. Personally I prefer the one without all the slashes.

The do part of the branch location relates to the GNOME Do project on Launchpad. There is a little magic (a.k.a. configuration) that is needed to make this work. Projects in Launchpad get an initial development focus series created for them. This is intended to relate to the branch of development that is where current or new work goes. In order to have the code available through the Launchpad directory service, the code has to be available through Launchpad as a normal branch.

Once a branch has been either hosted, mirrored or imported for the project, one of the people responsible for the project in Launchpad can relate the branch with the series. Once this is done the branch is easily accessible. People that have permission to make these links will be shown a link on the main code tab for their project (we don't taunt people who can't make the links with an invalid option).

If there is another series, say 1.0 for our project fooix, and we have branches associated with them, then we could get the 1.0 branch using lp:fooix/1.0. Normal Launchpad branches are also accessible using the lp protocol using lp:~username/project/branch-name.

Read more