Canonical Voices

Posts tagged with 'process'

Matt Fischer

I was trying to explain how our team did workflow to a former colleague last week and I so I started thinking about all the different workflows I’ve dealt with in my career. This one is by far my favorite, although I know it’s not git which everyone loves, I’m curious what workflows other groups use with launchpad. Take a look at this one and let me know, can our team do anything better, can yours?

First a brief note about our team at Canonical. We work on “premium” customer-facing projects, typically on ARM based hardware. We are downstream from Ubuntu for the most part, and although we do send fixes upstream when it makes sense, often we make customizations to packages that cannot go upstream. I’ll use a real-world example for this workflow explanation, we have a platform where we want to remove the user list, help menu entry, and the logout menu enty from the session indicator, so we needed to modify indicator-session to do so.

The tl;dr version of our workflow is Decentralized with shared mainline, with parts of Decentralized with automatic gatekeeper added.

Setup a Shared Master (mainline)

Grab the source for indicator-session for the distroseries we’re based on, precise in this case. We usually grab it from launchpad or apt-get source if launchpad’s precise copy is out of date. This code gets pushed to lp:~project-team/project/indicator-session. This is now the master/mainline version. Everyone on the team has write access to this, provided they follow team rules.

Setting Up My Local Branch

I have a pbuilder already setup for our project usually, so my first step is to setup my local tree. I like to use a two level hierarchy here so that builds don’t “pollute” my main project area where I have dozens of different branches checked out. So I setup a subdirectory and checkout a copy to master.

cd ~/Projects/project-precise-amd64
mkdir indicator-session
cd indicator-session
bzr branch lp:~project-team/project/indicator-session master

Now I branch master, if this wasn’t a fresh checkout, I would bzr pull in master first.

bzr branch master remove-buttons

Make Changes

At this point we make fixes or whatever changes are needed. The package is built, changes are tested, and lintian is run (this one gets forgotten many times).

We have a few goals to meet for changes, we don’t always succeed, but here they are:

  1. No new lintian errors, if it’s a new package that we made, 0 is better.
  2. If the package has unit tests, add a new test case to cover what we just fixed/changed.
  3. Patches should have minimal DEP3 headers.
  4. Coding style should follow upstream.
  5. No new compiler warnings without explanation.
  6. Good changelog entries with bug numbers if applicable. Entries should list what files were modified. Distroseries set to UNRELEASED still (more on why later).

A note on lintian, Jenkins is capable of rejecting packages with lintian errors. We have this disabled because we need to fix the errors that crept in first when we didn’t follow this rule.

Push to a Remote Branch for Review

We code review everything we do, so the next step is to make the branch public for a review.

bzr commit -m "good message, usually we just use the changelog entry" --fixes lp:BUGNUM
bzr push lp:~project-team/project/indicator-session-remove-buttons

Setup a Code Review

Everything is reviewed and all reviews are sent to the team, though the onus is on the submitter to ping appropriate people if they don’t get a timely review. For code reviews, everyone is expected to provide a good explanation of what they’re doing and what testing was done.

We also have one of the “enhancements” here as we have a Jenkins instance (similar to this one) setup for some projects and Jenkins gets to “vote” on the review. Packages that fail to build or fail unit tests are marked as “Rejected” in the review by Jenkins.

Merge Back to Master

After the review is approved, the code submitter merges the code and commits it up to the mainline. I’m paranoid about master changing, although the push will fail if it did, so I always update it first.

We have to also fix the distroseries back. We do this on our team because it reduces the chance that someone will dput a package that is built from a local or non-master branch. If somone were to try and dput the changes file built from the remove-buttons branch, it would fail. We really want the archive to only have packages built from master, it’s more repeatable and easier to track changes.

cd ~/Projects/project-precise-amd64/indicator-session
cd master
bzr pull
bzr merge ../remove-buttons
dch -e (modify distroseries from UNRELEASED to precise)
debcommit -r
bzr push :parent

Jenkins Does dput

Our team is slowly moving into the world of Jenkins and build/test automation, so we have Jenkins watching the master branch for interesting projects and it will manage the dput for us. This also provides a final round of build testing before we dput.

Some teams have autolanding setup, that is when the review is approved, the Jenkins instance will do the merge. For now, we’ve kept a human in the loop.

Update the Bug

It is annoying to look at a bug 3 months after you fixed it and wonder what version it’s fixed in. Although the debian/changelog tracks this, we generally always add a bug comment saying when a bug was fixed. For the most part people usually just paste the relevant changelog entry into the bug and make sure it’s marked as Fix Committed.

Read more
Martin Pool

We’re going to release bzr 2.2b4 this week, which will be the final beta for the bzr 2.2 series and the start of the 2.2 release branch.  From this point on the 2.2 will be an API freeze, so that any plugins that are updated to work with 2.2b4 will also work with 2.2.0 and future bugfix updates.  We plan to do 2.2.0 at the end of July.

2.2 brings a bunch of performance, correctness and usability improvements.

Read more
Szilveszter Farkas

In this blog entry I would like to describe our deployment strategies we use at the different stages of our development process. The stages are the following:

  1. local development
  2. QA
  3. staging (+ QA)
  4. production (+ QA)

For local development everyone is welcome to use his preferred way, but most of us bet on Virtualenv. Especially given the fact that we maintain a large number of different projects, it makes our lives a lot easier to have a separate environment for each of those. We also have to make sure that we align with the production environment, which is in some cases still Python 2.5-based, but we’re currently in transition to 2.6.

If a feature or bugfix is ready to be QA’d, we deploy the application to an Amazon EC2 instance. Our team mate, ?ukasz Czy?ykowski, wrote a collection of extensions to Fabric, that provides a few useful functions (e.g. using private PPAs very easily). With a few dozens of lines of simple Python code, we can deploy the whole application to a running EC2 instance. We also use EC2 to QA all the features and bugfixes targeted at a release together before deploying to staging, so that if there is an issue, we can re-deploy very quickly (during the next two stages, QA is mainly about testing regressions).

The staging and production environments are identical from the deployment process perspective. We simply create a binary Debian package from our application: Launchpad’s PPA feature makes the build process a breeze. The main reason we decided to go with Debian packages is that we can also specify system level dependencies, not only Python packages (and of course there’s some dogfooding involved since the company supports Ubuntu). This also requires that all of the team members have packaging skills, so we had several training sessions, and a two-day online sprint where we packaged lazr.restful and all of its dependencies which were not available in Ubuntu 8.04 (around 30 packages, half of them backports, half of them new packages – thanks to our hard-working team members, these are available for Ubuntu 10.04 as well).

For configuration we don’t use Django’s built-in settings mechanism, but a custom solution that will be open sourced in the near future (one more reason to keep an eye on our blog). It consists of two components: schemaconfig is responsible for parsing the config files (which are INI-style, but have some extra features, like layering, typing, and support for data structures like lists and dictionaries – basically we looked around for solutions, and stole a little bit from everywhere to put together one that fits us most), and there’s django-settings which is a glue between schemaconfig and Django’s settings (so in the end we still use django.conf.settings). One of the biggest problems we had with our previous setup that it was very prone to human error, and that caused us unexpected deployment issues between staging and production. This is solved by the layering and the non-Python style of the config files, so they are easily manageable by both us and IS (our operations team).

Watch this blog for more about schemaconfig and other exciting projects and articles!

Read more
Martin Pool

Patch Pilots

We’ve just started a patch pilot project in Bazaar, to make sure that patches aren’t lost in the noise, or stalled waiting for cleanups or tests to be added. In particular, we care a lot about testing for Bazaar, and testing well, but adding good tests can be harder than just making a two-line change that seems to fix the problem.

Andrew’s just finished the first stint as pilot, with good results: several patches landed, and people apparently having an easier time getting them.

I wonder how it will go? Will we keep finding the time to prioritize pilotage, and will there be positive feedback from contributors?

Read more