Canonical Voices

Martin Pool

Jelmer writes:

bzr-builddeb 2.8.1 has just landed on Debian Sid and Ubuntu Precise. This version contains some of my improvements from late last year for the handling of quilt patches in packaging branches. Most of these improvements depend on bzr 2.5 beta 5, which is also in Sid/Precise.

The most relevant changes (enabled by default) are:

  • ‘bzr merge-package’ is now integrated into ‘bzr merge’ (it’s just a hook that fires on merges involving packages)
  • patches are automatically unapplied in relevant trees before merges
  • before a commit, bzr will warn if you have some applied and some unapplied quilt patches

Furthermore, you can now specify whether you would like bzr to automatically apply all patches for stored data and whether you would like to automatically have them applied in your working tree by setting ‘quilt-tree-policy‘ and ‘quilt-commit-policy‘ to either ‘applied‘ or ‘unapplied‘. This means that you can have the patches unapplied in the repository, but automatically have them applied upon checkout or update. Setting these configuration options to an empty string causes bzr to not touch your patches during commits, checkout or update.

We’ve done some testing of it, as well as running through a package merge involving patches with Barry, but none of us do package merges regularly. If you do run into issues or if you think there are ways we can improve the quilt handling further, please comment here or file a bug report against the UDD project.

Caveats:

  • If there are patches to unapply for the OTHER tree, bzr will currently create a separate checkout and unapply the patches there. This may have performance consequences for big packages. The best way to prevent this is to set ‘quilt-commit-policy = unapplied‘.
  • bzr merge‘ will now fail if you are merging in a packaging tree that is lacking pristine tar metadata; I’m submitting a fix for this, but it’s not in 2.8.1.
  • if you set ‘quilt-commit-policy‘ and ‘quilt-tree-policy‘ but have them set to a different value, bzr will consider the tree to have changes.

To disable the automatic unapplying of patches and fall back to the previous behaviour, set the following in your builddeb configuration:

quilt-smart-merge = False

Read more
Brian de Alwis

I was recently asked to estimate how long I’d been working on a particular project. Unfortunately I hadn’t been keeping track of my time in any organized way.

Fortunately I realized that, since I like to commit frequently (though nothing like Stephen Turnbull’s commit-on-save!), I could come up with an estimate based on my commit dates.

But I quickly realized that bzr log --line puts the author name before the commit date:

  $ bzr log --line -r -3..
  150: Max Bowsher 2011-02-12 [merge] Fix invalid version_info.
  149: Jelmer Vernooij 2010-12-20 [merge] Fixes most of the remaining test fai...
  148: Gary van der Merwe 2010-10-20 [merge] Ignore build folder created by se...
  147: Martin 2010-09-09 [merge] Import xml escaping function through local mo...

The spaces could make extracting the date a bit fragile.

Fortunately I remembered the bzr-xmloutput plugin, which makes processing this kind of information really easy. bzr-xmloutput adds an “–xml” option to many of the standard bzr commands that encodes the output as an XML document. Combined with XMLStarlet, a command-line XML tool that provides XSLT/XPath processing (amongst other things), I was able to cook up a recipe in a matter of minutes:

  $ bzr log --xml \
  | xml sel -t -m '/logs' -m '//log' \
    -v 'substring-before(substring-after(timestamp," ")," ")' -n \
  | sort -u \
  | wc -l

The substring() is required to pull out the date; as bzr-xmloutput prints dates as ‘Day YYYY-MM-DD HH:MM:SS offset’. awk would have worked just as well too.

Too easy, thanks to Guillermo and the other bzr-xmloutput contributors! Now I’m thinking of other questions that can be answered…

Read more
Jonathan Riddell

What I did on my Rotation

Canonical has a company scheme where after working there for a few years you can rotate to work at another part of the company for 6 months. Having worked on the desktop team for over five years I decided to do a rotation to Bazaar. My hopes for this were to build up my own programming skills by learning more Python and by experiencing different programming practices from the ones I’m used to in KDE.

I started off with some fixes to the developer documentation. This got me used to the process that you can not commit directly to bzr’s trunk, instead all committers are required to make merge proposals on Launchpad, have those approved by a fellow developer, then send it to a programme called Patch Queue Manager which will integrate the patch and run the test suite to check everything still works.

Next I started fixing a few easy command line UI bugs, improving error messages or stopping exception output and so forth. This got me into the world of writing test cases. Everything in bzr needs a test case, merge proposals will not be accepted without them. Like much of bzr I find that the test cases lack API documentation and comments but it turns out they are easy enough to read and similarly easy to write. There are both internal test cases, which run a small part of the code within bzr, and blackbox test cases which run a bzr command.

Bazaar is the version control system used by top open source project hosting site Launchpad so I was surprised to come across a bug which prevented bzr from talking to Launchpad properly on errors. “This is really important to fix. We need error reporting.” said Jonathan Lange over 2 years before. Pleasingly I could fix it, very satisfying. I had to learn about the hooks mechanism in bzr which shows up some of the downside of Python, you have to guess the arguments to send the hook. But who needs API documentation when you can just read the code? :)

Bazaar’s main GUI is qbzr (which provides GUIs for individual commands) and Bazaar Explorer (which provides a complete GUI). I worked with Martin (gz) to make these two talk to the normal Ubuntu crash system, Apport, rather than showing a nasty crash backtrace to the user.

Then I noticed that Bazaar Explorer has a lot of “Refresh” toolbar buttons about the place, any time you make a change to the file you have to click one before the UI will update. Not very user friendly. So I added file watchers about the place to make it magically update. Nifty, except that after release it turns out this breaks horribly when doing some commands outside of Bazaar Explorer, oops. Quick fix and message to packagers, hang head in shame.

The first large feature I worked on was GPG signing of commits. The documentation for Bazaar promised that this was implemented and all you need do was set the various options in the config file. Alas it lied. I fixed up the documentation and started looking into the GPG python bindings, which turn out to be completely undocumented on the Python side and surprisingly badly documented on the C side. Security critical code which is badly documented seems scary to me, mistakes could easily be made which go unnoticed until it appears on full-disclosure. But I manage to implement signing and adding a GUI to Bazaar Explorer being cautious as I go.

Bazaar has a scheme called patch pilot where we review patches submitted by the community and help them on their way to being integrated. I started out with this by following John Meinel who can write code faster than I can write English prose. We made small changes to some patches and integrated them, we gave feedback to newer patches that needed some work and we chased up contributors who had not responded. The barrier to entry in Bazaar is pleasingly small, if you don’t have the skills to write a perfect patch it’s encouraged to say so and someone else will finish it off.

Why, I wondered, is bzr (the command line UI to Bazaar) not translated? There were parts of gettext scattered around the code, and some code to extract strings but it didn’t get used. Turns out this code was a half completed feature that had never been taken to completion. I finished off translations by adding gettext()s throughout the code, ensuring tests still pass, fix the installation of .mos and enable the generation of .pot. This missed the 2.4 release so I’m still waiting to see how it works for 2.5, I suspect some strings will be missing context needed to do a good translation and of course the occasionally technical output of bzr might need some thought on how to translate but it should make bzr easier to use for non-English speakers.

Ubuntu Distributed Development is the project to put all of Ubuntu’s packages and history into Bazaar branches and change our packages processes to use Bazaar. This makes a lot of sense, the Ubuntu archive is already a primitive revision control system (you upload for each new version, often its useful to look at older versions). This project has been a long time coming and is one of the original reasons why Canonical started Bazaar back in the day. It suffers from a number of problems, notably the failure of quite a lot of packages to import into Bazaar including currently the whole of KDE due to a patch into openSUSE’s bz2 package. Also the quilt patch system we use tends to clash with being held within a revision control system so you end up with diffs of diffs. I tend to think that would have been an easier win to import only the debian/ packaging into Bazaar branches.

I tidied up the new Ubuntu Packaging Guide which is a guide to packaging with UDD branches (named in the hope that UDD will soon become the definitive way to do packaging). I also added a new command bzr get-orig-source to make it easier to do packaging in the current directory rather than a separate directory as used by bzr builddeb. I also added a hook to set the bzr changelog from the debian/changelog entry which is the current behaviour with debcommit. I got mixed feedback on this so I added a config option to disable it too. I also tidied up some of the bzr-builddeb code by removing weird terms like “larstiq” and removing acronyms by default.

My Python programming has improved a lot and I’m a convert to the cause of unit tests. Python is a fun and productive language but the lack of culture for documenting APIs is disappointing and being dynamic it’s that much easier to make mistakes without realising it. My productivity is nothing like as high as others on the Bazaar team but it seems I’m better at improving (graphical and command) user interfaces than my colleagues who can memorise internal data structures trivially. My six months is now up, I’ve enjoyed them and now I’m looking forward to getting back into Kubuntu and KDE.

Read more
Vincent Ladeuil

The package importer is an important piece of the Ubuntu Distributed Development. It mirrors source packages and Bazaar branches and relies heavily on Launchpad to achieve that.

The past

During Launchpad downtimes, many (>1000) imports failed and they had to be re-queued semi-manually. The importer would have been better inspired by making tea instead of queuing imports that were bound to fail.

The circuit breaker

An automatically operated electrical switch designed to protect an electrical circuit <…> a circuit breaker can be reset (either manually or automatically) to resume normal operation.

This looks like a good candidate to avoid import failures while Launchpad is down.

In this automaton representing the behaviour of a circuit breaker, three events are used (remember that here closed == works ;)):

  • attempt: we try to use the circuit,
  • failure: an undesired event has occurred,
  • success: the circuit is working.

The main scenario here is:

closed — failure –> open — attempt –> half open — success –> closed

The reality test

A Launchpad rollout happened Friday 30 September 08:32. The importer log file said:

2011-09-30 08:32:02,308 – __main__ – INFO – Launchpad is down, re-trying jcifs

2011-09-30 08:34:09,337 – __main__ – INFO – Launchpad *is* back

The successful import took 27″, so the importer knew Launchpad was down for 1’40″ (back – down – duration(import)). I asked the Launchpad admins how long it took them and their log said:

2011-09-30 08:33:41 INFO    Outage complete. 0:01:40.919527

Make tea… or not

Another interesting number here is that we retried 498 times during this downtime. This is probably excessive and can be fixed by reducing the importer concurrency while Launchpad is down. These 498 attempts were previously seen as failures for 498 different packages.

In the end, not only did we avoid these 498 spurious failures but the imports were only suspended for as long as Launchpad was down, up to the second !

But that’s a bit short to make tea…

Read more
Brian de Alwis

It’s sometimes useful to be able to revert a branch to a previous known state. For example, I recently updated a bzr plugin to its latest and greatest to discover a severe regression. If I had had some foresight, I might have recorded the revision (or the “tip”) before the update to allow me to rollback to the previous stable version. But as I rarely have such foresight, and have more important uses for my little grey cells, I set out to create ‘tiplog’, a new bzr plugin for recording and referencing the history of a branch’s tip.

‘tiplog’ is inspired by git’s ‘reflog’, and records commits, uncommits, pushes (to the branch), pulls (into the branch) — basically any change that causes the tip to change. ‘tiplog’ only records pushes to local branches. But the plugin can also be run within the smart server, although it cannot distinguish the causes of the tip change.

In my updating scenario described above, for example, say the plugin was at revision 20. Running ‘bzr pull‘ pulls in revision 40. ‘bzr tiplog‘ will inform me that my plugin was previously at r20:

$ bzr tiplog
2011-09-23 tip:0 40 bsd@mt.ca [pull] update version numbers
2011-08-25 tip:1 20 bsd@mt.ca [pull] fix bug #12009
2011-06-23 tip:2 1 bsd@mt.ca [pull] initial commit

Better yet, I can easily return to that previous stable revision using the new ‘tip:‘ revspec, with ‘bzr pull -r tip:1‘. ‘tip:0 is the current tip.

I also find tiplog useful when developing with others, as I can quickly review the changes since I last pulled:

$ bzr log --line -r tip:1..
17705: dev1@xxx.com 2011-09-27 fixes #1474
17704: dev2@xxx.com 2011-09-27 fixing Job Cost
17703: bsd@mt.ca 2011-09-25 fixes #1377

And in fact I have that bound to an alias.

To install tiplog, simply perform the following:

$ bzr branch lp:bzr-tiplog ~/.bazaar/plugins/tiplog

Please direct report any bugs or questions through the plugin’s Launchpad page.

Read more
Jonathan Riddell

qbzr with curves

Nice little visual change to qbzr, curves on the diff view..

Before:

After:

Thanks to Iwata Hidetaka.

Being bored of the IRC poll on blogs.kde.org I made a new poll for revision control systems. I’m glad to see that after one vote Bazaar is at 100%.

Read more
Jonathan Riddell

Bug 83941 “bzr doesn’t speak my tongue” has been closed: bzr core can now be translated. (The qbzr and bzr-explorer guis have been internationalized for a couple of years.) If you want to help bring bzr to those who prefer to work in non-English languages please help translate at Launchpad.

The translation will involve quite a bit of specialist language (what is French for “colocated branch”?) and I expect there are strings yet that need to be added to the translation file. I also need to look at translations for plugins.  Please send issues to either the Bazaar mailing list or as bugs on bzr on Launchpad.

Philippe Lhoste wrote a while ago about the issues of translating DVCS terminology.

Read more
Martin Pool

Our blog has moved

The Bazaar team blog is moving to a new location in our own domain: blog.bazaar.canonical.com.


Read more
Martin Pool

Aaron has announced a bzr plugin fault-line:

It works by looking at previous revisions where the file was changed,
and seeing what test files were changed at the same time. You can
specify the files, or it will autodetect them by looking at your working
tree.

For example:

$ bzr faultline
lib/lp/soyuz/test/test_archive.py
lib/lp/soyuz/scripts/tests/test_copypackage.py
lib/lp/soyuz/browser/tests/test_archive_webservice.py
lib/lp/code/windmill/tests/test_recipe_request_build.py

Read more
maxbowsher

As the final step of consolidating all of the official Bazaar PPAs on Launchpad under one Launchpad team, the Bazaar Beta PPA formerly found at https://launchpad.net/~bzr-beta-ppa/+archive/ppa has moved to live under the main ~bzr team at https://launchpad.net/~bzr/+archive/beta. If you are a user and tester of Bazaar beta releases via this PPA, you will need to update your APT sources.list lines -  you can see the new sources.list lines under the “Technical details about this PPA” section at the above link.

Read more
Martin gz

Last week was the Bazaar sprint, which was fantastic and tiring. Somehow even the people who’d been at UDS just before made it through five packed days of fixing bugs, preparing releases, and debugging package imports. We were most hospitably hosted at the Canonical offices a long way up Millbank tower. But even those who couldn’t be there in person to enjoy the view were part of the experience. At home in the Ukraine Alexander wore his Bazaar shirt in support during the first day. On IRC larstiq and santagada ran the test suite on pypy and investigated incompatibilities. And all week we had a small robot John sitting in the middle of the table on the line from the Netherlands, working on performance bugs and offering helpful advice.

There were two new faces introduced. Max has been a stalwart maintaining the ~bzr PPAs and getting daily builds working. Jonathan is joining the Bazaar team on rotation from Kubuntu, which is very exciting for fans of qbzr. He started getting to know bzrlib by taking on some bugs tagged ‘easy’ and pair programming on harder ones. It was a bit tough to keep track of everything going on, but good progress was made on the Ubuntu Distributed Development front, the translation framework branches Naoki put together were landed, and lots of pet bugs were fixed. Download bzr 2.4b3 now to see the rest of the results for yourself.

After these long days in front of screens a nice meal out was a welcome treat. Over dinner we even managed to get on to topics other than code on occasion. On Thursday evening everyone went to As You Like It at the Globe as groundlings. Even with the language barrier to overcome for some of the sprinters, the comedy lived up to the categorisation. Trying to use the cycle hire scheme to travel there and back proved more of an obstacle. The bikes themselves were fine, provided you could get past the terrible computer interface and persuade the system to let you rent them. Now, if only they took patches for that…

Read more
jameinel

The last of my patches is queued up to land, so I figured I’d post an update about the performance improvements I’ve been working on. I’m also just excited about how well it has all come together.

There were essentially 3 changes that mattered for performance on large trees.

  1. Fixing iter_entries_by_dir() to preload the data in Repository- optimal ordering rather than by-request ordering. In large trees this was causing us to thrash and become pathologically slow. In the 70,000-file test tree, thrashing took about 3 minutes, the preloading version takes about 15s. This affected a lot of our commands, though I guess the next two fixes would actually reduce the number of commands affected by this.
  2. Fixing several code paths to use optimized iter_changes() rather than the generic iter_changes(). The generic path walks both inventories iter_entries_by_dir() and compares them. Our 2a format Repository can do iter_changes without loading the whole tree. (It internally uses a hash_trie to store the inventory, and so nodes with matching sub-trees can be skipped for comparison.) This generally shows up as something that was taking 15s (to load the whole inventory) dropping to <2s for the improved comparison. (bzr revert and bzr pull were both directly impacted here)
  3. Changing WT.set_parent_trees([one_tree]) to update itself using current_basis.iter_changes(one_tree), rather than setting the state from scratch. This basically adds another case where we can avoid reading the whole inventory state again, which is another 15s to <2s sort of change. This only showed up after fixing (2), because once the tree is loaded, the other actions are generally pretty quick. (bzr up, bzr pull)

This is the chart I put together for “whats-new-in-2.4.txt”. bzr-2.3.2 will have fix (1), but not (2) or (3), to give a feel for how much of an impact different fixes have had.

    bzr-2.3.1 bzr-2.3.2 bzr-2.4  action
    3m39s         1m08s   1m03s  bzr co --lightweight
      38s            8s      2s  bzr revert (in a clean tree)
    4m47s         3m56s     15s  bzr merge
    4m45s           20s      3s  bzr pull
    4m58s         3m00s      2s  bzr up
    9m33s           21s     19s  bzr uncommit (including a merge)
    4m44s           17s      2s  bzr uncommit (simple commit)

So yes, some operations that were taking almost 5 minutes have now dropped down to taking <3s.

You won’t see that dramatic of an improvement for smaller trees, though most cases will have a pleasant improvement. Here is a short list for the ‘Launchpad‘ tree (with ~8k items).

    bzr-2.3.1   bzr-2.4     action
    5.3s        5.2s        bzr co --lightweight
    0.9s        0.3s        bzr revert
    1.4s        0.4s        bzr pull
    3.9s        3.7s        bzr uncommit (with merge)
    0.9s        0.3s        bzr uncommit (without merge)

Anyway, I’m quite happy about how much better bzr-2.4 will be in large trees.

update:Add graphs…

Read more
Martin Pool

new shelve/unshelve GUI

Iwata is working on a new GUI for interactively shelving and unshelving changes, which is a way in Bazaar for temporarily setting aside some changes from your working tree. (At the moment there is an interactive text interface for shelving.)

Read more
bialix

Thanks to our contributors we have a plenty of new exciting features and improvements in upcoming 0.21 release. I would like to give you a short tour over some of new features.

qdiff dialog now has new look and feel based on the nice toolbar (similar toolbar Gary has added for qannotate in past releases):

qdiff toolbar

New toolbar in qdiff dialog

As you can see qdiff now has Find text feature, you can invoke this mode clicking by “Find” button on toolbar (or using Ctrl+F hotkey):

Find in qdiff

Found

search: not found (as in firefox)

Not found

qdiff options are in View Options submenu on toolbar, you’ll find there “Complete” knob (to see full-text for all changed files),  encoding selectors (for every pane) and even new knob: Ignore whitespace changes.

qdiff options submenu

qdiff options

You can also force option “Ignore whitespace changes” from command-line with new qdiff command-line option –ignore-whitespace (short form -w).

Also, now you can control the size of tab characters in qdiff, qannotate and qcat windows via configurable option (tab_width). You can set this option either in bazaar.conf or individually for branches in their branch.conf. Global option can be edited via qconfig dialog:

qconfig dialog

Configure tab width via qconfig

Other improvements in upcoming release also include changes to qinfo dialog (it now shows the same information as CLI bzr info), qcommit now remembers “Show non-versioned” knob state, you can open files in their native applications while browsing treeless branches (or branches on remote servers) and other changes (see NEWS.txt for the full list of improvements and bugfixes).

Many thanks to all people who help us make new QBzr release awesome, especially: A. S. Budden, Dorin Scutara?u, Glen Mailer and many others, see NEWS.txt for details.

Read more
Martin Pool

bzr support in Qt Creator

Hugues Delorme has added support for Bazaar into Qt Creator 2.2 beta.

Read more
Martin Pool

Bazaar nightly Ubuntu packages are available from ppa:bzr/daily. The old ppa, ppa:bzr-nightly-ppa/ppa was out of date and has just been deleted.

To get nightly bzr builds on Ubuntu, say:


sudo add-apt-repository ppa:bzr/daily
sudo apt-get update
sudo apt-get install bzr

and as of today you should get a 2.4 beta snapshot.

Read more
Martin Pool

A new plugin bzr-wincrypt stores passwords encrypted in the Windows CryptoAPI keyring.

Read more
Martin Pool

A 5min demo showing how large features can be managed easily in small interdependent branches using bzr pipeline, with a few nice Launchpad integrations built in.


Read more
jameinel

We just did one final tweak to bzr to be compatible with python2.7 for natty.

Vincent tells us that babune is finally blue running the test suite, and Jelmer says that the daily builds of python now finish for natty. Yay!


Read more
Martin Pool

maxb has built packages of bzr-svn and its dependencies into ppa:bzr/proposed, including a new and much better version of the Subvertpy subversion bindings.  He is looking for positive or negative test results from people using these on Ubuntu hardy and jaunty.  If you do use bzr-svn on those OS releases, please give them a try, or at least Max know (here or on the list) that you care about getting updates…


Read more