Canonical Voices

Posts tagged with 'bug tracking'

Matthew Revell

Some bugs get reported more than once. That’s why we’ve got the dupe finder.

Some duplicate bugs slip through the dupe finder. Really common issues get quite a few dupes and someone from the relevant project usually goes through and marks them as duplicates of the master bug where the actual discussion and tracking is taking place.

There has been a really annoying bug in the way Launchpad has handled all this, though, and Deryck’s just fixed it :)

Let’s say you’ve got a bug report that has a few duplicates attached to it. You then discover that, actually, there’s an older bug with a more mature discussion and that, really, that’s the master bug. Until now, before you marked your bug as a duplicate of the master, you’d first have to take all the dupes of your bug and manually make them dupes of the master.

Still with me? :)

For a busy bug with many dupes, some of which have their own dupes, that’s a real disincentive to clearing up the multiple duplicates and gathering everything together on the one true master bug.

Now, though, simply mark your bug as a duplicate and Launchpad will automatically transfer your bug’s dupes to the new master bug.

Simple :)

Read more
Matthew Revell

Some bugs attract many, many comments.

For a while now, Launchpad has displayed only the first 80 comments on any bug report, with the option of viewing the full comment history. That’s been good for speeding up page loads but not so great at offering an accurate view of the current state of discussion about the bug.

Bryce has fixed that. Now, a bug report page still shows only 80 comments, by default. However, to give a better overview of the state of discussion, it now shows the first 40 and the last 40 comments.

So, half way down the comments you’d see something like:

Here's what you see in the middle of the bug comment history

Here's what you see in the middle of the bug comment history

Then, at the bottom of the summarised comment history there’s this:

Comment history summary

Comment history summary

With any luck, this should result in new bug comments that take into account the most recent discussion.

Read more
Deryck Hodge

Recently, we changed the way assigning bugs works on Launchpad. It used to be that anyone could assign anyone else to a bug. This was open to abuse as you can imagine. Bug 511269 was filed about the potential problems with this, and we recently changed Launchpad so that only bug supervisors can assign a bug to someone else.

You can still assign a bug to yourself, but this does keep you from assigning someone to a bug to draw their attention to said bug. In the end, this is a good thing, though, as people should only be assigned bugs who are going to be responsible for working on them.

Now there is one issue with this change. Projects that had not established a bug supervisor for the project will find their developers can no longer assign bugs to each other. The easy fix for this is to create a bug supervisor team for your project and have the people working on your bugs assigned to this team. We do realize this might be a bit heavy weight for some projects, so we’ve opened bug 603281 about this issue.  A fix for this — only requiring bug supervisor permissions if bug supervisor is defined — should be appearing on Launchpad soon.

Read more
Deryck Hodge

Many different types of information are stored in bug reports in Launchpad.

Some are actual defects, some are feature requests, some are general issues, and so on.  It is not uncommon on Launchpad to have a bug that deals with an issue that a developer cannot resolve.  In Launchpad, we offer a couple of bug statuses that allow a developer to close a bug report without actually doing what is requested in the report: these are Won’t Fix and Invalid.

Often, though, there may still be a discussion. Won’t Fix and Invalid are useful for the developer, and the project, to know that they don’t need to schedule time for a fix. However, they can sometimes — rightly or wrongly — be seen as an attempt to close down to discussion.

We’ve just added a new bug status to Launchpad: Opinion. Now, this is a fairly momentous occasion; we hardly ever make changes to bug statuses because they, naturally, have a great impact on how you and others use Launchpad to track bugs. However, we feel it’s important to find a way to balance a project’s need for useful work planning with the need for intelligent and open discussion.

Opinion says: there’s a difference of opinion around this bug and people are free to continue the discussion, but the project or package maintainers need to move to other work and are considering the issue closed.

Like I said, adding a new bug status to Launchpad is a big deal. So, we’re treating Opinion as an experiment. We’ll watch how it is used over the next three months and then we’ll decide if the status is proving useful and effective at closing bugs while leaving the discussion open.

I’d love to hear your views on this new status: leave a comment here, join us on the launchpad-users mailing list or mail me directly.

Read more
Matthew Revell

When you’re new to a bug report that’s already had quite a bit of activity, it can take a few minutes to get a hang of what’s been happening.

Launchpad gives you a shortcut that lets you quickly see the history of the bug: the bug activity log.

Let’s take a look at a bug I’ve been working on recently: bug 544799. While the main bug page gives you the current description, comment history and details of status changes, you can get a concise yet comprehensive overview of the bug’s history by following the See full activity log link.

So, when you need to get up to speed on a bug report, head for the activity log.

Read more
Matthew Revell

As I’ve said before, Launchpad is pretty big. Getting to know everything you can to do in Launchpad can take a while.

Of course, there are the user guide, the tour and the dev wiki even has a feature check-list. It’s still easy to miss things.

So, I’m turning the fifth day of each week into Feature Friday :)

I’m gonna kick off with something I use a lot and that, to be honest, is more a Bazaar feature, than a Launchpad feature:

bzr commit -m "Adds email functionality to the client, thereby obeying Zawinski's law." --fixes lp:1234

Adding --fixes lp:1234 to a commit tells Bazaar that the branch contains a fix for bug 1234 tracked in Launchpad.

The next time you push the branch to Launchpad, Launchpad will create a link between the branch and bug 1234.

Read more
Matthew Revell

I want to learn more about how and why people triage bugs, whether that’s in Launchpad or another bug tracker.

So, I’m inviting people who often triage bugs to come to either London or UDS in Brussels to tell me more about their experience.

If we get two or three people somewhere other than London or Brussels, we may also be able to come to you.

All I need is around an hour of your time where, along with my colleague Charline, I’ll ask you to show me how you triage bugs. If you’re interested, or have questions about how it works, mail me: matthew DOT revell AT canonical DOT com

Read more
Gavin Panella

Launchpad has a feature where it periodically checks the status of remote bugs (as in, bugs recorded in another bug tracker, like bug 12720 in Django).

When someone links a bug on Launchpad with a remote bug it’s called a bug watch. All the bug watches for a bug appear in the Launchpad bug page in an area called “Remote bug watches”. Check out bug 513719 to see a bug watch for bug 12720 in Django.

If the remote bug tracker has been set as the bug tracker for a project in Launchpad, bug tasks for that project can be linked to a specific remote bug too. When the status of the remote bug changes, Launchpad changes the status of the bug task to match, and sends out email to subscribers, the same as if the status had been changed in Launchpad. See the Django bug task in bug 513719 for an example.

Going further, comments can be synchronized too, in both directions. Recent versions of Bugzilla have this capability built in, but older versions can be supported with a plugin. There’s also a plugin for Trac.

This is all very nifty stuff, but it suffers because it doesn’t work very well! Yet.

Part of this is down to complexity

We support 7 different remote bug tracker types: Debian Bugs, Trac, Bugzilla, Mantis, RT (Request Tracker), Roundup and SourceForge.

We try to support a range of versions for each of these trackers, and a range of different access methods.

For example, with Bugzilla, we support old (v2) installations, more recent (v3) ones, recent ones with the Launchpad plugin, and very recent ones with the API built-in. And we support Issuezilla, a variant of Bugzilla.

We try to work around many idiosyncrasies and customizations in the remote systems.

With Mantis, for example, sometimes we need to log in anonymously before we can download bug statuses. Some Mantis installations allow us to download status information in CSV form, in a batch, but not all. If not, we screen-scrape the individual bug pages. Even if we get CSV, it’s often slightly corrupt, so we try and correct for that where we can.

Then there are simply hundreds of things that are beyond our control that Launchpad must cope with and move on, like HTTP errors, errors without correct HTTP codes, slow responses, unrecognized responses, and so forth. We check about 7k remote bugs every day (we aim to do more than 30k a day, but we’re not there yet) so there are a huge number of errors to sift through. It’s a big task to figure out if each problem is transient, a problem in Launchpad, or a remote problem, never mind actually fixing it!

We must also be gentle with the remote systems, and not issue too many requests in a short time, or otherwise make unreasonably large demands on them. We must be nice.

Part of it is down to the development of the code base

The bug watch code in Launchpad is a bit creaky. Originally it was designed to run in a single thread in a single process. This means that one badly performing bug tracker can starve the whole system of updates.

In the past we’ve tried to alleviate this by choosing the bug watches to check more carefully, or by batching requests, but this has not been enough for a some time.

Also, while we do record a lot of information about errors, it is still a Herculean task to constantly monitor the errors coming out of the system.

We also can’t test or do QA against other people’s systems. For testing we must use test doubles to simulate the behaviour of remote systems, but this can get confusing. For QA, sometimes we must test against remote systems. This is acceptable for testing the status fetching code, but not for comment synchronization.

However, we have not had the time available to give the bug watch system a big overhaul, only to make small improvements and bug fixes. For a long time it’s needed some big changes to make it work with the volumes of work it’s expected to cope with.

Fixing it

We’re trying to fix these issues now.

We’ve made checkwatches – the program that drives the bug watch machinery – run multi-threaded. This works, but it hammers our database, so we need to figure out how to alleviate that next.

We’ve started the work to move the code base over to using Twisted. This is a better model for managing a lot of concurrent network activity. As more and more bug watches are registered with Launchpad, we’re going to need it.

We’re going to keep more history of success and failure in the database, so that checkwatches can throttle back checks for remote bugs that persistently error. This is really important because it will reduce the work that checkwatches needs to do each day. It will also, we hope, reduce the deluge of errors to a more manageable stream, so that we’re better able to spot genuine bugs that need our attention. Lastly, we hope to display this information on the web pages so that users can help to diagnose problems themselves.

We’re getting some more infrastructure in place, so that we can develop and QA against real installations of Bugzilla, Trac, et al.

We’re also fixing up as many bugs in checkwatches as we can.

Who are we?

Abel Deuring, Graham Binns and Gavin Panella. We’re meeting up next week in Norwich, England for a sprint, a culmination of a couple of months of dedicated effort on the bug watch system.

Deryck Hodge is also coordinating with the Canonical IS team to deliver the QA infrastructure.

For all of us, the aim is to get checkwatches to work reliably, even if it’s not the most efficient or elegant system inside. We also want future development to be more sustainable than it has been in the past.

We want users to be able to rely on this feature.

Read more
Graham Binns

Imagine the scene, dear reader. You’re running the latest Ubuntu pre-release, dilligently testing that everything works the way it ought. An application crashes horribly; Apport pops up to tell you that it spotted the crash – would you like to report a bug?

“Of course I would,” you cry, “I am a desktop testing hero!” And so Apport does its thing and takes you to Launchpad to file the bug report. And then Launchpad times out.

At this point, if you’re like me, you might shout a bit. You refresh and refresh with all your might but still Launchpad will only give you an error page. Finally, defeated, broken, you close your browser and shuffle off into the corner of your room, where you bury yourself under the mountain of discarded CD-Rs that contain daily ISOs of Ubuntus past and sob into your coffee.

Okay, so maybe I’ve exaggerated it a bit, but that doesn’t change the basic fact that timeouts on the +filebug page when you’re filing a bug via Apport are intensely, soul-destroyingly annoying. They’re annoying to us in the Launchpad Bugs team too, because we see them in our error reports. They’re so annoying, dear reader, that we’re moved to hyperbole when writing about them for the official Launchpad blog.

The problem with processing data supplied by Apport has always been one of scale. Originally the extra data that Apport would upload to Launchpad wouldn’t be massive, maybe hitting 10MB if the application was particularly busy. Because of this, Launchpad simply processed those data synchronously whilst loading the bug-filing form for you, so that the data you’d uploaded would be included with the bug.

Of course, that approach doesn’t scale very well, and recently we’ve been seeing the data blobs that Apport uploads hit ~100MB in size. That’s far too big for Launchpad to handle in a timely manner whilst doing all the additional work of rendering the bug filing form for you, so in those circumstances it invariably times out and you get that ever-annoying error page.

This was an interesting problem to solve, and we came up with a number of different possible solutions. One viable solution for this, which we eventually decided not to implement, involved having a separate request for processing the extra bug data and then loading the data into the filebug form with AJAX. We discarded that idea because there was always the chance that that request would time out, leaving us in an only slightly better position than we were already.

There was some discussion on the Launchpad developers mailing list about whether we could just defer loading the extra data into the bug until after it had been filed, but we quickly realised that not only could the extra data carry an initial subject for the bug, but it could also indicate that the bug should be filed as private, something which currently can’t be done via the Launchpad web UI.

The solution we chose to implement, which we’ve now rolled out to the Launchpad edge and production servers, is to have a queue of blobs waiting to be processed. We already have quite a robust job-processing system built into the Launchpad codebase, which we use for creating the diffs in merge proposals and calculating bug heat, amongst other things. Adding support for processing uploaded blobs was quite simple, since the existing blob parsing code was well documented. The blob processing jobs are picked up and run by a cron script that runs every minute or so, and the data retrieved from the blobs are stored with the original processing job as a JSON-formatted string. When you hit the +filebug page it checks to see if the relevant blob has been processed. If not, it waits until processing is complete. Once processing is complete the serialized data are loaded into the +filebug page for use later on.

The advantage to you as a user, then, is that you should never again see a timeout on the +filebug page due to the size of the blob that Apport has uploaded. With the upcoming release of Ubuntu Lucid, we’re hopeful that this will make a real difference to those people testing the pre-release versions of the distro.

Read more
Karl Fogel

Getting Patches Upstream


We heard from Ubuntu maintainers and upstream developers that it was too easy to lose track of patches in bugs filed on Launchpad — or, put another way, that Launchpad wasn’t doing enough to bring patches to developers’ attention.

According to the Launchpad Upstream Improvements Spec that came out of a recent UDS:

Upstreams frequently complain that it is difficult for them to find Ubuntu patches. is a diff between Ubuntu and Debian, so it’s not really useful to some upstreams. We want an upstream patch report that shows patches in packages and their age, so we can act on them. …

To help solve this problem, we’ve added two new features to Launchpad.

1. The upstream report now shows bugs carrying patches.

The upstream report now shows the number of bugs carrying patches — see the rightmost column in this screenshot:

Upstream report, showing number of patches in rightmost column.
(click image to enlarge)

If you click on the number in that column, it brings you to a listing of the bugs with patches.

2. A new “patch report” is available wherever bugs are available.

The second feature is a new patch report available for projects, project groups, source packages, people, teams, distributions, and distroseries. It lists bugs that have patches attached — for those of you who’ve seen Bryce Harrington’s mockup, this is the same thing implemented directly in Launchpad.

The goal of the patch report is to make it easier for packagers and upstream developers to find patches that could be upstreamed, and, especially, to enable them to evaluate those patches quickly.

You can reach the patch report from a new link on any bugs page. On the project bugs page below, for example, see the link “7 bugs with patches” in the portlet on the right (and you can click on any of these screenshots to get a larger version):

The product bugs page bugfilters portlet shows the number of bugs with patches.

Clicking on the “7 bugs with patches” link appends “+patches” to the URL (hint, hint) which takes you to the corresponding patch report. There, patches are listed by bugtask, sorted by the age of the most recent patch in each bug, on the theory that new patches are probably more interesting than old ones:

The patch report for a project.

But you can sort by other things as well — say, by bugtask importance:

Selecting a sort by importance from the dropdown box of sort orders.

Sorting applies across all applicable patches, not just the ones that happen to be on the current page of a potentially multi-page list. Thus changing the sort order might change the set of bugs visible on the current page:

Patch tasks can be sorted by importance.

When you hover over a patch, a popup shows details about that patch (who uploaded it, the patch’s filename, and its attachment message):

Hovering over a patching displays a popup of information about that patch.

The patch report for entities like distributions and distroseries arranges patches by source package, since the same bug may have a different bugtask for each package, and each bugtask may have its own status, importance, etc:

Patch report for Ubuntu distro.

You can also view the patch report on a particular source package:

Patch report for a source package, about to click on the single patch.

Note that distroseries, persons, teams, and project groups all support this feature too. I didn’t include screenshots for them, to save space, but their patch reports can be reached in the expected way.

Naturally, clicking on a patch takes you straight to the diff:

A patch file.

We’re hoping that these features, combined with the recently-landed patch notification improvements and bug heat, will make it easier to find the patches worth immediate attention.

These patch reports are new in Launchpad 10.02. About a month after they go live, we’re going to look at the stats on how people have been using them, and we’re going to survey some upstreams and Ubuntu maintainers to see what they feel can be improved. Please also file bug reports as you use these new features, of course (if you feel like going the extra mile, tag your bug report with the patch-tracking tag — it’ll save us some time).

Finally, please feel free to help improve the features yourself! The bug links below will lead you to the branches on which these changes were made, so you can see what the code looks like. The Launchpad development wiki is the place to start for hacking on Launchpad.


Things we didn’t get to this cycle, but that we’re thinking about:

  • Finding patches in other distributions (but see Daniel Holbach’s amazing Harvest tool).

  • Bug #515674 is about showing the patches a distribution is carrying in its packages (i.e., in its debian/patches directories). Note in particular comment #4 there, about UI complexity.

  • Showing patches in upstream trackers. Such patches might be downstreamable, or they might be the same as patches Launchpad already has; either way, it’d be good to know. See bug #403443.

  • Package ↔ Branch equivalence. Really, a branch attached to a bug is equivalent to a patch attached to a bug (assuming the base source can be easily located, which it often can be). Ideally, we could represent either one as the other, and treat them as the same in filters and reports.

  • Launchpad isn’t yet making use of the Debian DEP-3 Patch Tagging Guidelines metadata, which is also the standard in Ubuntu now.

  • In general, see bugs tagged with patch-tracking and patch-tracking-external.

Read more