Canonical Voices

Posts tagged with 'bug fixes'

Colin Watson

Well, it’s been a while!  Since we last posted a general update, the Launchpad team has become part of Canonical’s Online Services department, so some of our efforts have gone into other projects.  There’s still plenty happening with Launchpad, though, and here’s a changelog-style summary of what we’ve been up to.

Answers

  • Lock down question title and description edits from random users
  • Prevent answer contacts from editing question titles and descriptions
  • Prevent answer contacts from editing FAQs

Blueprints

  • Optimise SpecificationSet.getStatusCountsForProductSeries, fixing Product:+series timeouts
  • Add sprint deletion support (#2888)
  • Restrict blueprint count on front page to public blueprints

Build farm

  • Add fallback if nominated architecture-independent architecture is unavailable for building (#1530217)
  • Try to load the nbd module when starting launchpad-buildd (#1531171)
  • Default LANG/LC_ALL to C.UTF-8 during binary package builds (#1552791)
  • Convert buildd-manager to use a connection pool rather than trying to download everything at once (#1584744)
  • Always decode build logtail as UTF-8 rather than guessing (#1585324)
  • Move non-virtualised builders to the bottom of /builders; Ubuntu is now mostly built on virtualised builders
  • Pass DEB_BUILD_OPTIONS=noautodbgsym during binary package builds if we have not been told to build debug symbols (#1623256)

Bugs

  • Use standard milestone ordering for bug task milestone choices (#1512213)
  • Make bug activity records visible to anonymous API requests where appropriate (#991079)
  • Use a monospace font for “Add comment” boxes for bugs, to match how the comments will be displayed (#1366932)
  • Fix BugTaskSet.createManyTasks to map Incomplete to its storage values (#1576857)
  • Add basic GitHub bug linking (#848666)
  • Prevent rendering of private team names in bugs feed (#1592186)
  • Update CVE database XML namespace to match current file on cve.mitre.org
  • Fix Bugzilla bug watches to support new versions that permit multiple aliases
  • Sort bug tasks related to distribution series by series version rather than series name (#1681899)

Code

  • Remove always-empty portlet from Person:+branches (#1511559)
  • Fix OOPS when editing a Git repository with over a thousand refs (#1511838)
  • Add Git links to DistributionSourcePackage:+branches and DistributionSourcePackage:+all-branches (#1511573)
  • Handle prerequisites in Git-based merge proposals (#1489839)
  • Fix OOPS when trying to register a Git merge with a target path but no target repository
  • Show an “Updating repository…” indication when there are pending writes
  • Launchpad’s Git hosting backend is now self-hosted
  • Fix setDefaultRepository(ForOwner) to cope with replacing an existing default (#1524316)
  • Add “Configure Code” link to Product:+git
  • Fix Git diff generation crash on non-ASCII conflicts (#1531051)
  • Fix stray link to +editsshkeys on Product:+configure-code when SSH keys were already registered (#1534159)
  • Add support for Git recipes (#1453022)
  • Fix OOPS when adding a comment to a Git-based merge proposal without using AJAX (#1536363)
  • Fix shallow git clones over HTTPS (#1547141)
  • Add new “Code” portlet on Product:+index to make it easier to find source code (#531323)
  • Add missing table around widget row on Product:+configure-code, so that errors are highlighted properly (#1552878)
  • Sort GitRepositorySet.getRepositories API results to make batching reliable (#1578205)
  • Show recent commits on GitRef:+index
  • Show associated merge proposals in Git commit listings
  • Show unmerged and conversation-relevant Git commits in merge proposal views (#1550118)
  • Implement AJAX revision diffs for Git
  • Fix scanning branches with ghost revisions in their ancestry (#1587948)
  • Fix decoding of Git diffs involving non-UTF-8 text that decodes to unpaired surrogates when treated as UTF-8 (#1589411)
  • Fix linkification of references to Git repositories (#1467975)
  • Fix +edit-status for Git merge proposals (#1538355)
  • Include username in git+ssh URLs (#1600055)
  • Allow linking bugs to Git-based merge proposals (#1492926)
  • Make Person.getMergeProposals have a constant query count on the webservice (#1619772)
  • Link to the default git repository on Product:+index (#1576494)
  • Add Git-to-Git code imports (#1469459)
  • Improve preloading of {Branch,GitRepository}.{landing_candidates,landing_targets}, fixing various timeouts
  • Export GitRepository.getRefByPath (#1654537)
  • Add GitRepository.rescan method, useful in cases when a scan crashed

Infrastructure

  • Launchpad’s SSH endpoints (bazaar.launchpad.net, git.launchpad.net, upload.ubuntu.com, and ppa.launchpad.net) now support newer key exchange and MAC algorithms, allowing compatibility with OpenSSH >= 7.0 (#1445619)
  • Make cross-referencing code more efficient for large numbers of IDs (#1520281)
  • Canonicalise path encoding before checking a librarian TimeLimitedToken (#677270)
  • Fix Librarian to generate non-cachable 500s on missing storage files (#1529428)
  • Document the standard DELETE method in the apidoc (#753334)
  • Add a PLACEHOLDER account type for use by SSO-only accounts
  • Add support to +login for acquiring discharge macaroons from SSO via an OpenID exchange (#1572605)
  • Allow managing SSH keys in SSO
  • Re-raise unexpected HTTP errors when talking to the GPG key server
  • Ensure that the production dump is usable before destroying staging
  • Log SQL statements as Unicode to avoid confusing page rendering when the visible_render_time flag is on (#1617336)
  • Fix the librarian to fsync new files and their parent directories
  • Handle running Launchpad from a Git working tree
  • Handle running Launchpad on Ubuntu 16.04 (upgrade currently in progress)
  • Fix delete_unwanted_swift_files to not crash on segments (#1642411)
  • Update database schema for PostgreSQL 9.5 and 9.6
  • Check fingerprints of keys received from the keyserver rather than trusting it implicitly

Registry

  • Make public SSH key records visible to anonymous API requests (#1014996)
  • Don’t show unpublished packages or package names from private PPAs in search results from the package picker (#42298, #1574807)
  • Make Person.time_zone always be non-None, allowing us to easily show the edit widget even for users who have never set their time zone (#1568806)
  • Let latest questions, specifications and products be efficiently calculated
  • Let project drivers edit series and productreleases, as series drivers can; project drivers should have series driver power over all series
  • Fix misleading messages when joining a delegated team
  • Allow team privacy changes when referenced by CodeReviewVote.reviewer or BugNotificationRecipient.person
  • Don’t limit Person:+related-projects to a single batch

Snappy

  • Add webhook support for snaps (#1535826)
  • Allow deleting snaps even if they have builds
  • Provide snap builds with a proxy so that they can access external network resources
  • Add support for automatically uploading snap builds to the store (#1572605)
  • Update latest snap builds table via AJAX
  • Add option to trigger snap builds when top-level branch changes (#1593359)
  • Add processor selection in new snap form
  • Add option to automatically release snap builds to store channels after upload (#1597819)
  • Allow manually uploading a completed snap build to the store
  • Upload *.manifest files from builders as well as *.snap (#1608432)
  • Send an email notification for general snap store upload failures (#1632299)
  • Allow building snaps from an external Git repository
  • Move upload to FAILED if its build was deleted (e.g. because of a deleted snap) (#1655334)
  • Consider snap/snapcraft.yaml and .snapcraft.yaml as well as snapcraft.yaml for new snaps (#1659085)
  • Add support for building snaps with classic confinement (#1650946)
  • Fix builds_for_snap to avoid iterating over an unsliced DecoratedResultSet (#1671134)
  • Add channel track support when uploading snap builds to the store (contributed by Matias Bordese; #1677644)

Soyuz (package management)

  • Remove some more uses of the confusing .dsc component; add the publishing component to SourcePackage:+index in compensation
  • Add include_meta option to SPPH.sourceFileUrls, paralleling BPPH.binaryFileUrls
  • Kill debdiff after ten minutes or 1GiB of output by default, and make sure we clean up after it properly (#314436)
  • Fix handling of << and >> dep-waits
  • Allow PPA admins to set external_dependencies on individual binary package builds (#671190)
  • Fix NascentUpload.do_reject to not send an erroneous Accepted email (#1530220)
  • Include DEP-11 metadata in Release file if it is present
  • Consistently generate Release entries for uncompressed versions of files, even if they don’t exist on the filesystem; don’t create uncompressed Packages/Sources files on the filesystem
  • Handle Build-Depends-Arch and Build-Conflicts-Arch from SPR.user_defined_fields in Sources generation and SP:+index (#1489044)
  • Make index compression types configurable per-series, and add xz support (#1517510)
  • Use SHA-512 digests for GPG signing where possible (#1556666)
  • Re-sign PPAs with SHA-512
  • Publish by-hash index files (#1430011)
  • Show SHA-256 checksums rather than MD5 on DistributionSourcePackageRelease:+files (#1562632)
  • Add a per-series switch allowing packages in supported components to build-depend on packages in unsupported components, used for Ubuntu 16.04 and later
  • Expand archive signing to kernel modules (contributed by Andy Whitcroft; #1577736)
  • Uniquely index PackageDiff(from_source, to_source) (part of #1475358)
  • Handle original tarball signatures in source packages (#1587667)
  • Add signed checksums for published UEFI/kmod files (contributed by Andy Whitcroft; #1285919)
  • Add support for named authentication tokens for private PPAs
  • Show explicit add-apt-repository command on Archive:+index (#1547343)
  • Use a per-archive OOPS timeline in archivepublisher scripts
  • Link to package versions on DSP:+index using fmt:url rather than just a relative link to the version, to avoid problems with epochs (#1629058)
  • Fix RepositoryIndexFile to gzip without timestamps
  • Fix Archive.getPublishedBinaries API call to have a constant query count (#1635126)
  • Include the package name in package copy job OOPS reports and emails (#1618133)
  • Remove headers from Contents files (#1638219)
  • Notify the Changed-By address for PPA uploads if the .changes contains “Launchpad-Notify-Changed-By: yes” (#1633608)
  • Accept .debs containing control.tar.xz (#1640280)
  • Add Archive.markSuiteDirty API call to allow requesting that a given archive/suite be published
  • Don’t allow cron-control to interrupt publish-ftpmaster part-way through (#1647478)
  • Optimise non-SQL time in PublishingSet.requestDeletion (#1682096)
  • Store uploaded .buildinfo files (#1657704)

Translations

  • Allow TranslationImportQueue to import entries from file objects rather than having to read arbitrarily-large files into memory (#674575)

Miscellaneous

  • Use gender-neutral pronouns where appropriate
  • Self-host the Ubuntu webfonts (#1521472)
  • Make the beta and privacy banners float over the rest of the page when scrolling
  • Upgrade to pytz 2016.4 (#1589111)
  • Publish Launchpad’s code revision in an X-Launchpad-Revision header
  • Truncate large picker search results rather than refusing to display anything (#893796)
  • Sync up the lists footer with the main webapp footer a bit (#1679093)

Read more
Colin Watson

Here’s what the Launchpad team did in October.

Code

  • Ensure that enabled and restricted processors are left untouched when submitting Archive:+edit (#1501519)
  • Describe merge proposal listings in the second person (#991945)
  • Increase Git backend communication timeout to 30 seconds (from 5), as detecting merges can be quite slow on large repositories (#1502748)
  • Link directly to the recipe/snap in the one-recipe/snap case, rather than redirecting from +recipes/+snaps views (#891918)
  • Add GitRepository:+subscription traversal (#1503749)
  • Send notification mail when a merge proposal is superseded (#716169)
  • Add Branch:+dependent-merges and GitRef:+dependent-merges views (#496056)
  • Parse extended header lines in git diffs correctly (#1510337)

Additionally, we’ve added webhook support for merge proposals, made a few more UI improvements, and enabled webhooks for beta testers.  Feedback welcome!

Snap packages

  • Allow snap package owners to enable/disable unrestricted processors (matches the change made for PPAs last month)

Soyuz (package building)

  • Launchpad now accepts uploads of binary packages from builders with file modification times up to 24 hours in the future (rather than 8) or as early as 1975 (rather than 1984), to match Debian
  • Notifications are no longer generated for package copy jobs that in fact end up being no-ops

Also, as previously announced, we and our sysadmins have been putting a lot of work into extending ScalingStack to other architectures and making it more reliable, and as a result we can now offer general access to PPAs for the ppc64el architecture.

Read more
Colin Watson

October already! As the leaves start to turn red here in the northern hemisphere, here’s a brief summary of what we did in September.

Bugs

  • BugTask:+addcomment’s title doesn’t duplicate the bug number (#1323808)
  • The duplicates portlet of bug tasks no longer links to invisible private bugs and now uses the correct sprite for each bug (#1443418, #1465880)
  • Show hidden bug comments to their owners (#1391394)

Code

  • Line numbers in merge proposal preview diffs are now unselectable (#1483925)
  • Branch revision listings link more sensibly to merged branches and merge proposals (#711647)
  • Git repositories and refs have more sensible breadcrumb links (#1466271)
  • Source package recipe builds now include the distribution series in their titles (#1491336)
  • Allow archive owners to cancel their recipe builds, and make recipes use the normal cancellation infrastructure (#624630)
  • Make it possible to retry superseded builds, since there are situations where they can become unsuperseded (#444030)
  • The index page for Git refs is more useful
  • Fix crashes on various +activereviews pages when there are no active reviews (#1499744, #1501134)
  • Precache branch permissions in Branch.landing_candidates, fixing API timeouts (#1500576)
  • Add webhook support for Bazaar branches, currently enabled on qastaging though not yet on production

Mail

  • Team membership notifications now honour the “Include filtering information in email footers” setting, and have rationale information in the headers and footer (#296889, #508897)
  • There is a new X-Launchpad-Message-For mail header (or Launchpad-Message-For in expanded footers) giving just the name of the person or team directly subscribed to the notification (#1493844)
  • Treat “me” in person-or-team contexts in mail handlers as the current user (#340397)

We now consider the “Gmail filtering improvements” work complete.  Let us know if there are further categories of mail sent by Launchpad that you’re finding difficult to filter using Gmail.

Registry

  • Distribution index pages hide links to disabled features (#80315, #257627)
  • Set consistent colour for the “Opinion” bug task status (#648645)
  • Blueprints on milestone pages have icons again and are sortable (#1354387)
  • Old user-to-user email database references no longer prevent changing team privacy (#1498497)

Snap packages

  • Snap packages now have edit, administer, and delete views
  • Branches, Git repositories and refs, people, teams, and products now have snap package listing views
  • Members of ~launchpad-snap-builders can create snap packages in the web UI based on Bazaar or Git branches
  • Snap package owners can request builds in the web UI

Soyuz (package building)

  • Stop showing the confusing .dsc component on source package index pages (#521722)
  • Export archive deletion on the API (#814633)
  • Allow overriding the build version for live filesystem builds (#1496074)
  • Archive owners can change the “build debug symbols” and “publish debug symbols” settings on their own archives, rather than needing to ask an admin
  • The publisher no longer sometimes tries to update Contents files for immutable suites such as the release pocket of stable series, which could cause checksums in Release to get out of sync (#1448270)
  • Publishing PPAs now creates clearsigned InRelease files to improve reliability of updates (#804252); we intend to add this to the primary Ubuntu archive as well once one last piece of mirroring infrastructure is made ready for it
  • Archive owners can enable or disable unrestricted architectures on their own archives, rather than needing to ask an admin; we will unrestrict further architectures once they have adequate virtualised build infrastructure available
  • Enabling a test rebuild archive no longer times out (#1500973)

Read more
Colin Watson

Here’s a summary of what the Launchpad team got up to in August.

Code

  • Webhook support for Git repositories is almost finished, and only needs a bit more web UI work (#1474071)
  • The summary of merge proposal pages now includes a link to the merged revision, if any (#892259)
  • Viewing individual comments on Git-based merge proposals no longer OOPSes (#1485907)

Mail notifications

Our internal stakeholders in Canonical recently asked us to work on improving the ability to filter Launchpad mail using Gmail.  The core of this was the “Include filtering information in email footers” setting that we added recently, but we knew there was some more to do.  Launchpad’s mail notification code includes some of the oldest and least consistent code in our tree, and so improving this has entailed paying off quite a bit of technical debt along the way.

  • Bug notifications and package upload notifications now honour the “Include filtering information in email footers” setting (#1474071)
  • Bug notifications now log an OOPS rather than crashing if the SMTP server rejects an individual message (#314420, #916939)
  • Recipe build notifications now include an X-Launchpad-Archive header (#776160)
  • Question notification rationales are now more consistent, including team annotations for subscribers (#968578)
  • Package upload notifications now include X-Launchpad-Message-Rationale and X-Launchpad-Notification-Type headers, and have more specific footers (#117155, #127917)

Package build infrastructure

  • Launchpad now supports building source packages that use Debian’s new build profiles syntax, currently only with no profiles activated
  • Launchpad can now build snap packages (#1476405), with some limitations; this is currently only available to a group of alpha testers, so let us know if you’re interested
  • Builders can now access Launchpad’s Git hosting (HTTPS only) in the same way that they can access its Bazaar hosting
  • All amd64/i386 builds now take place in ScalingStack, and the corresponding bare-metal builders have been detached pending decommissioning; some of the newer of those machines will be used to further expand ScalingStack capacity
  • We have a new ScalingStack region including POWER8-based ppc64el builders, which is currently undergoing production testing; this will replace the existing POWER7-based builders in a few weeks, and also provide virtualised build capacity for ppc64el PPAs
  • We’ve fixed a race condition that sometimes caused a user’s first PPA to be published unsigned for a while (#374395)

Miscellaneous

  • The project release file upload limit is now 1 GiB rather than 200 MiB (#1479441)
  • We spent some more time supporting translations for the overlay PPA used for current Ubuntu phone images, copying a number of existing translations into place from before the point when they were redirected automatically
  • Your user index page now has a “Change password” link (#1471961)
  • Bug attachments are no longer incorrectly hidden when displaying only some bug comments (#1105543)

Read more
Colin Watson

Here’s a summary of what the Launchpad team got up to in July.

Code

  • We fixed a regression in the wrapping layout of side-by-side diffs on bazaar.launchpad.net (#1436483)
  • Various code pages now have meta tags to redirect “go get" to the appropriate Bazaar or Git URL, allowing the removal of special-casing from the “go" tool (#1465467)
  • Merge proposal diffs including mention of binary patches no longer crash the new-and-improved code review comment mail logic (#1471426), and we fixed some line-counting bugs in that logic as well (#1472045)
  • Links to the Git code browsing interface now use shorter URL forms

We’ve also made a fair amount of progress on adding support for triggering webhooks from Launchpad (#342729), which will initially be hooked up for pushes to Git repositories.  The basic code model, webservice API, and job retry logic are all in place now, but we need to sort out a few more things including web UI and locking down the proxy configuration before we make it available for general use.  We’ll post a dedicated article about this once the feature becomes available.

Mail notifications

We posted recently about improved filtering options (#1474071).  In the process of doing so, we cleaned up several older problems with the mails we send:

  • Notifications for a bug’s initial message no longer include a References header, which confuses some versions of some mail clients (#320034)
  • Package upload notifications no longer attempt to transliterate non-ASCII characters in package maintainer names into ASCII equivalents; they now use RFC2047 encoding instead (#362957)
  • Notifications about duplicate bugs now include an X-Launchpad-Bug-Duplicate header (#363995)
  • Package build failure notifications now include a “You are receiving this email because …” rationale (#410893)

Package build infrastructure

  • The sbuild upgrade last month introduced some regressions in our handling of package builds that need to wait for dependencies (e.g. #1468755), and it’s taken a few goes to get this right; this is somewhat improved now, and the next builder deployment will fix all the currently-known bugs in this area
  • In the same area, we’ve made some progress on adding minimal support for Debian’s new build profiles syntax, applying fixes to upload processing and dependency-wait analysis, although this should still be considered bleeding-edge and unlikely to work from end to end
  • We’ve been working on adding support for building snap packages (#1476405), but there’s still more to do here; we should be able to make this available to some alpha testers around mid-August

Miscellaneous

  • We’ve arranged to redirect translations for the overlay PPA used for current Ubuntu phone images to the ubuntu-rtm/15.04 series so that they can be translated effectively (#1463723); we’re still working on copying translations into place from before this fix
  • Projects and project groups no longer have separately-editable “display name” and “title” fields, which were very similar in purpose; they now just have display names (#1853, #4449)
  • Cancelled live file system builds are sorted to the end of the build history, rather than the start (#1424672)

Read more
Curtis Hovey

I have been analysing Launchpad’s critical bugs to track the Purple squad’s progress while on Launchpad maintenance duty. In January of 2011, the Cloud Engineering team né Launchpad Engineering team was reorganised into squads, where one or more squads would maintain Launchpad while other squads work on features. This change also aligned with a new found effort to enforce the zero-oops policy. The two maintenance squads had more than 332 critical bugs to close before we could consider adding features that the stakeholders and community wanted. By July 2011, the count dropped to its lowest point, 250 known critical bugs. Why did the count stop falling for fifteen months? Why is the count falling again?

Charting and analysing critical bugs

Chart of Launchpad's critical bugs since the formation of Launchpad squads and maintenance duties
The chart above needs some explanation to understand what is happening in Launchpad’s critical bugs over time. (You may want to open the image in a separate window to see everything in detail.) Each iteration is one week. The backlog represent the open critical bugs in launchpad at the start of the iteration. The future bugs are either bugs that are not discovered, not introduced, or reported and fixed within the iteration. The last group is crucial to understand the lines plotting the number of bugs fixed and added during the iteration. We strive to close critical bugs immediately. Most critical bugs are reported and fixed in a few days, so most bugs were not open long enough to be show up in the backlog. The number of bugs fixed must exceed the number added to make the backlog count fall. You can see that the maintenance squads have always been burning down the critical bugs, but if you are just watching the number of open bugs in Launchpad, you get the sense that the squads are running to just stand still.

I use the lp-milestone YUI widget to chart the bugs and analyse the our progress through the critical bugs. It allows me to summarise a set of bugs, or analyse a subset by bug tag.

Launchpad maintenance analysis -- driving critical bugs to zero

Though 22 bugs were fixed this past week, 14 were added, thus the critical count dropped by 8. The last eight iterations are used to calculate the average bugs closed and open per iteration. The relative velocity (velocity – flux) is used to estimate the remaining number of days to drive the count to zero. When the Purple squad started maintenance on September 10th of 2012, the estimated days of effort was more than 1,200. In just three weeks, the number has fallen dramatically. The principle reason the backlog of critical bugs has fallen is that the Purple squad is now giving those bugs their full attention, but that generalisation is unsatisfactory.

Why is the Purple squad so good at closing bugs in the critical backlog?

I do not know the answer to my question. The critical backlog reached its all-time low of 250 bugs with the release of the Purple squad’s maintenance work in July 2011. There was supposition that  Purple fixed the easy bugs, or that the fixes did not address the root cause, so another critical bug was opened. I disagree. The squad had no trouble finding easy bugs, and it too would have been fixing secondary bugs if the first fix was incomplete. I can tell you how the squad works on critical bugs, but not why it is successful.

I was surprised to see the Purple squad were still the top critical bug closers when it returned to maintenance after 15 months of feature work. How could that be?  The squad fixed a lot of old timeout and JavaScript bugs in the last few months through systemic changes — enough to significantly affect the statistics. About 600 critical bugs were closed while Purple squad were on feature work. The squad closed 210 of those bugs. 60 were regressions that were fixed within the iteration, so they never showed up in the backlog. 70 critical bugs were fixed because they blocked the feature, and 80 critical bugs were because Purple was the only squad awake when the issue was reported. The 4 other squads fixed an average of 98 bugs each when they were on maintenance. The Purple squad fixed more bugs then maintenance squads on average even when they were not officially doing maintenance work.  The data, charts, and analysis always includes the Purple squad.

I suspect the Purple squad has more familiarity with bugs in the critical backlog. They never stopped reading the critical bugs when they were on feature work. They saw opportunities to fix critical bugs while solving feature problems. I know some of the squad members are subscribed to all critical bugs and re-read them often. They triage and re-triage Launchpad bugs. This familiarity means that many bugs are ready to code — they know where the problem is and how to fix it before the work is assigned to them. They fixed many bugs in less than a day, often doing exactly what was suggested in the bug comments.

During the first week of their return to maintenance, about 30 critical bugs were discovered to be dupes of other bugs. Though this change does make the backlog count fall, it also revised all the data, so the chart is not showing these 30 bugs as at all now. The decline of backlog bugs does not include dupes. While the squad was familiar enough to find many bugs that they close in a single day, they were not so familiar as to have known that there were 30 duplicate bugs in the backlog when they started.

Most squad have only one person with DB access, but the Purple squad is blessed with 3 people who can test queries against production-level data. This could be a significant factor. It is nigh impossible to fix a timeout bug without proper database testing. Only 13 of the recent bugs closed were timeouts though. The access also helps plan proper fixes for other bugs as well, so maybe 20% of the fixed bugs can be attributed to database access.

Maybe the Purple squad are better maintenance engineers than other squads who work on maintenance. For 28 months, I was the leading bug closer working on Launchpad. I closed 3 times more bugs than the average Launchpad engineer. I am not a great engineer though. My “winning” streak came to a closed shortly after William Grant started working on Launchpad full time; he soundly trounced me over several months. Then he and I were put on the same squad and asked to fix critical bugs. Purple also had Jon Sackett, who was closing almost 2 times the number bugs than the other engineers. I don’t think I need to be humble on this matter. To use the vulgar, we rocked! Ian was the odd man on the Purple squad. He was the slowest bug closer, often going beyond our intended scope to fix an issue. Then Purple switched to feature work…Ian lept to the first rank while the rest of the squad struggled. Ian fixed almost double the number of Disclosure bugs than other squad members. The leading critical bug closer on the squad at the moment though is Steve Kowalik. This is his first time working on maintenance. His productivity has jumped since transitioning to maintenance.

I can only speculate as to why some engineers are better at maintenance, or can just close more bugs than others. A maintenance engineer must be familiar with the code and the rules that guide it. Feature engineers need to analyse issues and create new rules to guide code. I did not gradually become a leading bug closer, it happen in a single day when I realised while solving one issue that the code I was looking at was flawed, it certainly was causing a bug, I knew how to fix it, and with a few extra hours of extra effort, I could close two bugs in a single day. Closing bugs has always been easy since that moment.

I believe the Purple squad values certainty over severity and small scope over large scope when choosing which critical backlog bugs to fix. I created several charts that break the critical bugs into smaller categories. I suggested the squad burn down sub-categories of bugs like regressions, or 404s. The squad members are instead fixing bugs from the entire backlog. They are choosing bugs that they are certain they can fix in a few hours.  I think the squad has tacitly agreed to fix bugs that are less than a day of effort. When this group is exhausted, they will fix issues that require days of effort, but also fix as many bugs. The last bugs to be fixed will be those that require many days to fix a single bug. Fixing the bugs with the highest certainty reduced our churn through the critical bugs, there are fewer to triage, to dupe, to get ready to code.

The Purple squad avoids doing feature-level design and effort to fix critical bugs. Feature-level efforts entail more risk, more planning, and much more time. There is often no guarantee, low certainty, that a feature will fix the issue. A faster change with higher certainty can fix the issue, but leaves cruft in the code that the engineers do not like. Choosing to do feature-level fixes when a more certain fix is available indicates there is tension between the Launchpad users who have a “critical” issue that stops them from using Launchpad, and the engineers who have a “high” issue maintaining mediocre code. I contend it is easier to do feature-level work when you are not interrupted with maintenance issues. When the Purple squad does choose to do feature-level work to fix a critical, they have a list of the bugs they expect to fix, and they cut scope when fixing a single bug delays the fix of the others. The Launchpad Answers email subsystem was re-written when other options were not viable, there we about 20 leading timeouts represented by 5 specific bugs to justify 10 days of effort to fix them.

The Purple squad is not unique

Nothing that I have written explains why the Purple squad are better are closing critical bugs. All squads have roughly the same skills and make decisions like Purple. Maybe the issue is just a matter of degree. If the maintenance squad is not closing enough bugs to burn down the backlog, their time is consumed by triaging and duping new critical bug reports. Familiarity with Launchpad’s 1000’s of bugs is an advantage when triaging bugs and getting a bug ready to code. Being able to test queries yourself on a production-level database takes hours or days off the time needed to fix an issue. Familiarity with the code and the reasoning that guided it increases the certainty of success. The only domain that Purple is not comfortable working with is lp.translations; the squad is comfortable changing 90% of Launchpad’s code. There may be correlation between familiarity with code, and the facts that the squad members participated in the apocalypse that  re-organised the code base, and that some have a LoC credit count in the 1000’s.

Read more
Martin Pool

We’ve just upgraded Launchpad’s builder machines to Bazaar 2.4. Most importantly, this means that recipe builds of very large trees will work reliably, such as the daily builds of the Linaro ARM-optimized gcc. (This was bug 746822 in Launchpad).

We are going to do some further rollouts over the next week to improve supportability of recipe builds, support building non-native packages, handle muiltiarch package dependencies, improve the buildd deployment story etc.

Read more
Martin Pool

Continuing on from our earlier work of sending less but better mail and making it faster to import i18n translation templates: Launchpad will no longer send mail when it successfully imports a template. You can see in the web ui when the template was last imported, and you will still get mail if there’s a problem.

I could hardly put it better than Riddell:

Danilo asked for my reasoning. My reasoning is that pointless e-mails are a pain.

Big pile of junk mail from Verizon

(I hope we’ll eventually have a more structured notification model, that will let you choose to see some notifications by mail and others in the web ui. One step at a time.)

Read more