Canonical Voices

Posts tagged with 'front-page'

Colin Watson

Well, it’s been a while!  Since we last posted a general update, the Launchpad team has become part of Canonical’s Online Services department, so some of our efforts have gone into other projects.  There’s still plenty happening with Launchpad, though, and here’s a changelog-style summary of what we’ve been up to.

Answers

  • Lock down question title and description edits from random users
  • Prevent answer contacts from editing question titles and descriptions
  • Prevent answer contacts from editing FAQs

Blueprints

  • Optimise SpecificationSet.getStatusCountsForProductSeries, fixing Product:+series timeouts
  • Add sprint deletion support (#2888)
  • Restrict blueprint count on front page to public blueprints

Build farm

  • Add fallback if nominated architecture-independent architecture is unavailable for building (#1530217)
  • Try to load the nbd module when starting launchpad-buildd (#1531171)
  • Default LANG/LC_ALL to C.UTF-8 during binary package builds (#1552791)
  • Convert buildd-manager to use a connection pool rather than trying to download everything at once (#1584744)
  • Always decode build logtail as UTF-8 rather than guessing (#1585324)
  • Move non-virtualised builders to the bottom of /builders; Ubuntu is now mostly built on virtualised builders
  • Pass DEB_BUILD_OPTIONS=noautodbgsym during binary package builds if we have not been told to build debug symbols (#1623256)

Bugs

  • Use standard milestone ordering for bug task milestone choices (#1512213)
  • Make bug activity records visible to anonymous API requests where appropriate (#991079)
  • Use a monospace font for “Add comment” boxes for bugs, to match how the comments will be displayed (#1366932)
  • Fix BugTaskSet.createManyTasks to map Incomplete to its storage values (#1576857)
  • Add basic GitHub bug linking (#848666)
  • Prevent rendering of private team names in bugs feed (#1592186)
  • Update CVE database XML namespace to match current file on cve.mitre.org
  • Fix Bugzilla bug watches to support new versions that permit multiple aliases
  • Sort bug tasks related to distribution series by series version rather than series name (#1681899)

Code

  • Remove always-empty portlet from Person:+branches (#1511559)
  • Fix OOPS when editing a Git repository with over a thousand refs (#1511838)
  • Add Git links to DistributionSourcePackage:+branches and DistributionSourcePackage:+all-branches (#1511573)
  • Handle prerequisites in Git-based merge proposals (#1489839)
  • Fix OOPS when trying to register a Git merge with a target path but no target repository
  • Show an “Updating repository…” indication when there are pending writes
  • Launchpad’s Git hosting backend is now self-hosted
  • Fix setDefaultRepository(ForOwner) to cope with replacing an existing default (#1524316)
  • Add “Configure Code” link to Product:+git
  • Fix Git diff generation crash on non-ASCII conflicts (#1531051)
  • Fix stray link to +editsshkeys on Product:+configure-code when SSH keys were already registered (#1534159)
  • Add support for Git recipes (#1453022)
  • Fix OOPS when adding a comment to a Git-based merge proposal without using AJAX (#1536363)
  • Fix shallow git clones over HTTPS (#1547141)
  • Add new “Code” portlet on Product:+index to make it easier to find source code (#531323)
  • Add missing table around widget row on Product:+configure-code, so that errors are highlighted properly (#1552878)
  • Sort GitRepositorySet.getRepositories API results to make batching reliable (#1578205)
  • Show recent commits on GitRef:+index
  • Show associated merge proposals in Git commit listings
  • Show unmerged and conversation-relevant Git commits in merge proposal views (#1550118)
  • Implement AJAX revision diffs for Git
  • Fix scanning branches with ghost revisions in their ancestry (#1587948)
  • Fix decoding of Git diffs involving non-UTF-8 text that decodes to unpaired surrogates when treated as UTF-8 (#1589411)
  • Fix linkification of references to Git repositories (#1467975)
  • Fix +edit-status for Git merge proposals (#1538355)
  • Include username in git+ssh URLs (#1600055)
  • Allow linking bugs to Git-based merge proposals (#1492926)
  • Make Person.getMergeProposals have a constant query count on the webservice (#1619772)
  • Link to the default git repository on Product:+index (#1576494)
  • Add Git-to-Git code imports (#1469459)
  • Improve preloading of {Branch,GitRepository}.{landing_candidates,landing_targets}, fixing various timeouts
  • Export GitRepository.getRefByPath (#1654537)
  • Add GitRepository.rescan method, useful in cases when a scan crashed

Infrastructure

  • Launchpad’s SSH endpoints (bazaar.launchpad.net, git.launchpad.net, upload.ubuntu.com, and ppa.launchpad.net) now support newer key exchange and MAC algorithms, allowing compatibility with OpenSSH >= 7.0 (#1445619)
  • Make cross-referencing code more efficient for large numbers of IDs (#1520281)
  • Canonicalise path encoding before checking a librarian TimeLimitedToken (#677270)
  • Fix Librarian to generate non-cachable 500s on missing storage files (#1529428)
  • Document the standard DELETE method in the apidoc (#753334)
  • Add a PLACEHOLDER account type for use by SSO-only accounts
  • Add support to +login for acquiring discharge macaroons from SSO via an OpenID exchange (#1572605)
  • Allow managing SSH keys in SSO
  • Re-raise unexpected HTTP errors when talking to the GPG key server
  • Ensure that the production dump is usable before destroying staging
  • Log SQL statements as Unicode to avoid confusing page rendering when the visible_render_time flag is on (#1617336)
  • Fix the librarian to fsync new files and their parent directories
  • Handle running Launchpad from a Git working tree
  • Handle running Launchpad on Ubuntu 16.04 (upgrade currently in progress)
  • Fix delete_unwanted_swift_files to not crash on segments (#1642411)
  • Update database schema for PostgreSQL 9.5 and 9.6
  • Check fingerprints of keys received from the keyserver rather than trusting it implicitly

Registry

  • Make public SSH key records visible to anonymous API requests (#1014996)
  • Don’t show unpublished packages or package names from private PPAs in search results from the package picker (#42298, #1574807)
  • Make Person.time_zone always be non-None, allowing us to easily show the edit widget even for users who have never set their time zone (#1568806)
  • Let latest questions, specifications and products be efficiently calculated
  • Let project drivers edit series and productreleases, as series drivers can; project drivers should have series driver power over all series
  • Fix misleading messages when joining a delegated team
  • Allow team privacy changes when referenced by CodeReviewVote.reviewer or BugNotificationRecipient.person
  • Don’t limit Person:+related-projects to a single batch

Snappy

  • Add webhook support for snaps (#1535826)
  • Allow deleting snaps even if they have builds
  • Provide snap builds with a proxy so that they can access external network resources
  • Add support for automatically uploading snap builds to the store (#1572605)
  • Update latest snap builds table via AJAX
  • Add option to trigger snap builds when top-level branch changes (#1593359)
  • Add processor selection in new snap form
  • Add option to automatically release snap builds to store channels after upload (#1597819)
  • Allow manually uploading a completed snap build to the store
  • Upload *.manifest files from builders as well as *.snap (#1608432)
  • Send an email notification for general snap store upload failures (#1632299)
  • Allow building snaps from an external Git repository
  • Move upload to FAILED if its build was deleted (e.g. because of a deleted snap) (#1655334)
  • Consider snap/snapcraft.yaml and .snapcraft.yaml as well as snapcraft.yaml for new snaps (#1659085)
  • Add support for building snaps with classic confinement (#1650946)
  • Fix builds_for_snap to avoid iterating over an unsliced DecoratedResultSet (#1671134)
  • Add channel track support when uploading snap builds to the store (contributed by Matias Bordese; #1677644)

Soyuz (package management)

  • Remove some more uses of the confusing .dsc component; add the publishing component to SourcePackage:+index in compensation
  • Add include_meta option to SPPH.sourceFileUrls, paralleling BPPH.binaryFileUrls
  • Kill debdiff after ten minutes or 1GiB of output by default, and make sure we clean up after it properly (#314436)
  • Fix handling of << and >> dep-waits
  • Allow PPA admins to set external_dependencies on individual binary package builds (#671190)
  • Fix NascentUpload.do_reject to not send an erroneous Accepted email (#1530220)
  • Include DEP-11 metadata in Release file if it is present
  • Consistently generate Release entries for uncompressed versions of files, even if they don’t exist on the filesystem; don’t create uncompressed Packages/Sources files on the filesystem
  • Handle Build-Depends-Arch and Build-Conflicts-Arch from SPR.user_defined_fields in Sources generation and SP:+index (#1489044)
  • Make index compression types configurable per-series, and add xz support (#1517510)
  • Use SHA-512 digests for GPG signing where possible (#1556666)
  • Re-sign PPAs with SHA-512
  • Publish by-hash index files (#1430011)
  • Show SHA-256 checksums rather than MD5 on DistributionSourcePackageRelease:+files (#1562632)
  • Add a per-series switch allowing packages in supported components to build-depend on packages in unsupported components, used for Ubuntu 16.04 and later
  • Expand archive signing to kernel modules (contributed by Andy Whitcroft; #1577736)
  • Uniquely index PackageDiff(from_source, to_source) (part of #1475358)
  • Handle original tarball signatures in source packages (#1587667)
  • Add signed checksums for published UEFI/kmod files (contributed by Andy Whitcroft; #1285919)
  • Add support for named authentication tokens for private PPAs
  • Show explicit add-apt-repository command on Archive:+index (#1547343)
  • Use a per-archive OOPS timeline in archivepublisher scripts
  • Link to package versions on DSP:+index using fmt:url rather than just a relative link to the version, to avoid problems with epochs (#1629058)
  • Fix RepositoryIndexFile to gzip without timestamps
  • Fix Archive.getPublishedBinaries API call to have a constant query count (#1635126)
  • Include the package name in package copy job OOPS reports and emails (#1618133)
  • Remove headers from Contents files (#1638219)
  • Notify the Changed-By address for PPA uploads if the .changes contains “Launchpad-Notify-Changed-By: yes” (#1633608)
  • Accept .debs containing control.tar.xz (#1640280)
  • Add Archive.markSuiteDirty API call to allow requesting that a given archive/suite be published
  • Don’t allow cron-control to interrupt publish-ftpmaster part-way through (#1647478)
  • Optimise non-SQL time in PublishingSet.requestDeletion (#1682096)
  • Store uploaded .buildinfo files (#1657704)

Translations

  • Allow TranslationImportQueue to import entries from file objects rather than having to read arbitrarily-large files into memory (#674575)

Miscellaneous

  • Use gender-neutral pronouns where appropriate
  • Self-host the Ubuntu webfonts (#1521472)
  • Make the beta and privacy banners float over the rest of the page when scrolling
  • Upgrade to pytz 2016.4 (#1589111)
  • Publish Launchpad’s code revision in an X-Launchpad-Revision header
  • Truncate large picker search results rather than refusing to display anything (#893796)
  • Sync up the lists footer with the main webapp footer a bit (#1679093)

Read more
Colin Watson

Launchpad has had Git-to-Bazaar code imports since 2009, along with imports from a few other systems.  These form part of Launchpad’s original mission to keep track of free software, regardless of where it’s hosted.  They’re also very useful for automatically building other artifacts, such as source package recipes or snap packages, from code hosted elsewhere.  Unfortunately they’re quite complicated: they need to be able to do a full round-trip conversion of every revision from the other version control system, which has made it difficult to add support for Git features such as signed commits or submodules.  Once one of these features is present anywhere in the history of a branch, importing it to Bazaar becomes impossible.  This has been a headache for many users.

We can do better nowadays.  As of last year, we have direct Git hosting support in Launchpad, and we can already build snaps and recipes straight from Git, so we can fulfil our basic goal more robustly now with a lot less code.  So, Launchpad now supports Git-to-Git code imports, also known as Git mirroring.  You can use this to replace many uses of Git-to-Bazaar imports (note that there’s no translations integration yet, and of course you won’t be able to branch the resulting import using bzr).

See our Git documentation for more details.

Read more
Colin Watson

We just rolled out a new feature for Launchpad’s Git repository hosting: Git-based merge proposals can now be linked to Launchpad bugs.  This can be done manually from the web UI for the merge proposal, but normally you should just mention the Launchpad bug in the commit message of one of the commits you want to merge.  The required commit message text to link to bugs #XXX and #YYY looks like this:

LP: #XXX, #YYY

This is the same form used for Launchpad bug references in debian/changelog files in source packages, and the general approach of mentioning bugs in commit messages is similar to that of various other hosting sites.

Bugs are not automatically closed when merge proposals land, because the policy for when that should happen varies from project to project: for example, projects often only close bugs when they make releases, or when their code is deployed to production sites.

Users familiar with Bazaar on Launchpad should note that the model for Git bug linking is slightly different: bugs are linked to merge proposals rather than to individual branches.  This difference is mainly because individual branches within a Git repository are often much more ephemeral than Bazaar branches.

Documentation is here, along with other details of Launchpad’s Git hosting.

Read more
Colin Watson

Here’s what the Launchpad team did in October.

Code

  • Ensure that enabled and restricted processors are left untouched when submitting Archive:+edit (#1501519)
  • Describe merge proposal listings in the second person (#991945)
  • Increase Git backend communication timeout to 30 seconds (from 5), as detecting merges can be quite slow on large repositories (#1502748)
  • Link directly to the recipe/snap in the one-recipe/snap case, rather than redirecting from +recipes/+snaps views (#891918)
  • Add GitRepository:+subscription traversal (#1503749)
  • Send notification mail when a merge proposal is superseded (#716169)
  • Add Branch:+dependent-merges and GitRef:+dependent-merges views (#496056)
  • Parse extended header lines in git diffs correctly (#1510337)

Additionally, we’ve added webhook support for merge proposals, made a few more UI improvements, and enabled webhooks for beta testers.  Feedback welcome!

Snap packages

  • Allow snap package owners to enable/disable unrestricted processors (matches the change made for PPAs last month)

Soyuz (package building)

  • Launchpad now accepts uploads of binary packages from builders with file modification times up to 24 hours in the future (rather than 8) or as early as 1975 (rather than 1984), to match Debian
  • Notifications are no longer generated for package copy jobs that in fact end up being no-ops

Also, as previously announced, we and our sysadmins have been putting a lot of work into extending ScalingStack to other architectures and making it more reliable, and as a result we can now offer general access to PPAs for the ppc64el architecture.

Read more
Colin Watson

If you are a member of Launchpad’s beta testers team, you can now try out webhooks for Bazaar branches and Git repositories. These can be used to set up integration with external sites for various purposes, such as running CI jobs or publishing documentation. We expect to open this up to all Launchpad users soon, but in the meantime please do file a bug against Launchpad itself if you encounter any problems.

See our webhooks documentation for more details.

Update: as of 2015-11-20, this feature is enabled for all Launchpad users.

Read more
Colin Watson

PPAs for ppc64el

Personal package archives on Launchpad only build for the amd64 and i386 architectures by default, which meets most people’s needs.  Anyone with an e-mail address can have a PPA, so they have to be securely virtualised, but that’s been feasible on x86 for a long time.  Dealing with the other architectures that Ubuntu supports (currently arm64, armhf, powerpc, and ppc64el) in a robust and scalable way has been harder.  Until recently, all of those architectures were handled either by running one builder per machine on bare metal, or in some cases by running builders on a small number of manually-maintained persistent virtual machines per physical machine.  Neither of those approaches scales to the level required to support PPAs, and we need to make sure that any malicious code run by a given build is strictly confined to that build.  (We support virtualised armhf PPAs, but only by using qemu-user-static in an amd64 virtual machine, which is very fragile and there are many builds that it simply can’t handle at all.)

We’ve been working with our sysadmins for several months to extend ScalingStack to non-x86 architectures, and at the start of Ubuntu’s 16.04 development cycle we were finally able to switch all ppc64el builds over to this system.  Rather than four builders, we now have 30, each of which is reset to a clean virtual machine instance between each build.  Since that’s more than enough to support Ubuntu’s needs, we’ve now “unrestricted” the architecture so that it can be used for PPAs as well, and PPA owners can enable it at will.  To do this, visit the main web page for your PPA (which will look something like “https://launchpad.net/~<person-name>/+archive/ubuntu/<ppa-name>”) and follow the “Change details” link; you’ll see a list of checkboxes under “Processors”, and you can enable or disable any that aren’t greyed out.  This also means that you can disable amd64 or i386 builds for your PPA if you want to.

We’re working to extend this to all the existing Ubuntu architectures at the moment.  arm64 is up and running but we’re still making sure it’s sufficiently robust; armhf will run on arm64 guests, and just needs a kernel patch to set its uname correctly; and powerpc builds will run in different guests on the same POWER8 compute nodes as ppc64el once we have suitable cloud images available.  We’ll post further announcements when further architectures are unrestricted.

Read more
Colin Watson

October already! As the leaves start to turn red here in the northern hemisphere, here’s a brief summary of what we did in September.

Bugs

  • BugTask:+addcomment’s title doesn’t duplicate the bug number (#1323808)
  • The duplicates portlet of bug tasks no longer links to invisible private bugs and now uses the correct sprite for each bug (#1443418, #1465880)
  • Show hidden bug comments to their owners (#1391394)

Code

  • Line numbers in merge proposal preview diffs are now unselectable (#1483925)
  • Branch revision listings link more sensibly to merged branches and merge proposals (#711647)
  • Git repositories and refs have more sensible breadcrumb links (#1466271)
  • Source package recipe builds now include the distribution series in their titles (#1491336)
  • Allow archive owners to cancel their recipe builds, and make recipes use the normal cancellation infrastructure (#624630)
  • Make it possible to retry superseded builds, since there are situations where they can become unsuperseded (#444030)
  • The index page for Git refs is more useful
  • Fix crashes on various +activereviews pages when there are no active reviews (#1499744, #1501134)
  • Precache branch permissions in Branch.landing_candidates, fixing API timeouts (#1500576)
  • Add webhook support for Bazaar branches, currently enabled on qastaging though not yet on production

Mail

  • Team membership notifications now honour the “Include filtering information in email footers” setting, and have rationale information in the headers and footer (#296889, #508897)
  • There is a new X-Launchpad-Message-For mail header (or Launchpad-Message-For in expanded footers) giving just the name of the person or team directly subscribed to the notification (#1493844)
  • Treat “me” in person-or-team contexts in mail handlers as the current user (#340397)

We now consider the “Gmail filtering improvements” work complete.  Let us know if there are further categories of mail sent by Launchpad that you’re finding difficult to filter using Gmail.

Registry

  • Distribution index pages hide links to disabled features (#80315, #257627)
  • Set consistent colour for the “Opinion” bug task status (#648645)
  • Blueprints on milestone pages have icons again and are sortable (#1354387)
  • Old user-to-user email database references no longer prevent changing team privacy (#1498497)

Snap packages

  • Snap packages now have edit, administer, and delete views
  • Branches, Git repositories and refs, people, teams, and products now have snap package listing views
  • Members of ~launchpad-snap-builders can create snap packages in the web UI based on Bazaar or Git branches
  • Snap package owners can request builds in the web UI

Soyuz (package building)

  • Stop showing the confusing .dsc component on source package index pages (#521722)
  • Export archive deletion on the API (#814633)
  • Allow overriding the build version for live filesystem builds (#1496074)
  • Archive owners can change the “build debug symbols” and “publish debug symbols” settings on their own archives, rather than needing to ask an admin
  • The publisher no longer sometimes tries to update Contents files for immutable suites such as the release pocket of stable series, which could cause checksums in Release to get out of sync (#1448270)
  • Publishing PPAs now creates clearsigned InRelease files to improve reliability of updates (#804252); we intend to add this to the primary Ubuntu archive as well once one last piece of mirroring infrastructure is made ready for it
  • Archive owners can enable or disable unrestricted architectures on their own archives, rather than needing to ask an admin; we will unrestrict further architectures once they have adequate virtualised build infrastructure available
  • Enabling a test rebuild archive no longer times out (#1500973)

Read more
Colin Watson

Here’s a summary of what the Launchpad team got up to in August.

Code

  • Webhook support for Git repositories is almost finished, and only needs a bit more web UI work (#1474071)
  • The summary of merge proposal pages now includes a link to the merged revision, if any (#892259)
  • Viewing individual comments on Git-based merge proposals no longer OOPSes (#1485907)

Mail notifications

Our internal stakeholders in Canonical recently asked us to work on improving the ability to filter Launchpad mail using Gmail.  The core of this was the “Include filtering information in email footers” setting that we added recently, but we knew there was some more to do.  Launchpad’s mail notification code includes some of the oldest and least consistent code in our tree, and so improving this has entailed paying off quite a bit of technical debt along the way.

  • Bug notifications and package upload notifications now honour the “Include filtering information in email footers” setting (#1474071)
  • Bug notifications now log an OOPS rather than crashing if the SMTP server rejects an individual message (#314420, #916939)
  • Recipe build notifications now include an X-Launchpad-Archive header (#776160)
  • Question notification rationales are now more consistent, including team annotations for subscribers (#968578)
  • Package upload notifications now include X-Launchpad-Message-Rationale and X-Launchpad-Notification-Type headers, and have more specific footers (#117155, #127917)

Package build infrastructure

  • Launchpad now supports building source packages that use Debian’s new build profiles syntax, currently only with no profiles activated
  • Launchpad can now build snap packages (#1476405), with some limitations; this is currently only available to a group of alpha testers, so let us know if you’re interested
  • Builders can now access Launchpad’s Git hosting (HTTPS only) in the same way that they can access its Bazaar hosting
  • All amd64/i386 builds now take place in ScalingStack, and the corresponding bare-metal builders have been detached pending decommissioning; some of the newer of those machines will be used to further expand ScalingStack capacity
  • We have a new ScalingStack region including POWER8-based ppc64el builders, which is currently undergoing production testing; this will replace the existing POWER7-based builders in a few weeks, and also provide virtualised build capacity for ppc64el PPAs
  • We’ve fixed a race condition that sometimes caused a user’s first PPA to be published unsigned for a while (#374395)

Miscellaneous

  • The project release file upload limit is now 1 GiB rather than 200 MiB (#1479441)
  • We spent some more time supporting translations for the overlay PPA used for current Ubuntu phone images, copying a number of existing translations into place from before the point when they were redirected automatically
  • Your user index page now has a “Change password” link (#1471961)
  • Bug attachments are no longer incorrectly hidden when displaying only some bug comments (#1105543)

Read more
Colin Watson

Here’s a summary of what the Launchpad team got up to in July.

Code

  • We fixed a regression in the wrapping layout of side-by-side diffs on bazaar.launchpad.net (#1436483)
  • Various code pages now have meta tags to redirect “go get" to the appropriate Bazaar or Git URL, allowing the removal of special-casing from the “go" tool (#1465467)
  • Merge proposal diffs including mention of binary patches no longer crash the new-and-improved code review comment mail logic (#1471426), and we fixed some line-counting bugs in that logic as well (#1472045)
  • Links to the Git code browsing interface now use shorter URL forms

We’ve also made a fair amount of progress on adding support for triggering webhooks from Launchpad (#342729), which will initially be hooked up for pushes to Git repositories.  The basic code model, webservice API, and job retry logic are all in place now, but we need to sort out a few more things including web UI and locking down the proxy configuration before we make it available for general use.  We’ll post a dedicated article about this once the feature becomes available.

Mail notifications

We posted recently about improved filtering options (#1474071).  In the process of doing so, we cleaned up several older problems with the mails we send:

  • Notifications for a bug’s initial message no longer include a References header, which confuses some versions of some mail clients (#320034)
  • Package upload notifications no longer attempt to transliterate non-ASCII characters in package maintainer names into ASCII equivalents; they now use RFC2047 encoding instead (#362957)
  • Notifications about duplicate bugs now include an X-Launchpad-Bug-Duplicate header (#363995)
  • Package build failure notifications now include a “You are receiving this email because …” rationale (#410893)

Package build infrastructure

  • The sbuild upgrade last month introduced some regressions in our handling of package builds that need to wait for dependencies (e.g. #1468755), and it’s taken a few goes to get this right; this is somewhat improved now, and the next builder deployment will fix all the currently-known bugs in this area
  • In the same area, we’ve made some progress on adding minimal support for Debian’s new build profiles syntax, applying fixes to upload processing and dependency-wait analysis, although this should still be considered bleeding-edge and unlikely to work from end to end
  • We’ve been working on adding support for building snap packages (#1476405), but there’s still more to do here; we should be able to make this available to some alpha testers around mid-August

Miscellaneous

  • We’ve arranged to redirect translations for the overlay PPA used for current Ubuntu phone images to the ubuntu-rtm/15.04 series so that they can be translated effectively (#1463723); we’re still working on copying translations into place from before this fix
  • Projects and project groups no longer have separately-editable “display name” and “title” fields, which were very similar in purpose; they now just have display names (#1853, #4449)
  • Cancelled live file system builds are sorted to the end of the build history, rather than the start (#1424672)

Read more
Colin Watson

Users of some email clients, particularly Gmail, have long had a problem filtering mail from Launchpad effectively.  We put lots of useful information into our message headers so that heavy users of Launchpad can automatically filter email into different folders.  Unfortunately, Gmail and some other clients do not support filtering mail on arbitrary headers, only on message bodies and on certain pre-defined headers such as Subject.  Figuring out what to do about this has been tricky.  Space in the Subject line is at a premium – many clients will only show a certain number of characters at the start, and so inserting filtering tags at the start would crowd out other useful information, so we don’t want to do that; and in general we want to avoid burdening one group of users with workarounds for the benefit of another group because that doesn’t scale very well, so we had to approach this with some care.

As of our most recent code update, you’ll find a new setting on your “Change your personal details” page:

Screenshot of email configuration options

If you check “Include filtering information in email footers”, Launchpad will duplicate some information from message headers into the signature part (below the dash-dash-space line) of message bodies: any “X-Launchpad-Something: value” header will turn into a “Launchpad-Something: value” line in the footer.  Since it’s below the signature marker, it should be relatively unobtrusive, but is still searchable.  You can search or filter for these in Gmail by putting the key/value pair in double quotes, like this:

Screenshot of Gmail filter dialog with "Has new words" set to "Launchpad-Notification-Type: code-review"

At the moment this only works for emails related to Bazaar branches, Git repositories, merge proposals, and build failures.  We intend to extend this to a few other categories soon, particularly bug mail and package upload notifications.  If you particularly need this feature to work for some other category of email sent by Launchpad, please file a bug to let us know.

Read more
William Grant

Today we’re announcing early support for hosting Git repositories directly on Launchpad, as well as or instead of Bazaar branches. This has been by far the single most commonly requested feature from Launchpad code hosting for a long time; we’ve been working hard on it for several months now, and we’re very happy to be able to release it for general use.

This is distinct from the facility to import code from Git (and some other systems) into Bazaar that Launchpad has included for many years. Code imports are useful to aggregate information from all over the free software ecosystem in a unified way, which has always been one of the primary goals of Launchpad, and in the future we may add the facility to import code into Git as well. However, what we’re releasing today is native support: you can use git push to upload code to Launchpad, and your users and collaborators can use git clone to download it, in the same kind of way that you can with any Git server.

Our support is still in its early stages, and we still have several features to add to bring it up to parity with Bazaar hosting in Launchpad, as well as generally making it easier and more pleasant to use. We’ve released it before it’s completely polished because many people are clamouring to be able to use it and we’re ready to let you all do so. From here on in, we’ll be adding features, applying polish, and fixing bugs using Launchpad’s normal iterative deployment process: changes will be rolled out to production once they’re ready, so you’ll see the UI gradually improving over time.

What’s supported?

You can:

  • push Git repositories to Launchpad over SSH
  • clone repositories over git://, SSH, or HTTPS
  • see summary information on repositories and the branches they contain in the Launchpad web UI
  • follow links from the Launchpad web UI to a full-featured code browser (cgit)
  • push and clone private repositories, if you have a commercial subscription to Launchpad
  • propose merges from one branch to another, including in a different repository, provided that they are against the same project or package

What will be supported later?

Launchpad’s Bazaar support has grown many features over the years, and it will take some time to bring our Git support up to full parity with it and beyond. Git repositories use a somewhat different model from Bazaar branches, which we’ve had to account for in many places, and some facilities will require complete reimplementation before we can support them with Git.

Here’s an incomplete list of some of the features we hope to add:

  • useful subscriptions (currently only attribute change notifications work, which are not usually very interesting in themselves)
  • RSS feeds
  • mirroring
  • webhooks
  • an integrated code browser

See our help page for more known issues and instructions on using Launchpad with Git.

Helping out

This is a new service, and we welcome your feedback: you can ask questions in #launchpad on freenode IRC, on our launchpad-users mailing list, or on Launchpad Answers, and if you find a bug then please tell us about that too.

Launchpad is free software, licensed under the GNU AGPLv3. We’d be very happy to mentor people who want to help out with parts of this service, or to build things on top of it using our published API. Some preliminary documentation on this is on our developer wiki, and you can always contact us for help.

Read more
Curtis Hovey

Launchpad’s bug and branch privacy features were replaced by information sharing that permits project maintainers to share kinds of confidential information with people at the project level. No one needs to manage bug and branch subscriptions to ensure trusted users have access to confidential information.

The Disclosure features

Disclosure is a super feature composed on many features that will allow commercial projects to work in private. Untrusted users cannot see the project’s data. Project maintainers can share their project with trusted users to reveal all or just some of the project’s data. The ultimate goal is to create private project in Launchpad, but that feature required several other features to be completed first. The Purple squad worked on Trusted Pickers, Privacy Transitions, Hardened Projects, Social Private Teams, and Sharing.

There was a lot of overlap between each feature the Purple squad worked on. Though we could start each feature independent of one another, we could only complete about 90% of each. When the Sharing UI changes entered beta, we were unblocked and fixes about most of the remaining issues, but fixing all the issues required all projects to switch to Sharing.   We did not consider Sharing, or any of the required features complete until we fixed all the bugs.

Disclosure facts

  • Planning started in June 2010 to replace the existing privacy mechanisms with something that would scale.
  • Early testing revealed that users did not trust Launchpad because the UI could not explain what was confidential, or what the consequences of a change would be — this needed to be fixed too.
  • 149 related bugs were identified in Launchpad.
  • Work started in June 2011 by the Purple squad.
  • Replacing the old privacy mechanisms and addressing the trust and information issues took 16 months.
  • About 45,000 lines code were added to support the features.
  • About 15% of the lines were for missing JavaScript test coverage.
  • More that 700 bugs were fixed in total.
  • About 5% of the fixed bugs were caused by the old non-scaling privacy mechanisms.
  • About 4% of the fixed bugs were caused by old JavaScript enhancements that broke features for non-JavaScript users.

Lessons learned

  • Misrepresentation of what is confidential, or what will be confidential or public is very important to users — more important than supporting private data.
  • Privacy/Sharing must be a first-class mechanism beneath all the mechanisms that work with confidential data.
    • Privacy was added on top of bugs, and it failed to scale to 100’s of bugs.
    • Privacy was added on top of branches, and it failed to scale to 1000’s of branches.
    • Filtering private items in code, or in database joins is not fast enough to work with 100,000’s of items.
  • Launchpad’s ReSTful object API is not suitable for working with large collections of objects like bugs or branches; a lighter, service-based approach was used to quickly work with large amounts of data.
  • Users need to work with confidential data via the API, using a text web browser from servers, using a browser with accessibility tools, as well as the common case of using a JavaScript enabled browser.
  • Lots of mock-ups and interactive tests will not predict all the interactions a user will have with real data; test with real code and data early to developer the final design.

Read more
Curtis Hovey

Launchpad’s bug and branch privacy features are being replaced by information sharing that permits project maintainers to share kinds of information with people at the project level. No one needs to manage bug and branch subscriptions to ensure trusted users have access to confidential information.

Maintainers can share and unshare their project with people

Project maintainers and drivers can see the “Sharing” link on their project’s front page. The page lists every user and team that the project shares with. During the transition period of the beta, you might see many users with “Some” access to “Private Security” or “Private” user information. They have this access because they are subscribed to bugs and branches. Maintainers can unshare with users who do not need access to any confidential information, or just unshare a bug or branch with a user. Maintainers can share share with a team to give them full access to one or more kinds of confidential information.

I have prepared a video that demonstrates the features (my apologies for the flickering)

Commercial projects can set bug and branch policies

Projects with commercial subscriptions can also change bug and branch sharing policies to set the default information type of a bug or branch, and control what types they may be changed to. Maintainers can set policies that ensure that bugs and branches are proprietary, and only proprietary, to ensure confidential information is never disclosed.

Sharing can be managed using API scripts

I maintain many project which have a lot of private bugs and branches. The sharing page lists a lot of people, too many to read quickly. I know most work for my organisation, but I don’t even know everyone in my organisation. So I wrote a Launchpad API script that can be run by any project maintainer to share the project with a team, then unshare with the team members. The members still have access to the bugs and branches and their subscriptions still work, but they will lose access to my project when they leave the team. This arrangement makes it very easy to manage who has access to my projects. share-projects-with-team.py is run with the name of the team and a list of projects to share with it.

./share-projects-with-team.py my-team project1 project2

Read more
Curtis Hovey

Project maintainers can now see all the private bugs in their project. While Launchpad tried to ensure the proper people could see private bugs in the past, the old subscription mechanism was brittle. Users could unsubscribe themselves and lose access, or retarget a bug to another projects which does not update bug subscriptions. The Purple squad migrated project configurations to project sharing so that all private information was shared with project maintainers. Project sharing ensures that confidential information is disclosed to the proper people.

If you are a project maintainer, you might be surprised to find old private bugs that you have never seen before. This happened to me. Some ancient private bugs were in the “New” listing of bugs, other were buried in search results. You can search for just private bugs to review all private bugs.

advanced search for private information types

Privacy terminology is restored

We reverted the information type terminology changes introduced a few months ago.

  • User data ? Private
  • Embargoed Security ? Private Security
  • Unembargoed Security ? Public Security

While the jargon-laden terms helped the small number of people who work with confidential information, the people who report bugs were confused. The most common reason for unwanted disclosure is that people enter confidential information, and cannot see how to make it private. Sometimes a user may not notice the mistake until a few minutes later. We also revised the descriptions of the information types to help new users quickly select the correct information type.

change information type

Read more
Curtis Hovey

You can now hide your own bug and question comments. If you want to hide a comment made in error, you can use the “Hide comment” action.

hide your comment

You can see it, and even unhide it if you choose. The project’s maintainer or the trusted people delegated to work with private information can still see your comment.

your hidden comment

This allows you, or the people the project shares private information with, to hide just the comments that contain personal information. The bug does not need to be made private if the comment can be hidden. Project maintainers can also hide comments because they contain spam or abuse.

 

Read more
Curtis Hovey

The Purple squad recently updated bug reporting and searching to understand the new privacy rules. Some of the changes were requirements to support sharing, others were opportunities we took advantage of.

Improvements to bug reporting and forms

The Purple squad updated the bug reporting UI to make it consistent with the bug pages. We choose to develop one consistent and tested UI rather than update the many kinds of widgets used in bug forms.

  • Project maintainers, drivers, and bug supervisors can report private bugs.
  • Autocomplete works with bug tags
  • The status and importance controls show their definitions.
  • Undecided is the first importance because it is the default importance.

Improvements to bug searching

Advanced bug search was updated after we discovered that recent changes made it possible fix some long standing issues with a few additional lines of code.

  • Anyone can search for Private or Embargoed Security bugs that are shared with them.
  • Autocomplete works with bug tags.

Usability and Accessibility fixes

We discovered that the popups that show bug status, importance and information type did not work with keyboards. It was possible to tab out of every other kind of popup by accident. We made deep fixes to the code so that all launchpad popups work with keyboard.

  • You can use the tab key to move between the items in popups.
  • You cannot accidentally tab out of any popup.

Read more
Curtis Hovey

Bug and branch information types

All users can now see the information type section that replaces the privacy and security section shown on bug and branch pages. This change allows users to clearly state the kind of information a bug or branch contains. Launchpad will soon permit project maintainers to share information types instead of managing individual bug and branch subscriptions. Project maintainers can see a link on their project’s front page to the  ”Sharing” page. Sharing lists all the users and teams their project shares some private bugs and branches with. This list might be surprising. Launchpad Beta testers will soon be able share and unshare kinds of information to simplify management of whom the project discloses private information to.

See Reimaging the nature of privacy in Launchpad for more details.

Read more
Curtis Hovey

Launchpad beta testers are seeing information types on bug reports. Launchpad replaced the private and security checkboxes with an information type chooser. The information types determine who may know about the bug.

When you report a bug, you can choose the information type that describes the bug’s content. The person who triages the bug may change the information type. Information types may also change as a part of a workflow, for example, a bug may start as Embargoed Security while the bug is being fixed, then the release manager can change the information type to Unembargoed Security after the release.

Testing in phases

In the first phase of this beta, Launchpad continues to share private information with bug supervisors and security contacts using bug subscriptions. The project maintainer may be managing hundreds of bug subscriptions to private bugs, and people are getting unwanted bug mail.

In the second phase of the beta, the project maintainer can share information types with people…the maintainer is only managing shares with a few teams and users and people are not getting unwanted bug mail.

Watch the video of information types and sharing to see the feature in use and hints of the future.

Read more
Curtis Hovey

Launchpad beta testers will now see the descriptions of bug status and importance when making updates to the bug page. Launchpad pickers can now show the descriptions of the options you can choose.

Launchpad’s rules for defining a list of options you can choose have always required descriptions, but the only places you could see them were in some forms where they were listed as radio buttons. Bug status and importance where never shown as radio buttons, so their description were only know to people who read Launchpad’s source code. Users need to see the descriptions so that there is a common understanding of terms that allows us to collaborate.  The original bug importance descriptions were written in 2006 and only made sense for Ubuntu bugs. We revised the descriptions for the improved picker.

There has been a lot of confusion and disagreement about the meaning of bug statuses. Since users could not see the descriptions, we posted the definition on help.launchpad.net. Separating the status description from the status title did not end the confusion. We revised the descriptions for the improved picker, but I think we need to make more changes before showing this to everyone. The picker  appears to rely on colour to separate the choice title from description. Not all choices will have a special colour, and in the case of bug status there are two choices that appear to be the same grey as the description text:

The picker enhancements were made for the disclosure feature. We are changing the presentation of bug and branch privacy to work with the forthcoming project sharing enhancements. Early testing revealed that users need to know who will be permitted to see the private information when the bug is changed. This issue was similar to the long standing problem with bug status and importance. We decided to create a new picker that solved the old problem, that we could then reuse to solve the new problem.

Read more
Curtis Hovey

Setting up a commercial project in Launchpad has gotten easier. You can now quickly register a proprietary project and enable private bugs. You can create private teams and private personal package archives, AKA private PPA or P3A without the assistance of a Launchpad admin.

When you select the Other/Proprietary license while registering a project, or changing the project’s details,

it is given a complimentary 30-day commercial subscription.

The delay between the moment when a commercial project was registered and when the commercial subscription was purchased and then applied to the project caused a lot of confusion. During this delay, proprietary data could be disclosed. We chose to award the project with a short term commercial subscription which enabled the project to be properly configured while the 12-month commercial subscription was being purchased and applied to the project.

Any project with a commercial subscription can enable

Default private bugs
Once enabled by configuring the project’s bug tracker, all new reported bugs are private. You can choose to make the report public.
Default private bugs
Default private branches
You can request a Launchpad admin to configure private branches for your teams. (You will be able to do this yourself in the near future when projects gain proprietary branches.)

As the maintainer of a project with a commercial subscription, you can register

Private teams
When you register a team, you can choose to set the team visibility to private. The team’s members and data is hidden from non-members.
Private mailing lists
When you create a mailing list for a private team, the archive is also private. Only team members may see the messages in the archive.
Private PPAs
When you create a PPA for your public team, you may choose to make it private; private teams can only have private PPAs. You can subscribe users to your archive so that they may install packages without revealing all your team’s members and data to the subscriber.

A secondary benefit of this change is that you can now try Launchpad’s commercial features before purchasing a 12-month commercial subscription. The features will be disabled at the end of 30-days. Your test data will remain private to ensure your data is not disclosed.

Any open source project may also have a commercial subscription to enable commercial features. You can purchase a commercial subscription at the Canonical store. Commercial subscriptions cost US$250/year/project + applicable V.A.T.

 

(Photo by Fred Dawson on flickr, creative commons license)

Read more