Canonical Voices

Colin Watson

Here’s a brief changelog for this month.

Build farm

  • Allow dispatching builds with base images selected based on the pocket and/or using LXD images instead of chroot tarballs where appropriate (#1811677)

Code

  • Store bzr-svn‘s cache in the import data store
  • Allow project owners to use the Bazaar branch rescan view
  • Canonicalise expected rule ordering in GitRepository.setRules (#1815431)
  • Upgrade to pygit2 0.27.4 (#1815517)

Infrastructure

  • Use full gpg key fingerprints in rocketfuel-setup (contributed by Andy Brody; #1814206)

Snappy

Soyuz (package management)

  • Allow source .changes files to omit the Binary field (#1813037)

Read more
Colin Watson

Here’s a brief changelog of what we’ve been up to since our last general update.

Bugs

  • Parse a few more possible Savane URL formats (#197250)
  • Compare Bugzilla versions properly when checking whether they support the Bugzilla API (part of #1802798)

Build farm

  • Configure snap proxy settings for Subversion (#1668358)
  • Support passing IMAGE_TARGETS, REPO_SNAPSHOT_STAMP, and COHORT_KEY variables into live filesystem builds
  • Set SNAPCRAFT_BUILD_ENVIRONMENT=host when building snaps (#1791201)
  • Prevent gathering results of large builds from blocking responses to XML-RPC requests (#1795877)
  • Add missing indexes on LiveFSFile(libraryfile) and SnapFile(libraryfile)
  • Direct build failure support to Launchpad Answers rather than to the launchpad-buildd-admins team (#1810001)

Code

  • Allow proposing merges between different branches of the same personal Git repository
  • Fix OOPS when trying to look up ~user/project:branch as a unique Git repository name (#1771118)
  • Optimise GitRepository.fetchRefCommits if there are no commits to fetch
  • Handle the case where a Bazaar branch and a Git repository have the same identity URL when creating a recipe (#1623924)
  • Push code imports over bzr+ssh rather than sftp (#1779572)
  • Fix crash when emailing inline comments on a diff with non-ASCII characters in hunk headers (#1787587)
  • Percent-encode reference names in GitRef URLs (#1787965)
  • Add active reviews link to Git-based project pages (#1777102)
  • Fix handling of non-ASCII ref names
  • Include the appropriate username in git+ssh:// URLs in the UI
  • Add instructions on creating personal Git repositories to people’s “View Git repositories” pages (#1590560)
  • Add available review targets and proposals to the Git repository overview page (#1789847)
  • Fix incorrect visibility check that broke code imports targeted at private Git repositories (#1789424)
  • Allow anonymous users to view votes for public merge proposals (#1786474)
  • Make Git ref scan jobs for repositories with large numbers of refs take much less memory
  • Tolerate backend timeouts while fetching commit information for GitRef:+index (#1804395)
  • Add Git per-branch permissions (#1517559)
  • Add rescan buttons when various kinds of code scanning jobs fail (#1808320)

Infrastructure

  • Convert all remaining code to use explicit proxy configuration settings rather than picking up a proxy from the environment, making the effective production settings easier to understand
  • Add support for ECDSA SSH keys (#907675)

Libraries

Registry

  • Add a suspend-bot-account.py script to suspend an account by email address
  • Weaken type of key_text in Person.deleteSSHKeysFromSSO so that more existing keys can be deleted (#1780411)
  • Fix SSHKey.getFullKeyText to not crash on some corrupt keys (#1798046)
  • Various improvements to the close-account script

Snappy

  • Extract initial Snap.store_name from snapcraft.yaml for Bazaar as well as Git
  • Add support for Snapcraft’s architectures keyword (#1770400)
  • Bump SnapStoreUploadJob.max_retries to 30 to allow for longer store scan times
  • Include the registered store package name for a snap recipe in its builds’ titles if it exists and differs from the snap recipe name
  • Move some metadata from SnapStoreUploadJob to SnapBuild, to prevent store upload jobs getting into states that cannot be retried

Soyuz (package management)

  • Add Archive.getSigningKeyData, currently just proxying through to the keyserver (#1667725)
  • Add extendedKeyUsage information to kmod signing keys so that they can only be used to sign modules, not boot loaders or kernels (#1774746)

Read more
Colin Watson

Git per-branch permissions

We’ve had Git hosting support in Launchpad for a few years now. One thing that some users asked for, particularly larger users such as the Ubuntu kernel team, was the ability to set up per-branch push permissions for their repositories. Today we rolled out the last piece of this work.

Launchpad’s default behaviour is that repository owners may push anything to their own repositories, including creating new branches, force-pushing (rewriting history), and deleting branches, while nobody else may push anything. Repository owners can now also choose to protect branches or tags, either individually or using wildcard rules. If a branch is protected, then by default repository owners can only create or push it but cannot force-push or delete; if a tag is protected, then by default repository owners can create it but cannot move or delete it.

You can also allow selected contributors to push to protected branches or tags, so if you’re collaborating with somebody on a branch and just want to be able to quickly pair-program via git push, or you want a merge robot to be able to land merge proposals in your repository without having to add it to the team that owns the repository and thus give it privileges it doesn’t need, then this feature may be for you.

There’s some initial documentation on our help site, and here’s a screenshot of a repository that’s been set up to give a contributor push access to a single branch:

Read more
Colin Watson

Here’s a brief changelog for this month.

Bugs

  • Handle Bugzilla.time() changes in Bugzilla 5.1.1 (#1774838)
  • Cope with the comment author field being renamed to creator in recent Bugzilla versions (#1774838)

Build farm

  • Set the hostname and FQDN of LXD containers to match the host system, though with an IP address pointing to the container (#1747015)
  • If the extra build arguments include fast_cleanup: True, then skip the final cleanup steps of the build; this can be used when building in a VM that is guaranteed to be torn down after the build
  • Allow checking out a git tag rather than a branch (#1687078, forum post)
  • Add a local unauthenticated proxy on port 8222, which proxies through to the remote authenticated proxy; this should allow running a wider range of network clients, since some of them apparently don’t support authenticated proxies very well (#1690834, #1753340, forum post)
  • Run tar with correct working directory when building source tarballs for snaps

Code

  • Port the loggerhead (Bazaar code browser) integration to gunicorn, allowing it to be used as an internal API as well
  • Optimise BuildableDistroSeries.findSeries (#1778732)
  • Proxy loggerhead branch diffs through the webapp, allowing AJAX MP revision diffs to work for private branches (#904070)

Infrastructure

  • Convert most code to use explicit proxy configuration settings rather than picking up a proxy from the environment, making the effective production settings easier to understand

Registry

  • Fix crash while adding an ssh key with unknown type (#1777507)

Miscellaneous

  • Improve documentation of what deactivating an account does (#993153)

Read more
Colin Watson

Here’s a brief changelog for this month.

Build farm

  • Send fast_cleanup: True to virtualised builds, since they can safely skip the final cleanup steps

Code

  • Add spam controls to code review comments (#1680746)
  • Only consider the most recent nine successful builds when estimating recipe build durations (#1770121)
  • Make updated code import emails more informative

Infrastructure

  • Upgrade to Twisted 17.9.0
  • Get the test suite passing on Ubuntu 18.04 LTS
  • Allow admins to configure users such that unsigned email from them will be rejected, as a spam defence (#1714967)

Snappy

  • Prune old snap files that have been uploaded to the store; this cleaned up about 5TB of librarian space
  • Make the snap store client cope with a few more edge cases (#1766911)
  • Allow branches in snap store channel names (#1754405)

Soyuz (package management)

  • Add DistroArchSeries.setChrootFromBuild, allowing setting a chroot from a file produced by a live filesystem build
  • Disambiguate URLs to source package files in the face of filename clashes in imported archives
  • Optimise SourcePackagePublishingHistory:+listing-archive-extra (#1769979)

Miscellaneous

  • Disable purchasing of new commercial subscriptions; existing customers have been contacted, and people with questions about this can contact Canonical support
  • Various minor revisions to the terms of service from Canonical’s legal department, and a clearer data privacy policy

Read more
Colin Watson

Once again it’s been a while since we posted a general update, so here’s a changelog-style summary of what we’ve been up to.  As usual, this changelog preserves a reasonable amount of technical detail, but I’ve omitted changes that were purely internal refactoring with no externally-visible effects.

Answers

  • Hide questions on inactive projects from the results of non-pillar-specific searches

Blueprints

  • Optimise the main query on Person:+upcomingwork (#1692120)
  • Apply the spec privacy check on Person:+upcomingwork only to relevant specs (#1696519)
  • Move base clauses for specification searches into a CTE to avoid slow sequential scans

Bugs

  • Switch to HTTPS for CVE references
  • Fix various failures to sync from Red Hat’s Bugzilla instance (#1678486)

Build farm

  • Send the necessary set of archive signing keys to builders (#1626739)
  • Hide the virt/nonvirt queue portlets on BuilderSet:+index if they’d be empty
  • Add a feature flag which can be used to prevent dispatching any build under a given minimum score
  • Write files fetched from builders to a temporary name, and only rename them into place on success
  • Emit the build URL at the start of build logs

Code

  • Fix crash when scanning a Git-based MP when we need to link a new RevisionAuthor to an existing Person (#1693543)
  • Add source ref name to breadcrumbs for Git-based MPs; this gets the ref name into the page title, which makes it easier to find Git-based MPs in browser history
  • Allow registry experts to delete recipes
  • Explicitly mark the local apt archive for recipe builds as trusted (#1701826)
  • Set +code as the default view on the code layer for (Person)DistributionSourcePackage
  • Improve handling of branches with various kinds of partial data
  • Add and export BranchMergeProposal.scheduleDiffUpdates (#483945)
  • Move “Updating repository…” notice above the list of branches so that it’s harder to miss (#1745161)
  • Upgrade to Pygments 2.2.0, including better formatting of *.md files (#1740903)
  • Sort cancelled-before-starting recipe builds to the end of the build history (#746140)
  • Clean up the {Branch,GitRef}:+register-merge UI slightly
  • Optimise merge detection when the branch has no landing candidates

Infrastructure

  • Use correct method separator in Allow headers (#1717682)
  • Optimise lp_sitecustomize so that bin/py starts up more quickly
  • Add a utility to make it easier to run Launchpad code inside lxc exec
  • Convert lp-source-dependencies to git
  • Remove the post-webservice-GET commit
  • Convert build system to virtualenv and pip, unblocking many upgrades of dependencies
  • Use eslint to lint JavaScript files
  • Tidy up various minor problems in the top-level Makefile (#483782)
  • Offering ECDSA or Ed25519 SSH keys to Launchpad SSH servers no longer causes a hang, although it still isn’t possible to use them for authentication (#830679)
  • Reject SSH public keys that Twisted can’t load (#230144)
  • Backport GPGME file descriptor handling improvements to fix timeouts importing GPG keys (#1753019)
  • Improve OOPSes for jobs
  • Switch the site-wide search to Bing Custom Search, since Google Site Search has been discontinued
  • Don’t send email to direct recipients without active accounts

Registry

  • Fix the privacy banner on PersonProduct pages
  • Show GPG fingerprints rather than collidable short key IDs (#1576142)
  • Fix PersonSet.getPrecachedPersonsFromIDs to handle teams with mailing lists
  • Optimise front page, mainly by gathering more statistics periodically rather than on the fly
  • Construct public keyserver links using HTTPS without an explicit port (#1739110)
  • Fall back to emailing the team owner if the team has no admins (#1270141)

Snappy

  • Log some useful information from authorising macaroons while uploading snaps to the store, to make it easier to diagnose problems
  • Extract more useful error messages when snap store operations fail (#1650461, #1687068)
  • Send mail rather than OOPSing if refreshing snap store upload macaroons fails (#1668368)
  • Automatically retry snap store upload attempts that return 502 or 503
  • Initialise git submodules in snap builds (#1694413)
  • Make SnapStoreUploadJob retries go via celery and be much more responsive (#1689282)
  • Run snap builds in LXD containers, allowing them to install snaps as build-dependencies
  • Allow setting Snap.git_path directly on the webservice
  • Batch snap listing views (#1722562)
  • Fix AJAX update of snap builds table to handle all build statuses
  • Set SNAPCRAFT_BUILD_INFO=1 to tell snapcraft to generate a manifest
  • Only emit snap:build:0.1 webhooks from SnapBuild.updateStatus if the status has changed
  • Expose extended error messages (with external link) for snap build jobs (#1729580)
  • Begin work on allowing snap builds to install snapcraft as a snap; this can currently be set up via the API, and work is in progress to add UI and to migrate to this as the default (#1737994)
  • Add an admin option to disable external network access for snap builds
  • Export ISnapSet.findByOwner on the webservice
  • Prefer Snap.store_name over Snap.name for the “name” argument dispatched to snap builds
  • Pass build URL to snapcraft using SNAPCRAFT_IMAGE_INFO
  • Add an option to build source tarballs for snaps (#1763639)

Soyuz (package management)

  • Stop SourcePackagePublishingHistory.getPublishedBinaries materialising rows outside the current batch; this fixes webservice timeouts for sources with large numbers of binaries (#1695113)
  • Implement proxying of PackageUpload binary files via the webapp, since DistroSeries:+queue now assumes that that works (#1697680)
  • Truncate signing key common-names to 64 characters (#1608615)
  • Allow setting a relative build score on live filesystems (#1452543)
  • Add signing support for vmlinux for use on ppc64el Opal (and compatible) firmware
  • Run live filesystem builds in LXD containers, allowing them to install snaps as build-dependencies
  • Accept a “debug” entry in live filesystem build metadata, which enables detailed live-build debugging
  • Accept and ignore options (e.g. [trusted=yes]) in sources.list lines passed via external_dependencies
  • Send proper email notifications about most failures to parse the .changes file (#499438)
  • Ensure that PPA .htpasswd salts are drawn from the correct alphabet (#1722209)
  • Improve DpkgArchitectureCache‘s timeline handling, and speed it up a bit in some cases (#1062638)
  • Support passing a snap channel into a live filesystem build through the environment
  • Add support for passing apt proxies to live-build
  • Allow anonymous launchpad.View on IDistributionSourcePackage
  • Handle queries starting with “ppa:” when searching the PPA vocabulary
  • Make PackageTranslationsUploadJob download librarian files to disk rather than memory
  • Send email notifications when an upload is signed with an expired key
  • Add Release, Release.gpg, and InRelease to by-hash directories
  • After publishing a custom file, mark its target suite as dirty so that it will be published (#1509026)

Translations

  • Fix text_to_html to not parse HTML as a C format string
  • Fall back to the package name from AC_INIT when expanding $(PACKAGE) in translation configuration files if no other definition can be found

Miscellaneous

  • Show a search icon for pickers where possible rather than “Choose…”

Read more
Colin Watson

Summary

Mohamed Alaa reported that Launchpad’s Bing site search implementation had a cross-site-scripting vulnerability.  This was introduced on 2018-03-29, and fixed on 2018-04-10.  We have not found any evidence of this bug being actively exploited by attackers; the rest of this post is an explanation of the problem for the sake of transparency.

Details

Some time ago, Google announced that they would be discontinuing their Google Site Search product on 2018-04-01.  Since this served as part of the backend for Launchpad’s site search feature (“Search Launchpad” on the front page), we began to look around for a replacement.  We eventually settled on Bing Custom Search, implemented appropriate support in Launchpad, and switched over to it on 2018-03-29.

Unfortunately, we missed one detail.  Google Site Search’s XML API returns excerpts of search results as pre-escaped HTML, using <b> tags to indicate where search terms match.  This makes complete sense given its embedding in XML; it’s hard to see how that API could do otherwise.  The Launchpad integration code accordingly uses TAL code along these lines, using the structure keyword to explicitly indicate that the excerpts in question do not require HTML-escaping (like most good web frameworks, TAL’s default is to escape all variable content, so successful XSS attacks on Launchpad have historically been rare):

<div class="summary" tal:content="structure page/summary" />

However, Bing Custom Search’s JSON API returns excerpts of search results without any HTML escaping.  Again, in the context of the API in question, this makes complete sense as a default behaviour (though a textFormat=HTML switch is available to change this); but, in the absence of appropriate handling, this meant that those excerpts were passed through to the TAL code above without escaping.  As a result, if you could craft search terms that match a portion of an existing page on Launchpad that shows scripting tags (such as a bug about an XSS vulnerability in another piece of software hosted on Launchpad), and convince other people to follow a suitable search link, then you could cause that code to be executed in other users’ browsers.

The fix was, of course, to simply escape the data returned by Bing Custom Search.  Thanks to Mohamed Alaa for their disclosure.

Read more
Colin Watson

Here’s a brief changelog for this month.

Bugs

  • Export searchTasks for the top-level bugs collection on the webservice, and implement a global bugs feed to go with it (#434244)

Code

  • Fix git-to-git code imports on xenial
  • Backport upstream serf commit to fix svn-to-bzr code imports on xenial (#1690613)
  • Fix crash when unlinking a bug from a Git-based MP in UpdatePreviewDiffJob
  • Handle revision ID passed to BranchMergeProposal.setStatus for transitions to MERGED

Infrastructure

  • Fix processing of purchased Launchpad commercial subscriptions
  • Some progress towards converting the build system to pip, though there’s quite a bit more work to do there

Registry

  • Remove extra internal slashes from URLs requested by the mirror prober (#1692347)

Snappy

  • Record the branch revision used to build a snap and return it along with other XML-RPC status information (#1679157)
  • Configure a git:// proxy for snap builds (#1663920)
  • Allow configuring a snap to build from the current branch of a Git repository rather than explicitly naming a branch (#1688224)

Soyuz (package management)

  • Precache permissions for archives returned by Person.getVisiblePPAs (#1685202)
  • Drop requirement for source .buildinfo files to be signed
  • Make DistroSeries:+queue link to upload files via the webapp, to help dget users

Read more
Colin Watson

Well, it’s been a while!  Since we last posted a general update, the Launchpad team has become part of Canonical’s Online Services department, so some of our efforts have gone into other projects.  There’s still plenty happening with Launchpad, though, and here’s a changelog-style summary of what we’ve been up to.

Answers

  • Lock down question title and description edits from random users
  • Prevent answer contacts from editing question titles and descriptions
  • Prevent answer contacts from editing FAQs

Blueprints

  • Optimise SpecificationSet.getStatusCountsForProductSeries, fixing Product:+series timeouts
  • Add sprint deletion support (#2888)
  • Restrict blueprint count on front page to public blueprints

Build farm

  • Add fallback if nominated architecture-independent architecture is unavailable for building (#1530217)
  • Try to load the nbd module when starting launchpad-buildd (#1531171)
  • Default LANG/LC_ALL to C.UTF-8 during binary package builds (#1552791)
  • Convert buildd-manager to use a connection pool rather than trying to download everything at once (#1584744)
  • Always decode build logtail as UTF-8 rather than guessing (#1585324)
  • Move non-virtualised builders to the bottom of /builders; Ubuntu is now mostly built on virtualised builders
  • Pass DEB_BUILD_OPTIONS=noautodbgsym during binary package builds if we have not been told to build debug symbols (#1623256)

Bugs

  • Use standard milestone ordering for bug task milestone choices (#1512213)
  • Make bug activity records visible to anonymous API requests where appropriate (#991079)
  • Use a monospace font for “Add comment” boxes for bugs, to match how the comments will be displayed (#1366932)
  • Fix BugTaskSet.createManyTasks to map Incomplete to its storage values (#1576857)
  • Add basic GitHub bug linking (#848666)
  • Prevent rendering of private team names in bugs feed (#1592186)
  • Update CVE database XML namespace to match current file on cve.mitre.org
  • Fix Bugzilla bug watches to support new versions that permit multiple aliases
  • Sort bug tasks related to distribution series by series version rather than series name (#1681899)

Code

  • Remove always-empty portlet from Person:+branches (#1511559)
  • Fix OOPS when editing a Git repository with over a thousand refs (#1511838)
  • Add Git links to DistributionSourcePackage:+branches and DistributionSourcePackage:+all-branches (#1511573)
  • Handle prerequisites in Git-based merge proposals (#1489839)
  • Fix OOPS when trying to register a Git merge with a target path but no target repository
  • Show an “Updating repository…” indication when there are pending writes
  • Launchpad’s Git hosting backend is now self-hosted
  • Fix setDefaultRepository(ForOwner) to cope with replacing an existing default (#1524316)
  • Add “Configure Code” link to Product:+git
  • Fix Git diff generation crash on non-ASCII conflicts (#1531051)
  • Fix stray link to +editsshkeys on Product:+configure-code when SSH keys were already registered (#1534159)
  • Add support for Git recipes (#1453022)
  • Fix OOPS when adding a comment to a Git-based merge proposal without using AJAX (#1536363)
  • Fix shallow git clones over HTTPS (#1547141)
  • Add new “Code” portlet on Product:+index to make it easier to find source code (#531323)
  • Add missing table around widget row on Product:+configure-code, so that errors are highlighted properly (#1552878)
  • Sort GitRepositorySet.getRepositories API results to make batching reliable (#1578205)
  • Show recent commits on GitRef:+index
  • Show associated merge proposals in Git commit listings
  • Show unmerged and conversation-relevant Git commits in merge proposal views (#1550118)
  • Implement AJAX revision diffs for Git
  • Fix scanning branches with ghost revisions in their ancestry (#1587948)
  • Fix decoding of Git diffs involving non-UTF-8 text that decodes to unpaired surrogates when treated as UTF-8 (#1589411)
  • Fix linkification of references to Git repositories (#1467975)
  • Fix +edit-status for Git merge proposals (#1538355)
  • Include username in git+ssh URLs (#1600055)
  • Allow linking bugs to Git-based merge proposals (#1492926)
  • Make Person.getMergeProposals have a constant query count on the webservice (#1619772)
  • Link to the default git repository on Product:+index (#1576494)
  • Add Git-to-Git code imports (#1469459)
  • Improve preloading of {Branch,GitRepository}.{landing_candidates,landing_targets}, fixing various timeouts
  • Export GitRepository.getRefByPath (#1654537)
  • Add GitRepository.rescan method, useful in cases when a scan crashed

Infrastructure

  • Launchpad’s SSH endpoints (bazaar.launchpad.net, git.launchpad.net, upload.ubuntu.com, and ppa.launchpad.net) now support newer key exchange and MAC algorithms, allowing compatibility with OpenSSH >= 7.0 (#1445619)
  • Make cross-referencing code more efficient for large numbers of IDs (#1520281)
  • Canonicalise path encoding before checking a librarian TimeLimitedToken (#677270)
  • Fix Librarian to generate non-cachable 500s on missing storage files (#1529428)
  • Document the standard DELETE method in the apidoc (#753334)
  • Add a PLACEHOLDER account type for use by SSO-only accounts
  • Add support to +login for acquiring discharge macaroons from SSO via an OpenID exchange (#1572605)
  • Allow managing SSH keys in SSO
  • Re-raise unexpected HTTP errors when talking to the GPG key server
  • Ensure that the production dump is usable before destroying staging
  • Log SQL statements as Unicode to avoid confusing page rendering when the visible_render_time flag is on (#1617336)
  • Fix the librarian to fsync new files and their parent directories
  • Handle running Launchpad from a Git working tree
  • Handle running Launchpad on Ubuntu 16.04 (upgrade currently in progress)
  • Fix delete_unwanted_swift_files to not crash on segments (#1642411)
  • Update database schema for PostgreSQL 9.5 and 9.6
  • Check fingerprints of keys received from the keyserver rather than trusting it implicitly

Registry

  • Make public SSH key records visible to anonymous API requests (#1014996)
  • Don’t show unpublished packages or package names from private PPAs in search results from the package picker (#42298, #1574807)
  • Make Person.time_zone always be non-None, allowing us to easily show the edit widget even for users who have never set their time zone (#1568806)
  • Let latest questions, specifications and products be efficiently calculated
  • Let project drivers edit series and productreleases, as series drivers can; project drivers should have series driver power over all series
  • Fix misleading messages when joining a delegated team
  • Allow team privacy changes when referenced by CodeReviewVote.reviewer or BugNotificationRecipient.person
  • Don’t limit Person:+related-projects to a single batch

Snappy

  • Add webhook support for snaps (#1535826)
  • Allow deleting snaps even if they have builds
  • Provide snap builds with a proxy so that they can access external network resources
  • Add support for automatically uploading snap builds to the store (#1572605)
  • Update latest snap builds table via AJAX
  • Add option to trigger snap builds when top-level branch changes (#1593359)
  • Add processor selection in new snap form
  • Add option to automatically release snap builds to store channels after upload (#1597819)
  • Allow manually uploading a completed snap build to the store
  • Upload *.manifest files from builders as well as *.snap (#1608432)
  • Send an email notification for general snap store upload failures (#1632299)
  • Allow building snaps from an external Git repository
  • Move upload to FAILED if its build was deleted (e.g. because of a deleted snap) (#1655334)
  • Consider snap/snapcraft.yaml and .snapcraft.yaml as well as snapcraft.yaml for new snaps (#1659085)
  • Add support for building snaps with classic confinement (#1650946)
  • Fix builds_for_snap to avoid iterating over an unsliced DecoratedResultSet (#1671134)
  • Add channel track support when uploading snap builds to the store (contributed by Matias Bordese; #1677644)

Soyuz (package management)

  • Remove some more uses of the confusing .dsc component; add the publishing component to SourcePackage:+index in compensation
  • Add include_meta option to SPPH.sourceFileUrls, paralleling BPPH.binaryFileUrls
  • Kill debdiff after ten minutes or 1GiB of output by default, and make sure we clean up after it properly (#314436)
  • Fix handling of << and >> dep-waits
  • Allow PPA admins to set external_dependencies on individual binary package builds (#671190)
  • Fix NascentUpload.do_reject to not send an erroneous Accepted email (#1530220)
  • Include DEP-11 metadata in Release file if it is present
  • Consistently generate Release entries for uncompressed versions of files, even if they don’t exist on the filesystem; don’t create uncompressed Packages/Sources files on the filesystem
  • Handle Build-Depends-Arch and Build-Conflicts-Arch from SPR.user_defined_fields in Sources generation and SP:+index (#1489044)
  • Make index compression types configurable per-series, and add xz support (#1517510)
  • Use SHA-512 digests for GPG signing where possible (#1556666)
  • Re-sign PPAs with SHA-512
  • Publish by-hash index files (#1430011)
  • Show SHA-256 checksums rather than MD5 on DistributionSourcePackageRelease:+files (#1562632)
  • Add a per-series switch allowing packages in supported components to build-depend on packages in unsupported components, used for Ubuntu 16.04 and later
  • Expand archive signing to kernel modules (contributed by Andy Whitcroft; #1577736)
  • Uniquely index PackageDiff(from_source, to_source) (part of #1475358)
  • Handle original tarball signatures in source packages (#1587667)
  • Add signed checksums for published UEFI/kmod files (contributed by Andy Whitcroft; #1285919)
  • Add support for named authentication tokens for private PPAs
  • Show explicit add-apt-repository command on Archive:+index (#1547343)
  • Use a per-archive OOPS timeline in archivepublisher scripts
  • Link to package versions on DSP:+index using fmt:url rather than just a relative link to the version, to avoid problems with epochs (#1629058)
  • Fix RepositoryIndexFile to gzip without timestamps
  • Fix Archive.getPublishedBinaries API call to have a constant query count (#1635126)
  • Include the package name in package copy job OOPS reports and emails (#1618133)
  • Remove headers from Contents files (#1638219)
  • Notify the Changed-By address for PPA uploads if the .changes contains “Launchpad-Notify-Changed-By: yes” (#1633608)
  • Accept .debs containing control.tar.xz (#1640280)
  • Add Archive.markSuiteDirty API call to allow requesting that a given archive/suite be published
  • Don’t allow cron-control to interrupt publish-ftpmaster part-way through (#1647478)
  • Optimise non-SQL time in PublishingSet.requestDeletion (#1682096)
  • Store uploaded .buildinfo files (#1657704)

Translations

  • Allow TranslationImportQueue to import entries from file objects rather than having to read arbitrarily-large files into memory (#674575)

Miscellaneous

  • Use gender-neutral pronouns where appropriate
  • Self-host the Ubuntu webfonts (#1521472)
  • Make the beta and privacy banners float over the rest of the page when scrolling
  • Upgrade to pytz 2016.4 (#1589111)
  • Publish Launchpad’s code revision in an X-Launchpad-Revision header
  • Truncate large picker search results rather than refusing to display anything (#893796)
  • Sync up the lists footer with the main webapp footer a bit (#1679093)

Read more
Colin Watson

Launchpad has had Git-to-Bazaar code imports since 2009, along with imports from a few other systems.  These form part of Launchpad’s original mission to keep track of free software, regardless of where it’s hosted.  They’re also very useful for automatically building other artifacts, such as source package recipes or snap packages, from code hosted elsewhere.  Unfortunately they’re quite complicated: they need to be able to do a full round-trip conversion of every revision from the other version control system, which has made it difficult to add support for Git features such as signed commits or submodules.  Once one of these features is present anywhere in the history of a branch, importing it to Bazaar becomes impossible.  This has been a headache for many users.

We can do better nowadays.  As of last year, we have direct Git hosting support in Launchpad, and we can already build snaps and recipes straight from Git, so we can fulfil our basic goal more robustly now with a lot less code.  So, Launchpad now supports Git-to-Git code imports, also known as Git mirroring.  You can use this to replace many uses of Git-to-Bazaar imports (note that there’s no translations integration yet, and of course you won’t be able to branch the resulting import using bzr).

See our Git documentation for more details.

Read more
Colin Watson

Beta test: new package picker

If you are a member of Launchpad’s beta testers team, you’ll now have a slightly different interface for selecting source packages in the Launchpad web interface, and we’d like to know if it goes wrong for you.

One of our longer-standing bugs has been #42298 (“package picker lists unpublished (invalid) packages”).  When selecting a package – for example, when filing a bug against Ubuntu, or if you select “Also affects distribution/package” on a bug – and using the “Choose…” link to pop up a picker widget, the resulting package picker has historically offered all possible source package names (or sometimes all possible source and binary package names) that Launchpad knows about, without much regard for whether they make sense in context.  For example, packages that were removed in Ubuntu 5.10, or packages that only exists in Debian, would be offered in search results, and to make matters worse search results were often ordered alphabetically by name rather than by relevance.  There was some work on this problem back in 2011 or so, but it suffered from performance problems and was never widely enabled.

We’ve now resurrected that work from 2011, fixed the performance problems, and converted all relevant views to use it.  You should now see something like this:

New package picker, showing search results for "pass"

Exact matches on either source or binary package names always come first, and we try to order other matches in a reasonable way as well.  The disclosure triangles alongside each package allow you to check for more details before you make a selection.

Please report any bugs you find with this new feature.  If all goes well, we’ll enable this for all users soon.

Update: as of 2016-09-22, this feature is enabled for all Launchpad users.

Read more
Colin Watson

We just rolled out a new feature for Launchpad’s Git repository hosting: Git-based merge proposals can now be linked to Launchpad bugs.  This can be done manually from the web UI for the merge proposal, but normally you should just mention the Launchpad bug in the commit message of one of the commits you want to merge.  The required commit message text to link to bugs #XXX and #YYY looks like this:

LP: #XXX, #YYY

This is the same form used for Launchpad bug references in debian/changelog files in source packages, and the general approach of mentioning bugs in commit messages is similar to that of various other hosting sites.

Bugs are not automatically closed when merge proposals land, because the policy for when that should happen varies from project to project: for example, projects often only close bugs when they make releases, or when their code is deployed to production sites.

Users familiar with Bazaar on Launchpad should note that the model for Git bug linking is slightly different: bugs are linked to merge proposals rather than to individual branches.  This difference is mainly because individual branches within a Git repository are often much more ephemeral than Bazaar branches.

Documentation is here, along with other details of Launchpad’s Git hosting.

Read more
Colin Watson

If you are a member of Launchpad’s beta testers team, you can now try out Git-based recipes.  These work very similarly to the Bazaar-based recipes that Launchpad has supported for a long time, with a few minor changes to handle differences in the underlying version control system: the main thing you’ll probably notice is that you almost always need to explicitly specify a branch name.  The recipes documentation explains what to do; if you’re already familiar with how recipes work for Bazaar, then you should probably skip straight to the recipes guide to look over the differences for Git.

Please try this out and report any bugs you find.  If all goes well, we’ll open this up to all users in a couple of weeks.

Update: as of 2016-03-15, this feature is enabled for all Launchpad users.

Read more
Colin Watson

Here’s what the Launchpad team did in October.

Code

  • Ensure that enabled and restricted processors are left untouched when submitting Archive:+edit (#1501519)
  • Describe merge proposal listings in the second person (#991945)
  • Increase Git backend communication timeout to 30 seconds (from 5), as detecting merges can be quite slow on large repositories (#1502748)
  • Link directly to the recipe/snap in the one-recipe/snap case, rather than redirecting from +recipes/+snaps views (#891918)
  • Add GitRepository:+subscription traversal (#1503749)
  • Send notification mail when a merge proposal is superseded (#716169)
  • Add Branch:+dependent-merges and GitRef:+dependent-merges views (#496056)
  • Parse extended header lines in git diffs correctly (#1510337)

Additionally, we’ve added webhook support for merge proposals, made a few more UI improvements, and enabled webhooks for beta testers.  Feedback welcome!

Snap packages

  • Allow snap package owners to enable/disable unrestricted processors (matches the change made for PPAs last month)

Soyuz (package building)

  • Launchpad now accepts uploads of binary packages from builders with file modification times up to 24 hours in the future (rather than 8) or as early as 1975 (rather than 1984), to match Debian
  • Notifications are no longer generated for package copy jobs that in fact end up being no-ops

Also, as previously announced, we and our sysadmins have been putting a lot of work into extending ScalingStack to other architectures and making it more reliable, and as a result we can now offer general access to PPAs for the ppc64el architecture.

Read more
Colin Watson

If you are a member of Launchpad’s beta testers team, you can now try out webhooks for Bazaar branches and Git repositories. These can be used to set up integration with external sites for various purposes, such as running CI jobs or publishing documentation. We expect to open this up to all Launchpad users soon, but in the meantime please do file a bug against Launchpad itself if you encounter any problems.

See our webhooks documentation for more details.

Update: as of 2015-11-20, this feature is enabled for all Launchpad users.

Read more
Colin Watson

PPAs for ppc64el

Personal package archives on Launchpad only build for the amd64 and i386 architectures by default, which meets most people’s needs.  Anyone with an e-mail address can have a PPA, so they have to be securely virtualised, but that’s been feasible on x86 for a long time.  Dealing with the other architectures that Ubuntu supports (currently arm64, armhf, powerpc, and ppc64el) in a robust and scalable way has been harder.  Until recently, all of those architectures were handled either by running one builder per machine on bare metal, or in some cases by running builders on a small number of manually-maintained persistent virtual machines per physical machine.  Neither of those approaches scales to the level required to support PPAs, and we need to make sure that any malicious code run by a given build is strictly confined to that build.  (We support virtualised armhf PPAs, but only by using qemu-user-static in an amd64 virtual machine, which is very fragile and there are many builds that it simply can’t handle at all.)

We’ve been working with our sysadmins for several months to extend ScalingStack to non-x86 architectures, and at the start of Ubuntu’s 16.04 development cycle we were finally able to switch all ppc64el builds over to this system.  Rather than four builders, we now have 30, each of which is reset to a clean virtual machine instance between each build.  Since that’s more than enough to support Ubuntu’s needs, we’ve now “unrestricted” the architecture so that it can be used for PPAs as well, and PPA owners can enable it at will.  To do this, visit the main web page for your PPA (which will look something like “https://launchpad.net/~<person-name>/+archive/ubuntu/<ppa-name>”) and follow the “Change details” link; you’ll see a list of checkboxes under “Processors”, and you can enable or disable any that aren’t greyed out.  This also means that you can disable amd64 or i386 builds for your PPA if you want to.

We’re working to extend this to all the existing Ubuntu architectures at the moment.  arm64 is up and running but we’re still making sure it’s sufficiently robust; armhf will run on arm64 guests, and just needs a kernel patch to set its uname correctly; and powerpc builds will run in different guests on the same POWER8 compute nodes as ppc64el once we have suitable cloud images available.  We’ll post further announcements when further architectures are unrestricted.

Read more
Colin Watson

October already! As the leaves start to turn red here in the northern hemisphere, here’s a brief summary of what we did in September.

Bugs

  • BugTask:+addcomment’s title doesn’t duplicate the bug number (#1323808)
  • The duplicates portlet of bug tasks no longer links to invisible private bugs and now uses the correct sprite for each bug (#1443418, #1465880)
  • Show hidden bug comments to their owners (#1391394)

Code

  • Line numbers in merge proposal preview diffs are now unselectable (#1483925)
  • Branch revision listings link more sensibly to merged branches and merge proposals (#711647)
  • Git repositories and refs have more sensible breadcrumb links (#1466271)
  • Source package recipe builds now include the distribution series in their titles (#1491336)
  • Allow archive owners to cancel their recipe builds, and make recipes use the normal cancellation infrastructure (#624630)
  • Make it possible to retry superseded builds, since there are situations where they can become unsuperseded (#444030)
  • The index page for Git refs is more useful
  • Fix crashes on various +activereviews pages when there are no active reviews (#1499744, #1501134)
  • Precache branch permissions in Branch.landing_candidates, fixing API timeouts (#1500576)
  • Add webhook support for Bazaar branches, currently enabled on qastaging though not yet on production

Mail

  • Team membership notifications now honour the “Include filtering information in email footers” setting, and have rationale information in the headers and footer (#296889, #508897)
  • There is a new X-Launchpad-Message-For mail header (or Launchpad-Message-For in expanded footers) giving just the name of the person or team directly subscribed to the notification (#1493844)
  • Treat “me” in person-or-team contexts in mail handlers as the current user (#340397)

We now consider the “Gmail filtering improvements” work complete.  Let us know if there are further categories of mail sent by Launchpad that you’re finding difficult to filter using Gmail.

Registry

  • Distribution index pages hide links to disabled features (#80315, #257627)
  • Set consistent colour for the “Opinion” bug task status (#648645)
  • Blueprints on milestone pages have icons again and are sortable (#1354387)
  • Old user-to-user email database references no longer prevent changing team privacy (#1498497)

Snap packages

  • Snap packages now have edit, administer, and delete views
  • Branches, Git repositories and refs, people, teams, and products now have snap package listing views
  • Members of ~launchpad-snap-builders can create snap packages in the web UI based on Bazaar or Git branches
  • Snap package owners can request builds in the web UI

Soyuz (package building)

  • Stop showing the confusing .dsc component on source package index pages (#521722)
  • Export archive deletion on the API (#814633)
  • Allow overriding the build version for live filesystem builds (#1496074)
  • Archive owners can change the “build debug symbols” and “publish debug symbols” settings on their own archives, rather than needing to ask an admin
  • The publisher no longer sometimes tries to update Contents files for immutable suites such as the release pocket of stable series, which could cause checksums in Release to get out of sync (#1448270)
  • Publishing PPAs now creates clearsigned InRelease files to improve reliability of updates (#804252); we intend to add this to the primary Ubuntu archive as well once one last piece of mirroring infrastructure is made ready for it
  • Archive owners can enable or disable unrestricted architectures on their own archives, rather than needing to ask an admin; we will unrestrict further architectures once they have adequate virtualised build infrastructure available
  • Enabling a test rebuild archive no longer times out (#1500973)

Read more
Colin Watson

Here’s a summary of what the Launchpad team got up to in August.

Code

  • Webhook support for Git repositories is almost finished, and only needs a bit more web UI work (#1474071)
  • The summary of merge proposal pages now includes a link to the merged revision, if any (#892259)
  • Viewing individual comments on Git-based merge proposals no longer OOPSes (#1485907)

Mail notifications

Our internal stakeholders in Canonical recently asked us to work on improving the ability to filter Launchpad mail using Gmail.  The core of this was the “Include filtering information in email footers” setting that we added recently, but we knew there was some more to do.  Launchpad’s mail notification code includes some of the oldest and least consistent code in our tree, and so improving this has entailed paying off quite a bit of technical debt along the way.

  • Bug notifications and package upload notifications now honour the “Include filtering information in email footers” setting (#1474071)
  • Bug notifications now log an OOPS rather than crashing if the SMTP server rejects an individual message (#314420, #916939)
  • Recipe build notifications now include an X-Launchpad-Archive header (#776160)
  • Question notification rationales are now more consistent, including team annotations for subscribers (#968578)
  • Package upload notifications now include X-Launchpad-Message-Rationale and X-Launchpad-Notification-Type headers, and have more specific footers (#117155, #127917)

Package build infrastructure

  • Launchpad now supports building source packages that use Debian’s new build profiles syntax, currently only with no profiles activated
  • Launchpad can now build snap packages (#1476405), with some limitations; this is currently only available to a group of alpha testers, so let us know if you’re interested
  • Builders can now access Launchpad’s Git hosting (HTTPS only) in the same way that they can access its Bazaar hosting
  • All amd64/i386 builds now take place in ScalingStack, and the corresponding bare-metal builders have been detached pending decommissioning; some of the newer of those machines will be used to further expand ScalingStack capacity
  • We have a new ScalingStack region including POWER8-based ppc64el builders, which is currently undergoing production testing; this will replace the existing POWER7-based builders in a few weeks, and also provide virtualised build capacity for ppc64el PPAs
  • We’ve fixed a race condition that sometimes caused a user’s first PPA to be published unsigned for a while (#374395)

Miscellaneous

  • The project release file upload limit is now 1 GiB rather than 200 MiB (#1479441)
  • We spent some more time supporting translations for the overlay PPA used for current Ubuntu phone images, copying a number of existing translations into place from before the point when they were redirected automatically
  • Your user index page now has a “Change password” link (#1471961)
  • Bug attachments are no longer incorrectly hidden when displaying only some bug comments (#1105543)

Read more
Colin Watson

Here’s a summary of what the Launchpad team got up to in July.

Code

  • We fixed a regression in the wrapping layout of side-by-side diffs on bazaar.launchpad.net (#1436483)
  • Various code pages now have meta tags to redirect “go get" to the appropriate Bazaar or Git URL, allowing the removal of special-casing from the “go" tool (#1465467)
  • Merge proposal diffs including mention of binary patches no longer crash the new-and-improved code review comment mail logic (#1471426), and we fixed some line-counting bugs in that logic as well (#1472045)
  • Links to the Git code browsing interface now use shorter URL forms

We’ve also made a fair amount of progress on adding support for triggering webhooks from Launchpad (#342729), which will initially be hooked up for pushes to Git repositories.  The basic code model, webservice API, and job retry logic are all in place now, but we need to sort out a few more things including web UI and locking down the proxy configuration before we make it available for general use.  We’ll post a dedicated article about this once the feature becomes available.

Mail notifications

We posted recently about improved filtering options (#1474071).  In the process of doing so, we cleaned up several older problems with the mails we send:

  • Notifications for a bug’s initial message no longer include a References header, which confuses some versions of some mail clients (#320034)
  • Package upload notifications no longer attempt to transliterate non-ASCII characters in package maintainer names into ASCII equivalents; they now use RFC2047 encoding instead (#362957)
  • Notifications about duplicate bugs now include an X-Launchpad-Bug-Duplicate header (#363995)
  • Package build failure notifications now include a “You are receiving this email because …” rationale (#410893)

Package build infrastructure

  • The sbuild upgrade last month introduced some regressions in our handling of package builds that need to wait for dependencies (e.g. #1468755), and it’s taken a few goes to get this right; this is somewhat improved now, and the next builder deployment will fix all the currently-known bugs in this area
  • In the same area, we’ve made some progress on adding minimal support for Debian’s new build profiles syntax, applying fixes to upload processing and dependency-wait analysis, although this should still be considered bleeding-edge and unlikely to work from end to end
  • We’ve been working on adding support for building snap packages (#1476405), but there’s still more to do here; we should be able to make this available to some alpha testers around mid-August

Miscellaneous

  • We’ve arranged to redirect translations for the overlay PPA used for current Ubuntu phone images to the ubuntu-rtm/15.04 series so that they can be translated effectively (#1463723); we’re still working on copying translations into place from before this fix
  • Projects and project groups no longer have separately-editable “display name” and “title” fields, which were very similar in purpose; they now just have display names (#1853, #4449)
  • Cancelled live file system builds are sorted to the end of the build history, rather than the start (#1424672)

Read more
Colin Watson

Users of some email clients, particularly Gmail, have long had a problem filtering mail from Launchpad effectively.  We put lots of useful information into our message headers so that heavy users of Launchpad can automatically filter email into different folders.  Unfortunately, Gmail and some other clients do not support filtering mail on arbitrary headers, only on message bodies and on certain pre-defined headers such as Subject.  Figuring out what to do about this has been tricky.  Space in the Subject line is at a premium – many clients will only show a certain number of characters at the start, and so inserting filtering tags at the start would crowd out other useful information, so we don’t want to do that; and in general we want to avoid burdening one group of users with workarounds for the benefit of another group because that doesn’t scale very well, so we had to approach this with some care.

As of our most recent code update, you’ll find a new setting on your “Change your personal details” page:

Screenshot of email configuration options

If you check “Include filtering information in email footers”, Launchpad will duplicate some information from message headers into the signature part (below the dash-dash-space line) of message bodies: any “X-Launchpad-Something: value” header will turn into a “Launchpad-Something: value” line in the footer.  Since it’s below the signature marker, it should be relatively unobtrusive, but is still searchable.  You can search or filter for these in Gmail by putting the key/value pair in double quotes, like this:

Screenshot of Gmail filter dialog with "Has new words" set to "Launchpad-Notification-Type: code-review"

At the moment this only works for emails related to Bazaar branches, Git repositories, merge proposals, and build failures.  We intend to extend this to a few other categories soon, particularly bug mail and package upload notifications.  If you particularly need this feature to work for some other category of email sent by Launchpad, please file a bug to let us know.

Read more