Canonical Voices

Posts tagged with 'api'

Curtis Hovey

Launchpad’s bug and branch privacy features are being replaced by information sharing that permits project maintainers to share kinds of information with people at the project level. No one needs to manage bug and branch subscriptions to ensure trusted users have access to confidential information.

Maintainers can share and unshare their project with people

Project maintainers and drivers can see the “Sharing” link on their project’s front page. The page lists every user and team that the project shares with. During the transition period of the beta, you might see many users with “Some” access to “Private Security” or “Private” user information. They have this access because they are subscribed to bugs and branches. Maintainers can unshare with users who do not need access to any confidential information, or just unshare a bug or branch with a user. Maintainers can share share with a team to give them full access to one or more kinds of confidential information.

I have prepared a video that demonstrates the features (my apologies for the flickering)

Commercial projects can set bug and branch policies

Projects with commercial subscriptions can also change bug and branch sharing policies to set the default information type of a bug or branch, and control what types they may be changed to. Maintainers can set policies that ensure that bugs and branches are proprietary, and only proprietary, to ensure confidential information is never disclosed.

Sharing can be managed using API scripts

I maintain many project which have a lot of private bugs and branches. The sharing page lists a lot of people, too many to read quickly. I know most work for my organisation, but I don’t even know everyone in my organisation. So I wrote a Launchpad API script that can be run by any project maintainer to share the project with a team, then unshare with the team members. The members still have access to the bugs and branches and their subscriptions still work, but they will lose access to my project when they leave the team. This arrangement makes it very easy to manage who has access to my projects. share-projects-with-team.py is run with the name of the team and a list of projects to share with it.

./share-projects-with-team.py my-team project1 project2

Read more
Colin Watson

The Ubuntu Foundations team has sponsored work on various improvements to Launchpad’s archive handling lately, mainly to expose various new facilities on the API where we were previously using privileged scripts.  This has involved cleaning up a substantial amount of old code along the way, and it has become possible to fix some other old bugs as spin-offs.

One of these old bugs is “Archive:+copy-packages nearly unusable due to timeouts”.  The +copy-packages page allows anyone who can upload to a PPA to instead copy packages from another PPA.  This saves effort, and in the “Copy existing binaries” mode it can save a substantial amount of build time as well.  For example, the LibreOffice packaging team uses this to deliver packages to different sets of users after they have passed various levels of testing.

Unfortunately, the very cases where this is most useful, namely large and complex packages, are also the cases where it is most likely to break.  Copying large numbers of binary packages involves large numbers of database queries and can quite easily overrun the timeout for a single request to the Launchpad web application.  Doing this for several series at once, a common case which seems reasonable, is proportionally less likely to work.  Various attempts have been made to optimise the database interactions here, but ultimately doing lots of complex synchronous work in time for a single web request is doomed to failure.

The solution to all this is to copy packages asynchronously.  For some time Launchpad has had the ability to schedule “package copy jobs” which run very shortly after the request (typically within a minute) but not immediately.  For example, the Ubuntu team uses these when copying new versions of packages from Debian unstable in cases where there are no Ubuntu-specific modifications, and when releasing proposed updates to stable releases for general use after verification.  A similar facility has been present in the code for the +copy-packages page for some time, but not exposed due to various bugs.  We believe that these bugs have been fixed now, and so we would like to start copying packages asynchronously when requested via the web UI.

We have exposed this to beta testers first.  The effect is that, if you are a beta tester when you ask for packages to be copied, you will be told something like “Requested sync of 2 packages.  Please allow some time for these to be processed.”  The processing should normally happen within a minute or two, and you will be able to see it in progress on the +packages page for the target archive.  If it succeeds, the in-progress notification will be removed and you will be able to see the changes in the target archive.  Otherwise, you will see a failure notification along these lines:

A notification of a failed copy to a PPA.

If beta-testing goes well, then we will enable this for all users, and remove the old synchronous copying code shortly afterwards; so please do report any problems you see.

If you are relying on package copies in the web UI happening immediately rather than within a few minutes, firstly, please contact us (e.g. #launchpad-dev on freenode IRC, or launchpad-users@lists.launchpad.net) as we would like to understand your requirements in more detail; secondly, you may be able to use the Archive.syncSource API method instead, which also has timeout constraints but is at least guaranteed to remain synchronous.  However, we hope that most people will not have such a requirement.

Read more
Curtis Hovey

We are reimagining the nature of privacy in Launchpad. The goal of the disclosure feature is to introduce true private projects, and we are reconciling the contradictory implementations of privacy in bugs and branches.

We are adding a new kind of privacy called “Proprietary” which will work differently than the current forms of privacy.

The information in proprietary data is not shared between projects. The conversations, client, customer, partner, company, and organisation data are held in confidence. proprietary information is unlikely to every be made public.

Many projects currently have private bugs and branches because they contain proprietary information. We expect to change these bugs from generic private to proprietary. We know that private bugs and branches that belong to projects that have only a proprietary license are intended to be proprietary. We will not change bugs that are public, described as security, or are shared with another project.

This point is a subtle change from what I have spoken and written previously. We are not changing the current forms of privacy. We do not assume that all private things are proprietary. We are adding a new kind of privacy that cannot be shared with other projects to ensure the information is not disclosed.

Launchpad currently permits projects to have default private bugs and branches. These features exist for proprietary projects. We will change the APIs to clarify this. eg:

    project.private_bugs = True  => project.default_proprietary_bugs = True
    project.setBranchVisibilityTeamPolicy(FORBIDDEN) => project.default_proprietary_branches = True

Projects with commercial subscriptions will get the “proprietary” classification. Project contributors will be able to classify their bugs and branches as proprietary. The maintainers will be able to enable default proprietary bugs and branches.

Next part: Launchpad will use policies instead of roles to govern who has access to a kind of privacy.

Read more
Curtis Hovey

Launchpad will soon permit you to say your project is not affected by a bug shared with another project — you can delete the spurious bug task. This action can be done from the bug’s web page, and using Launchpad API.

You can remove a project from a bug if you are the project maintainer, bug supervisor, or are the person who added the project to the bug. This action can only be performed when the bug affects more than one project — you cannot delete an entire bug. This feature permits you to undo mistakes.

Launchpad beta testers will see the remove action next to the affected project name in the affects table.

Delete spurious bugs

The delete() method was added to the bug_task entry in the Launchpad API. There is a example API script, delete_bugtasks.py, that can remove a project from many bugs. There is also a split action to create a separate bug just for the specified project to track separate conversations in bug comments.

Usage: delete_bugtasks.py [options] bug_id:project_name [bug_id:project_name]

Options:
  -h, --help            show this help message and exit
  -v, --verbose
  -q, --quiet
  -f, --force           Delete or split bug tasks that belong to a public bug
  -d, --delete          Delete a spurious project bug task from a bug
  -s, --split           Split a project bug task from an existing bug so that
                        the issue can be tracked separately
  -w WEBSITE, --website=WEBSITE
                        The URI of Launchpad web site.
                        Default: https://api.launchpad.net;
                        Alternates: https://api.staging.launchpad.net,

https://api.qastating.launchpad.net

Previously, you could not remove spurious bug reports about your project. Many were cause by poor bug target management; you could not move a bug between projects and distributions. You can now move bugs between projects and distributions, but thousands of bugs still wrongly claim to affect a project or distribution. This causes clutter on bug pages and searches, and it causes Launchpad performance problems.

This change is a part of a super-feature called Disclosure. To ensure that confidential data is not accidentally disclosed, Launchpad will only permit private bugs to affect a single project. Soon, you may need to remove a project from a bug before marking the bug private.

Read more
Aaron Bentley

I recently posted about Initializing page JavaScript from the JSONCache. Now I’m pleased to announce that you can also get updated copies of the IJSONRequestCache, to make it easier to update your page.

Brad Crittenden and I started work on this at the Dublin Thunderdome, and it’s finally been deployed. What this means is that for basically any page on Launchpad, you can append /++model++ to the URL, to get a fresh copy of the IJSONRequestCache. With ++model++, a change will typically require only two roundtrips; one to make a change, and one to retrieve an updated model. Future work may reduce this to a single roundtrip.

Why ++model++, not ++cache++? Cache is a really poor name for what the IJSONRequestCache is. Rather than providing fast access to whatever data has been previously retrieved, it is a complete collection of all the relevant data.

In Launchpad, the IJSonRequestCache is associated with the view, so we’re trying to rebrand it as the “view model”. This may seem strange from an MVC (Model, View Controller) perspective, but MVC can be recursive. A view may use a model to render itself.

Read more
Matthew Revell

Getting started with launchpadlib

zx81I’m not a developer. I’ve played around — various flavours of BASIC as a kid, Python nowadays — but it rarely goes beyond a little experimentation.

Lately, though, my lack of Python-fu has been increasingly frustrating. In particular, there are times when I want to know something that I’m pretty sure Launchpad can tell me but there’s not an easy way to get at that information through the web UI.

For just those occasions we have the Launchpad web services API and its Python library, launchpadlib.

So, I’m going to learn my way around launchpadlib and write about my experience here. Bear in mind, I’m writing this from the perspective of someone who isn’t a developer. That may seem odd but I’m willing to be there are plenty of people out there who want to do the same as me: get more out of Launchpad and the data it hosts.

Getting started

You can install launchpadlib straight from the primary Ubuntu archive in recent releases. That’s what I’ve done so, unless I say otherwise, assume I’m writing about the version of launchpadlib that’s available in Ubuntu 11.04 (Natty).

Installation takes nothing more than a simple:


sudo apt-get install python-launchpadlib

Authenticating with Launchpad

When you access Launchpad through launchpadlib you can opt to either:

  • access Launchpad anonymously
  • log in with a user account and grant that launchpadlib instance a certain level of access to the account’s data.

If all you need is read-only access to public data, you may as well access Launchpad anonymously. For anything where you need to write data or access data that’s available only when logged in, each person who uses your launchpadlib application will need to grant their instance of the app access to their Launchpad account.

The Launchpad API, and so launchpadlib, lets OAuth take care of the authentication. It’s pretty simple really: when you run your app it opens up a Launchpad page in your browser. On that page you can choose what level of access that particular application has to your account. That log-in should be a one-time thing, as launchpadlib will cache your credentials.

And you can check/change what access different API apps have at any time.

Accessing Launchpad data

launchpadlib lets you access Launchpad data and functionality just like they were any other Python object. That’s great for someone like me because it means I don’t have to care about REST APIs, HTTP or any of that.

In my next post I’ll start actually getting hold of Launchpad data and maybe trying to do something with it.

Photo by Sébastien Bertrand. Licence: CC BY 2.0

Read more
Matthew Revell

Launchpad is Go!

Go!There’s a new Launchpad client library, called lpad, for the Go programming language.

Gustavo writes:

lpad is based on a two-layered design. The top layer offers a static API which allows a more comfortable interaction with the API with static checks, better documentation, and more. The bottom layer is fully dynamic and enables the developer to access all the features of Launchpad, even those not supported by the top static layer.

There’s still work to do but the library is pretty much complete and it’s well tested, including integration tests which communicate with the real production servers.

You can get hold of lpad with a simple:

bzr branch lp:lpad

Check out the full API documentation.

Photo by Iain Farrell. Licence: CC BY-ND 2.0.

Read more
Martin Pool

Launchpad has a web UI, an email interface, and a ReST API that exposes every object in the database.

There are also a bunch of client programs, command line and graphical, that talk to Launchpad to do various things.

What we don’t yet have, and what I think would be great, is a systematic client that lets you manipulate
everything
from the command line. There’s some code that starts towards this in Hydrazine, lptools and others, but I think having just a single tool that eventually does everything would be more discoverable and avoid unnecessary fragmentation or duplication.

(That’s not to say there’s not room for others that are guis, that are specialized to particular projects or that encapsulate a lot of policy or opinion about what they’re doing.)

So dobey and I have agreed to gradually merge hydrazine into lptools, and with other people to work towards making lptools cover everything you can do through the web UI or the API. If you have scripts you’ve written yourself, perhaps you’d like to merge them in.

Read more
Martin Pool

Launchpad API client developers: have your say in bug 714043 on whether launchpadlib should connect by default to Launchpad’s staging server (so data is discarded), or to real Launchpad.

Read more
Martin Pool

Launchpad has a REST API that exposes almost every object within Launchpad. Most of them have API URLs that closely match the human-readable URL: for instance, https://bugs.launchpad.net/bugs/316694 can also be obtained in machine-oriented JSON or XML form from http://api.launchpad.net/1.0/bugs/316694. (The “1.0″ in that URL means this is in the 1.0 API namespace.)

Previously, many Launchpadlib clients used string transformation to get from the API URL to something they could show a human in a web browser.

We’ve just added a web_link property on all Launchpad objects, so client applications can (and should) now just use that instead. Because this is just sent in the object representation you don’t need to upgrade launchpadlib to see it.

Read more
Julian Edwards

An long-requested feature in Launchpad is to let people see who’s using a PPA. Finally, we’ve implemented this!

Initially, the stats are only available on Launchpad’s webservice API. but we aim to show something useful in the web UI at some point.

If you are already familiar with the webservice API, then you can use the following binary_package_publishing_history object methods to retrieve the information:

  • getDailyDownloadTotals
  • getDownloadCount
  • getDownloadCounts

Fabien Tassin is already using the stats to see how many people are using his daily build PPAs, and wrote an interesting blog post about it.

Read more
Martin Pool

Three tips from Leonard’s lightning talk in Prague about writing faster Launchpadlib API clients:

1. Use the latest launchpadlib. It gets faster from one release to the next. (The versions in the current Ubuntu release should be fine; otherwise run from the branch or the latest tarball.)

2. Profile:

    import httplib2
    httplib2.debuglevel = 1

will show each http request and response, so that you can see what’s taking time.

3. Fetch objects only once:

Don’t do this:

    if bug.person is not None:
        print bug.person.name

instead

    p = bug.person
    if p is not None:
        print p.name

In the first case, the client may fetch the Person object twice. (We may fix this in future.)

Read more
Matthew Revell

Launchpad’s strategist, Jonathan Lange, has started a series of blog posts on getting started with Launchpad’s Python library, launchpadlib:

launchpadlib is the Python client-side library that talks to Launchpad’s own REST API. It turns out that customize scripted control of a bug-tracker-code-hosting-translation-distribution-building-cross-project-collaboration thing is actually quite handy.

Catch-up with the first two posts in the series:

And you can subscribe to Jonathan’s blog or follow blogs from both Jonathan and other members of the Launchpad community on Planet Launchpad.

Read more