Canonical Voices

Jeroen T. Vermeulen

If you use Launchpad’s Translations feature, then as of today, you may notice a new kind of error for uploads in your translations import queue. Look for the “info” icon next to your upload’s status, and click to unfold. The error message says “Can’t auto-approve upload: it’s not clear what template it belongs to.”

Sample

You may find this annoying, and as the person responsible, let me apologize. I hope when I’m done explaining, it won’t seem so bad.

To the right is an example of the error.  Click it to see more.

Where does the error come from? Actually it’s a problem that’s always been there, but instead of telling you about it, we Launchpad developers tried to hide the complexity from you and take care of it all ourselves. And we weren’t keeping up. Some of your uploaded translations would just sit in the queue forever, with status Needs Review, for no clear reason. All that’s changing now is that Launchpad will tell you when this happens, so that you can deal with it without waiting for us.

Why do some translations sit in the queue forever?

Every translation you make in Launchpad has to go into a specific template and language. Usually it’s obvious where you want your translation to go: when you translate in Launchpad’s web UI, you’re already on the page for a specific template and language. If you have upload privileges for the project and language, you can follow the upload link from that same translation page and again it’s obvious which template you’re translating and to what language. If you always upload the same file with exactly the same name and path, new uploads of that file go to the same place as before. But what if you upload a new file from the release-series page, or the translation comes in from a Bazaar branch or an automated build?

Then it’s up to a script we call the Translations Import Queue Gardener. The Gardener periodically scans all waiting uploads—the ones marked Needs Review—and tries more advanced ways of matching them up with known templates and languages. When it finds a match, it approves the upload’s import to the template and language it has found.

One of the Gardener’s most important tools is a template’s ”translation domain.” This is a simple name; no slashes or weird characters allowed. Launchpad figures out a template’s domain when you first import the template, based on its directory path. In principle a template’s domain should identify the template uniquely on the end-user’s system, but Launchpad isn’t as strict about it. It just assumes that the domain name is tied to the template file’s path within a project’s source tree. You probably shouldn’t, but you can give your project two templates with the same domain if you want.

When you upload a translation, the Gardener tries to figure out its translation domain based on the file’s path, just like what happened when the template was created. The Gardener looks for a template with the right domain in the same release series. If it finds one, presto: it’s got the template that the upload is meant for. If not, then the Gardener tries a few other things and if nothing works, simply keeps the upload on the queue.

So the entry doesn’t get imported, but usually the Gardener can’t give any single reason: all it knows is that it tried many ways of matching the file to a template and none of them worked. Maybe it’s an error; maybe it’s just a matter of waiting for the right template to be created.

So what’s changed?

The new error message is about “approval conflicts.” You’ll see it when there is ”more than one” matching template for your upload. This can happen because your project’s directory structure is unusual and Launchpad can’t extract a meaningful domain from it. Or a template’s domain may have been changed, or an old deactivated template is reactivated when a new one has already taken its place.

Whatever the cause, the new error message tells you that this has happened, and what the matching templates are. It’s up to you to decide what needs to be done about it:

  • update one of the templates’ domains, or
  • deactivate an obsolete template, or
  • move a file, or
  • re-upload your file to a specific template, or
  • re-upload your file with a different name, or
  • upload translations as a tarball so that Launchpad sees their full directory paths.

How much you can do, of course, depends on the permisisons you have. Only the project’s owners (and Launchpad’s administrators) can deactivate templates or change their domains. But you can always delete your own uploads if you want to upload your file differently: on the import-queue page, click on Needs Review and select Deleted instead.

Let us know

There’s still plenty more we’d like to do to make imports easier and more efficient, if we can find the time. But I hope this small change will make your life a little easier.

Is this error message working for you? Is it helpful? Are you seeing a lot of these errors, or none at all where you were expecting them? Is the explanation unclear? Do you see something happen for lots of people that we could fix in the same way? Please get in touch:

  • Contact danilos or jtv on IRC, in #launchpad on Freenode.
  • Find or file a bug if it’s broken.

Tips

  • If you want to become more familiar with the translations import queue, check out the global queue to see all uploads in Launchpad. The version you see there is just a copy on a test server, so don’t be afraid to play with it.
  • By the way, did you know about our test servers? We test our changes on staging and qastaging servers before they go live. You can try out most of Launchpad’s features there. Look for the grey “demo” text in the background. They get restored to a fresh copy of the real Launchpad once a week.
  • Tired of creating translations tarballs and uploading them to your project? Automate it all with Bazaar integration.
  • You want Bazaar integration but your code is hosted outside Launchpad and/or in a different revision control system? You can tell Launchpad to mirror a branch from elsewhere. Then you can import translations from the mirrored copy.
  • We like transparency! If you’re interested in the engineering details of this change, it’s all online.
  • Read more
Francis J. Lacoste

Speeding up development

SpeedometerToday we reached a significant milestone, we completed our first fast down-time deployment. Two obvious reasons for doing this were already mentioned in the announcement and our technical architect”s post describing the change:

  1. We’ll have less downtime per month (at the cost of more frequent but short interruption).
  2. We’ll be able to deploy fixes and changes involving DB schema more frequently.

But from my perspective, the most important benefit I think we’ll get from this is a speed up in our rate of development, particularly, in terms of completing feature projects. It’s not a secret, our feature squads spends a lot of time to complete their projects. There are multiple reasons for this, but in the end, there usually fall under two broad cateries: the time it takes to actually make the change, and the delays in getting feedback on the change itself.

To help with the first category, you’ll want better and more powerful libraries, better architecture, developers’ training, etc. Think the time difference between developping a database web application in Django vs as a CGI C application using only the standard C library. Launchpad isn’t using the most modern libraries and toolkits, and we could still make a lot of improvements there.  But the costs of making changes in this space are compounded by the problems of the other category.

Once you wrote the change, you are far from done. There are lots of hoops you still have to jump through before saying “done-done“: you’ll need to make sure the tests pass, to get your changes reviewed, merged, QAed and then deployed. And finally, you’ll probably want to make sure that it matches the user’s expectations, but until it’s in production, this is hard to assess reliably. All of these steps takes time and introduce delays, some bigger than others. The Launchpad team is always on the look-out to cut in these delays and the new “fast down-time deployments” cut on of the biggest one we had.

Cycle time distribution

Since a picture is worth at least a thousand words, have a look at the chart above to have a better idea of what I’m talking about. It shows the distribution of the time it takes to complete a “change”. (What this plots is the cycle time from coding to deployment of our Kanban cards which roughly map to one logical change.) You’ll see that 50% of our changes are deployed to production in about a week. And the next 45% takes between 1 and 5 weeks. Now, our feature projects are composed of many many of these smaller changes. If those are all  relatively small changes, why do they take so long?

One of the big bottleneck was the batch size of our DB deployment. If a change required a DB schema it waited until the next downtime deployment which happened once every month. In theory, that means that on average a change involving the database would wait 2 weeks in the queue before deployment. In practice, it’s more complex than that, because squad leads would often plan around these. So a database change would be hold off onto because it was deemed that it couldn’t be safely completely to be part of the next downtime deployment. So it might be put on hold in favour of other work, and delayed to the next downtime deployment. It’s also frequent to have other changes building on the first also queued up waiting for the next deployment window. Add on top of that, that it’s common for the completion of a feature to require several iterations of DB change based on feedback and you quickly understand how you can be working months on a feature project!

But this major bottleneck is now gone! We’ll be able to land and deploy DB changes reliably within days, giving us much more rapid feedback. I’m looking forward to the change in the cycle time distribution in the coming months. The whole distribution should move toward the left. I’ll write a follow-up in two months to see if this prediction comes true.
Photo by Nathan E Photography. Licence: CC BY 2.0.

 

 

Read more
Matthew Revell

Tomorrow, you may notice a blip in Launchpad’s availability around 08.30 UTC. Believe it or not, this is good news :)

Until tomorrow, we’d been rolling out database changes — schema updates, database server maintenance, etc — once a month, with a 90 minute period where Launchpad was read-only.

Now, we’re doing things a little differently: two or three times a week, we’ll be doing a fast database update at 08.30 UTC (weekdays only). To start with, it won’t quite be “blink and you’ll miss it”. We’re talking around two minutes but we’ve already identified ways to cut this time. During the update, Launchpad will be effectively unavailable. But it’ll be quick and at a predictable time each day that we do it.

So, other than the obvious bonus of not having Launchpad go read-only for a big 90 minute block every month, why’s this good news? As it’s always at the same and for a short time, we think it’ll be easier to work around. The down-time won’t even be long enough to make a decent cup of tea or coffee. Importantly, it also means you’ll get new Launchpad code faster: if a new feature or a bug fix requires a database schema change, we can now roll it out pretty much within 24 hours rather than waiting up to a month for the next big 90 minute read-only time.

There’s a bug we need to fix: right now, during the fast down-time you’ll get an OOPS when Launchpad tries to access the database. Once we’ve fixed the bug you’ll get a somewhat friendlier and more appropriate 503 error.

While we’re all getting used to it, we’ll still announce these fast database updates on the status feed. We’re hopeful, though, that they’ll be quick enough and predictable enough (08.30 UTC weekdays, two or three times a week) that eventually you’ll have to try hard to notice them.

Read more
Matthew Revell

We’re looking for a couple of smart, motivated and experienced people to join us on the Launchpad team at Canonical.

First up is a Software Engineer, to join one of the Launchpad development squads working on both new Launchpad features and maintenance of existing functionality.

There’s also an opening for a Usability and Communications Specialist. This is to join Launchpad’s Product Team, where we’re looking for someone who can run a usability research programme and produce documentation, blog posts and so on.

If you’ve got any questions about either role, feel free to grab me (mrevell) on FreeNode.

Read more
Francis J. Lacoste

Matt RevellIt’s my pleasure to announce that as of today, Matthew Revell is the new
Launchpad Product Manager replacing Jonathan Lange.

We were seduced by his bold vision for Launchpad along with his data-centric approach that he intends to bring to the role. He also has an already extensive experience interacting with Launchpad developers and users. If you read this blog, you probably read something written by him! Or you might have interacted with him in one of the many user-research sessions he ran. The introduction of user-research helped us release better designed feature. Building on this experience, we hope that his leadership will bring Launchpad to the next level.

Matt will communicate more about his plans for Launchpad shortly. In the
mean time, let’s give him our warmest congratulations!

 

 

Read more
Matthew Revell

When you want to assign a bug to someone, subscribe them to a blueprint and so on, you see Launchpad’s person picker. It’s where you search for someone or a team, get a list of possible matches and then select the right one.

Fairly recently, we’ve made a couple of improvements to the person picker, such as adding the person/team’s unique Launchpad ID after their display name, so you stand a better chance of choosing the right person.

The trouble is, how many of us know the Launchpad ID of each person or team we’re likely to deal with?

I know I think more in terms of someone’s IRC nick or the various associations they might have, rather than what they chose as their Launchpad ID.

That’s why we’re changing the person picker: soon, everyone will get a new version of the person picker that shows you what I think is a much more useful set of information in helping find the right person.

Here’s what it might look like:

The new person picker

If you’re in the Launchpad beta testers team you might have seen it already.

If you’ve used it, let us know what you think: does it give you the information you need? Have you come across any bugs to report?

Either email feedback@launchpad.net or comment directly on this post to help us shape the new person picker!

Read more
Curtis Hovey

Users can use the affects form on the bug page to change which project or distribution the bug affects. You can also select the affected package. Lp API users can assign a project, distribution, or package to the BugTask target property to change the affected bug target. The behaviour is similar to the way questions can be retargeted between projects and distributions. Affected series cannot be changed, though the affected series package can be.

Retargeting a bug to a distribution, package, or project

Retargeting a bug to a distribution, package, or project

Previously, users had to mark a bug affecting a project or distribution as invalid, then add a new affected project or distribution. This cluttered the UI, caused excessive emails, and made pages slower.

Read more
Aaron Bentley

I recently posted about Initializing page JavaScript from the JSONCache. Now I’m pleased to announce that you can also get updated copies of the IJSONRequestCache, to make it easier to update your page.

Brad Crittenden and I started work on this at the Dublin Thunderdome, and it’s finally been deployed. What this means is that for basically any page on Launchpad, you can append /++model++ to the URL, to get a fresh copy of the IJSONRequestCache. With ++model++, a change will typically require only two roundtrips; one to make a change, and one to retrieve an updated model. Future work may reduce this to a single roundtrip.

Why ++model++, not ++cache++? Cache is a really poor name for what the IJSONRequestCache is. Rather than providing fast access to whatever data has been previously retrieved, it is a complete collection of all the relevant data.

In Launchpad, the IJSonRequestCache is associated with the view, so we’re trying to rebrand it as the “view model”. This may seem strange from an MVC (Model, View Controller) perspective, but MVC can be recursive. A view may use a model to render itself.

Read more
Matthew Revell

A road sign in Welsh and EnglishGood news if you run a project’s translation effort in Launchpad!

Until today, when you imported a template or translation file into Launchpad for the first time, you’d have to wait for a member of the Canonical Launchpad team to review and then approve that file before your project’s translation community could make use of it.

Now, if you’re a project maintainer, you can manage your project’s translations import queue yourself. All you need do is follow the “import queue” link on your project’s translations overview page and you’ll see something like this:

Translation import queue

Once you’ve approved a file, and it has been imported, subsequent changes will go through Launchpad’s automatic approval process.

Take a look at our guide to importing templates for more detail.

Road sign photo by Spixey. Licence: CC BY.

Read more
Robert Collins

Many mailing lists in Launchpad are open teams – that is, anyone is welcome to join, or leave, as they choose.

Until today, every time that happened all the list admins were mailed when someone joined or left their team, even though there is no action to take : in an open team, you cannot kick someone out.

We’ve fixed this – now for open teams (and only open teams) when someone joins or leaves the team, the team admins will not be notified.

In future we will have a subscription facility for team admins that do want these emails, and at that point we will make them optional for all team types.

Read more
Robert Collins

I’m thrilled to be writing this blog post just over a year after starting as Launchpad’s technical architect. During that year we have been steadily improving our ability to deploy changes to Launchpad without causing downtime (of any or all services). Our ability to do this directly impacts our ability to deliver bug fixes and new functionality – our users are very sensitive to downtime.

There has been one particularly tricky holdout though – our monthly 90 minute downtime window where we apply schema changes, do DB server maintenance and so forth.

Starting very soon we will instead have very short windows – approximately 60 seconds long – where we perform schema changes, database server failover (in order to permit DB maintenance on the master server) and so forth.

We expect to do these about 6 times a month based on our historical rate of schema patches, and we are – for now – planning on doing these at 0800 UTC consistently.

This will deliver much less total downtime – 6 minutes a month rather than 90 – at the cost of more frequent interruptions.

If you have API scripts running against Launchpad, you may want to build in a retry mechanism to deal with up to a few minutes of downtime.

We cannot remove downtime entirely for purely technical reasons: Our primary database (postgresql) blocks new readers (or writers) when a schema change is being executed, and the schema change blocks on existing readers (or writers) to complete – it needs an exclusive lock on each relation being altered.

What we can do is automate the process of disconnecting and interrupting existing database connections to let the schema change execute rapidly, and make our schema changes as minimal as possible. Previously, we shut down all the application servers (via a script, but shutting down gracefully takes time), and then ran schema changes which did data migration and so forth. In this new process we will leave the appservers running and just interrupt their connections for the time it take to apply the schema change. That, combined with moving data migration to a background job rather than doing it during the schema change, gives us the short downtimes we’re about to start doing.

More information is available in the LEP and my mailing list post about the project starting.

Read more
Matthew Revell

Automatic confirmation

ConfirmYou report a bug. Someone clicks “This affects me too” or marks the bug as a duplicate of another bug report.

Both of those are confirmation of your bug, right?

That’s what we think and members of the Ubuntu community agree.

So, from now on, that’s what’ll happen with New Ubuntu and Launchpad project bugs.

To recap: Launchpad will mark Confirmed any Ubuntu or Launchpad bug that has the New status and affects more than one person or is marked as a dupe.

Oh, but it doesn’t count if you mark your own bug as a dupe of another bug you reported.

We’re considering rolling this out for all projects in Launchpad. If your project uses Launchpad as its bug tracker, let us know what you think.

Photo by Seth Anderson. Licence: CC-BY-SA

Read more
Matthew Revell

Small thing this: when you change a tag on a bug, the email notification from Launchpad describes it as removing the original tag and adding a new tag.

That makes sense but something slightly irritating was that Launchpad would order the email like this:

Tags added: qa-needstesting
Tags removed: qa-untestable

The reverse order always seemed more sensible to me: i.e. first show the tags that were removed and then show the tags that were added.

Well, I wasn’t the only one to think that way and I’m glad to say that my team-mate Huw has fixed it so that bug mails now show the tags removed before the tags added.

Read more
Matthew Revell

Not long back, Sumana Harihareswara, who is Volunteer Development Coordinator at the Wikimedia Foundation, contacted some members of the Launchpad community to ask how we handle code review.

She’s written a summary of her research for the Wikimedia community.

It makes interesting reading for someone, such as me, who’s close to the process. If you’re interested in contributing to Launchpad, Sumana’s report is a really useful overview of how we review and deploy code.

Read more
Matthew Revell

Last week I announced Launchpad’s new bug subscriptions system!

While we were in Dublin, I recorded a chat with Graham Binns about the changes to bug subscriptions and the work that had gone into them.

Download the Ogg Vorbis file.

Read more
Francis J. Lacoste

Echoes from the Dublin Thunderdome

ThunderdomeI can’t believe it’s been two weeks already! From June 25th to July 1st, the whole Launchpad team was gathered in Dublin for our semi-annual all-hands event. Like usual, we also invited the Bazaar team and  one of our friendly sysadmin­.

The theme this time around was “UI, UI, UI”. Since the January squad reorganisation, it became evident that a big stumbling block for squads working on features was UI work. Our team is still inexperienced with JavaScript and the YUI3 framework. So each squad has encountered similar problems and sometime found different solutions. We wanted to take advantage of the face-time to come to a common understanding of the best patterns to use.

In the end, it was a week of intense hacking on UI infrastructure, with part of the mornings dedicated on presentation where people shared what they learnt in the past 6 months.

Projects that were worked on during the week:

  • An asynchronous notification system (extracted from Landscape) which will allow us to update pages as soon as tasks complete on the backend.
  • We ditched our buggy Windmill-based Javascript integration tests, in favour of a yuitest-based suite. We already use yuitest for JS unit tests, but now we’ll be able to use it  for integration tests where  we need to make XHR requests to a live server. We’ll even be able to set server-side fixtures from within the test!
  • Investigate the use of selenium2 for acceptance testing. These are tests ensuring a complete workflow works and that ideally will be able to run against both the staging server as well as within tree for regression testing.
  • Integrating loggerhead-backed branch data on some Launchpad pages. You should soon be able to get the diff of a particular revision directly on the merge proposal page.
  • Infrastructure to refresh all the client-side representation of objects related to a page in one request.
  • Many UI bugs were fixed.

This was also the week we said farewell to old friend: Jonathan our Product Strategist  and Ursula Junque who is now working as QA analyst for the Ubuntu Server team. In the end, Jono didn’t get pied as we still had a huge number of Critical bugs left open. We did rejoice in achieving our performance improvements target. Given the trend, I estimated that we have still 6 months before emptying the Critical bugs list. The new target is to reduce to a 100 for October. Let’s see if I was more realistic this time around.

And we also took the traditional team picture.

Launchpad Team Photo

Great fun it was, and I can’t wait for the next one in January.

Photo by Miss_Colleen. Licence: CC BY-NC-SA 2.0.

Read more
Matthew Revell

Squiggle

Bags of Tilda riceVisit this URL:

https://launchpad.net/~

Cool, eh?

Same works for https://bugs.launchpad.net/~, as well as code, translations, answers and blueprint.

Actually, it’ll work anywhere that you’d normally put /people/+me or your own Launchpad id — so long as you’re logged in.

For example: visit https://launchpad.net/~/+participation to see a list of all your team memberships.

Thanks to Martin Pool.

Photo based on an original by Michael Francis McCarthy. Licence: CC BY 2.0

Read more
Matthew Revell

Launchpad will be fully offline (not read-only) for around one hour from 08.00 UTC on the 14th July 2011.

Launchpad goes offline: 08.00 UTC 14th July 2011
Launchpad expected back: 09.00 UTC 14th July 2011

We’re sorry for the very short notice of this down-time. Following this work, Launchpad’s performance will be back to normal.

Read more
Matthew Revell

How to tell who has what role

So, you’re about to select someone in Launchpad — maybe you’re assigning a bug — and you’re not exactly sure you’ve got the right person. One way to confirm would be for Launchpad to tell you what relation that person has to the project you’re currently dealing with.

Sound like something that might help you?

If you’ve got 25 minutes today or tomorrow, you can help us choose the best way to show those affiliations. Sign up to let us know if you’re interested. There may even be an Amazon voucher in it for you :)

Read more
Matthew Revell

As of recently, you have more control of what email Launchpad sends you about bugs.

Instead of bug subscriptions being all or nothing, you can now tell Launchpad exactly what bug-related events you want to hear about. And if you find you’re getting too much, or too little, detail you can change your subscription at any time.

What’s more, it’s now easier to filter the bug mail that Launchpad sends you.

Subscribing to an individual bug

Let’s take a look at a bug report about Gwibber. Over on the right-hand side is the Subscribe link as usual. Clicking on it, though, gives us three new options for when Launchpad should email you about this bug:

  • a change is made to this bug or a new comment is added
  • any change is made to this bug, other than a new comment being added
  • this bug is fixed or re-opened.

Subscribe to a bug

You get to choose how much detail you want about the life of this bug, from receiving an email about everything through to Launchpad notifying you only when the bug is fixed or subsequently re-opened.

Editing a subscription to an individual bug

You can change your mind about your bug subscription. Simply click the Edit subscription link that appears once you’ve subscribed.

You get the same options, along with an additional option of unsubscribing entirely.

Edit a bug subscription

Subscribing to a project

You’ve been able to subscribe to all the bugs in a particular project or distribution (and milestone, series, packages within those) for a few years now. However, for anything but the quietest of projects this could bring a new intensity to the phrase “drinking from the fire hose”.

Now you can have sane subscriptions to these broader structures within Launchpad. Let’s subscribe to bugs in OpenStack to see how it works.

Clicking Subscribe to bug mail on OpenStack’s bug overview page gives allows you to:

  • subscribe yourself or one of your teams
  • name the subscription — this gets added to the emails that result
  • choose whether you want to hear when a bug is opened or closed, or if you want to go into more detail.

Subscribe to OpenStack

So, now you have a subscription called OpenStack overview that’ll generate an email whenever a bug report is opened or closed in OpenStack.

Refining a project subscription

Let’s say you’re particularly interested in OpenStack documentation. No problem, you can keep your OpenStack overview subscription for everything else and add a more detailed subscription just for documentation bugs.

Subscribe to a tag

If that’s not enough, you can also filter by importance and status.

Muting bugs

If a particular bug becomes too noisy, you can mute it. No matter how you’re subscribed to that bug — even if you have several subscriptions that generate email about that particular report — once you’ve muted it, you won’t hear about that bug again.

Mute bug mail

Of course, you can unmute it any time you like.

Unsubscribe in anger

At the bottom of every bug mail that Launchpad sends you’ll now find a link to edit your subscriptions to that bug. So, no matter how how you’re subscribed to the bug you’ll find all those subscription listed on that page and have the option to edit each of them.

You can also get to that page by clicking Edit bug mail on the bug page itself.

And there’s more

To help with filtering, Launchpad now adds a new header to bug emails:

X-Launchpad-Subscription

It quotes the name you gave the subscription when you created it. For Gmail users, all bug emails also feature the subscription name in the footer of the email body.

And don’t forget that you can already opt-out of email about your own actions on bugs and Launchpad won’t send email about quickly corrected mistakes.

I know I’m going to find bug mail easier to deal with now. Congratulations to the Yellow Squad on delivering this feature!

Read more