Canonical Voices

Posts tagged with 'django'

Gavin Panella

South, South 2, and Django Migrations

blockquote { background-color: #444; padding: 0.2em 1em; }

A couple of months ago we on the MAAS team found ourselves in a bit of a pickle: we needed to be able to support a product targeted at both Django <1.7 and Django ≥1.7 with database migrations. This is a problem because South is replaced by Django's own migration support in 1.7, and there are differences.

I emailed Andrew Godwin to ask his advice. He's the author of South 2and so apparently knows his stuff, but we also wondered if South 2 might be a way out of our mess. His reply confirmed him as knowledgable, kind, and helpful. Although he did not bless South 2 as our silver bullet, he did have some other useful advice instead.

I promised I would document our correspondence where others might learn from it, and this is it, somewhat overdue. I've edited it slightly for clarity.

Thanks Andrew!

Hi Andrew,

I found your south2 repository on GitHub today. It looks like you've not touched it in a while, but I wondered if I could ask you a few questions about it anyway? There's a lot of context but it boils down to two-ish questions:

  1. What would you recommend for transitioning a packaged product (i.e. one which we don't provide as a service) from South-based migrations to Django ≥1.7 migrations?

    As a general answer, I suggest the method described in the Django docs, which is to move the South migrations to a south_migrationsdirectory and generate new initial Django ones. As long as your users have South 1.0 or higher, that'll keep both versions running during a transition, and Django's automatic application of initial migrations makes things a lot easier. I don't recommend that you try and support both migration sets at the same time; make 1.7 or higher a hard dependency for a release. This obviously is a bit different for the case below, which I answered down there.

  2. How much work would be required to get south2 working?

    It was abandoned with good reason - it's around another two months of work to get it working remotely reliably, and I'm not sure it could be done at all without much more of a rewrite rather than the current source translation approach. I didn't abandon the idea lightly, but alas it just wasn't proving very stable.

We're in a tricky situation:

  • We have an application, MAAS, that we ship as a package in Ubuntu, i.e. end-users install it. It uses PostgreSQL.
  • It's supported in Ubuntu 14.04 (Trusty) and will be supported until April 2019. Trusty ships with Django 1.6, and this won't change (only security fixes and fixes for very serious bugs are back-ported).
  • Django 1.7 is now available in the development version of Ubuntu (Vivid).
  • Django 1.7 or later will be in the next LTS (Long Term Support) version of Ubuntu, out next year. (Trusty is the most recent LTS release.)
  • We have been using South for several years.
  • To support MAAS in Trusty we may need to back-port migrations from trunk. Once we base trunk on Django ≥1.7 we can't back-port directly; we'd need to recreate any migrations with South.
  • However, we also need a seamless upgrade path for users on Trusty when they upgrade to the next LTS release, where they can skip right over three intermediate releases of Ubuntu.
  • Between Trusty and the next LTS (hereafter just "Next"), the upgrade path might look like (where mXXX = "migration XXX"):

    Trusty -- m134 -- m135 -- m136 -- m137 (then EOL)
    \ \ \ \
    Next -- m0 ---- m1 ---- m2 ---- m3 ---- m4 ---- ...

    In other words, Trusty and the next LTS share a common ancestor in South migration 134; the Django ≥1.7 migration baseline is derived at that point.

    At any point after that a user could choose to upgrade to the next LTS. If they upgrade from an installation that's got m136, we could map that over to m2 in the new migrations model, tell Django to fake-apply m0, m1, and m2, then proceed from there.

  • In truth, a user could choose to upgrade from Trusty to Next beforehaving applied m134 because users can choose to follow only security fixes, and not updates. (They can choose to follow nothing at all, but that's getting into a very grey area w.r.t. support.)

    In this situation we'd want to apply all remaining South migrations up to at least m134 before switching over to the new Django migrations model.

    On the other hand, there may be a way to prevent a Trusty → Next upgrade based on a precondition, e.g. "m134 or greater is needed", but I don't currently know how that would be implemented.

  • There's a risk of South migrations not matching up to Django ≥1.7 migrations. That would most likely be an issue with our process, but it could be a software issue too.
  • With a variety of automated testing we can mitigate a lot of the process risk, and catch software issues early.
  • However, that all adds up to quite a lot of work.
  • Another option entirely would be for us to invest time into south2 and switch everything over to Django ≥1.7 migrations. That sounds like it would be a lot simpler, and thus carry a lot less risk.
  • The thing I don't know, which I hope you can answer, is how much work might it be to get south2 to a point where this would be possible? What would the ongoing maintenance look like?
  • What would you recommend?

    There's no clean solution, sorry. I'd document having to apply the most recent migrations before switching (and perhaps have a code entry on startup in the 1.7 dependent version that checks the south_migrations table directly and hard fails if you didn't), then have people clean switch over to the latest release.

    Can I ask why you won't just ship a newer version of Django with the newer releases of MAAS, even on Trusty? I know OS packaging is a tough thing to get around, but trying to backport migrations to work on South and older releases is only going to bring you pain (South is much more limited than Django migrations, and you might have to do a lot of manual workarounds).

    South2 isn't going to work - don't go down that path, I abandoned it for good reason, I'm not even sure the automated source translation approach is possible and a rewrite would take months. You're better off somehow shipping 1.7 bundled or as some kind of special dependency.

Read more
Michael Hall

I’ve been using Ubuntu on my only phone for over six months now, and I’ve been loving it. But all this time it’s been missing something, something I couldn’t quite put my finger on. Then, Saturday night, it finally hit me, it’s missing the community.

That’s not to say that the community isn’t involved in building it, all of the core apps have been community developed, as have several parts of our toolkit and even the platform itself. Everything about Ubuntu for phones is open source and open to the community.

But the community wasn’t on my phone. Their work was, but not the people.  I have Facebook and Google+ and Twitter, sure, but everybody is on those, and you have to either follow or friend people there to see anything from them. I wanted something that put the community of Ubuntu phone users, on my Ubuntu phone. So, I started to make one.

Community Cast

Community Cast is a very simple, very basic, public message broadcasting service for Ubuntu. It’s not instant messaging, or social networking. It doesn’t to chat rooms or groups. It isn’t secure, at all.  It does just one thing, it lets you send a short message to everybody else who uses it. It’s a place to say hello to other users of Ubuntu phone (or tablet).  That’s it, that’s all.

As I mentioned at the start, I only realized what I wanted Saturday night, but after spending just a few hours on it, I’ve managed to get a barely functional client and server, which I’m making available now to anybody who wants to help build it.


The server piece is a very small Django app, with a single BroadcastMessage data model, and the Django Rest Framework that allows you to list and post messages via JSON. To keep things simple, it doesn’t do any authentication yet, so it’s certainly not ready for any kind of production use.  I would like it to get Ubuntu One authentication information from the client, but I’m still working out how to do that.  I threw this very basic server up on our internal testing OpenStack cloud already, but it’s running the built-in http server and an sqlite3 database, so if it slows to a crawl or stops working don’t be surprised.  Like I said, it’s not production ready.  But if you want to help me get it there, you can get the code with bzr branch lp:~mhall119/+junk/communitycast-server, then just run syncdb and runserver to start it.


The client is just as simple and unfinished as the server (I’ve only put a few hours into them both combined, remember?), but it’s enough to use. Again there’s no authentication, so anybody with the client code can post to my server, but I want to use the Ubuntu Online Accounts to authenticate a user via their Ubuntu One account. There’s also no automatic updating, you have to press the refresh button in the toolbar to check for new messages. But it works. You can get the code for it with bzr branch lp:~mhall119/+junk/communitycast-client and it will by default connect to my test instance.  If you want to run your own server, you can change the baseUrl property on the MessageListModel to point to your local (or remote) server.


There isn’t much to show, but here’s what it looks like right now.  I hope that there’s enough interest from others to get some better designs for the client and help implementing them and filling out the rest of the features on both the client and server.


Not bad for a few hours of work.  I have a functional client and server, with the server even deployed to the cloud. Developing for Ubuntu is proving to be extremely fast and easy.


Read more
David Murphy (schwuk)

Today I was adding tox and Travis-CI support to a Django project, and I ran into a problem: our project doesn’t have a Of course I could have added one, but since by convention we don’t package our Django projects (Django applications are a different story) – instead we use virtualenv and pip requirements files – I wanted to see if I could make tox work without changing our project.

Turns out it is quite easy: just add the following three directives to your tox.ini.

In your [tox] section tell tox not to run

skipsdist = True

In your [testenv] section make tox install your requirements (see here for more details):

deps = -r{toxinidir}/dev-requirements.txt

Finally, also in your [testenv] section, tell tox how to run your tests:

commands = python test

Now you can run tox, and your tests should run!

For reference, here is a the complete (albeit minimal) tox.ini file I used:

envlist = py27
skipsdist = True

deps = -r{toxinidir}/dev-requirements.txt
setenv =
    PYTHONPATH = {toxinidir}:{toxinidir}
commands = python test

Read more
Michael Hall

Ubuntu API Website

For much of the past year I’ve been working on the Ubuntu API Website, a Django project for hosting all of the API documentation for the Ubuntu SDK, covering a variety of languages, toolkits and libraries.  It’s been a lot of work for just one person, to make it really awesome I’m going to need help from you guys and gals in the community.

To help smooth the onramp to getting started, here is a breakdown of the different components in the site and how they all fit together.  You should grab a copy of the branch from Launchpad so you can follow along by running: bzr branch lp:ubuntu-api-website


First off, let’s talk about the framework.  The API website uses Django, a very popular Python webapp framework that’s also used by other community-run Ubuntu websites, such as Summit and the LoCo Team Portal, which makes it a good fit. A Django project consists of one or more Django “apps”, which I will cover below.  Each app consists of “models”, which use the Django ORM (Object-Relational Mapping) to handle all of the database interactions for us, so we can stick to just Python and not worry about SQL.  Apps also have “views”, which are classes or functions that are called when a URL is requested.  Finally, Django provides a default templating engine that views can use to produce HTML.

If you’re not familiar with Django already, you should take the online Tutorial.  It only takes about an hour to go through it all, and by the end you’ll have learned all of the fundamental things about building a Django site.

Branch Root

When you first get the branch you’ll see one folder and a handful of files.  The folder, developer_network, is the Django project root, inside there is all of the source code for the website.  Most of your time is going to be spent in there.

Also in the branch root you’ll find some files that are used for managing the project itself. Most important of these is the README file, which gives step by step instructions for getting it running on your machine. You will want to follow these instructions before you start changing code. Among the instructions is using the requirements.txt file, also in the branch root, to setup a virtualenv environment.  Virtualenv lets you create a Python runtime specifically for this project, without it conflicting with your system-wide Python installation.

The other files you can ignore for now, they’re used for packaging and deploying the site, you won’t need them during development.


As I mentioned above, this folder is the Django project root.  It has sub-folders for each of the Django apps used by this project. I will go into more detail on each of these apps below.

This folder also contains three important files for Django:, and is used for a number of commands you can give to Django.  In the README you’ll have seen it used to call syncdbmigrate and initdb.  These create the database tables, apply any table schema changes, and load them with initial data. These commands only need to be run once.  It also has you run collectstatic and runserver. The first collects static files (images, css, javascript, etc) from all of the apps and puts them all into a single ./static/ folder in the project root, you’ll need to run that whenever you change one of those files in an app.  The second, runserver, runs a local HTTP server for your app, this is very handy during development when you don’t want to be bothered with a full Apache server. You can run this anytime you want to see your site “live”. contains all of the Django configuration for the project.  There’s too much to go into detail on here, and you’ll rarely need to touch it anyway. is the file that maps URLs to an application’s views, it’s basically a list of regular-expressions that try to match the requested URL, and a python function or class to call for that match. If you took the Django project tutorial I recommended above, you should have a pretty good understanding of what it does. If you ever add a new view, you’ll need to add a corresponding line to this file in order for Django to know about it. If you want to know what view handles a given URL, you can just look it up here.


If you followed the README in the branch root, the first thing it has you do is grab another bzr branch and put it in ./developer_network/ubuntu_website.  This is a Django app that does nothing more than provide a base template for all of your project’s pages. It’s generic enough to be used by other Django-powered websites, so it’s kept in a separate branch that each one can pull from.  It’s rare that you’ll need to make changes in here, but if you do just remember that you need to push you changes branch to the ubuntu-community-webthemes project on Launchpad.


This is a 3rd party Django app that provides the RESTful JSON API for the site. You should not make changes to this app, since that would put us out of sync with the upstream code, and would make it difficult to pull in updates from them in the future.  All of the code specific to the Ubuntu API Website’s services are in the developer_network/service/ app.


This app isn’t being used yet, but it is intended for giving better search functionality to the site. There are some models here already, but nothing that is being used.  So if searching is your thing, this is the app you’ll want to work in.


This is another app that isn’t being used yet, but is intended to allow users to link additional content to the API documentation. This is one of the major goals of the site, and a relatively easy area to get started contributing. There are already models defined for code snippets, Images and links. Snippets and Links should be relatively straightforward to implement. Images will be a little harder, because the site runs on multiple instances in the cloud, and each instance will need access to the image, so we can’t just use the Django default of saving them to local files. This is the best place for you to make an impact on the site.


The common app provides views for logging in and out of the app, as well as views for handling 404 and 500 errors when the arise.  It also provides some base models the site’s page hierarchy. This starts with a Topic at the top, which would be qml or html5 in our site, followed by a Version which lets us host different sets of docs for the different supported releases of Ubuntu. Finally each set of docs is placed within a Section, such as Graphical Interface or Platform Service to help the user browse them based on use.


This app provides models that correspond directly to pieces of documentation that are being imported.  Documentation can be imported either as an Element that represents a specific part of the API, such as a class or function, or as a Page that represents long-form text on how to use the Elements themselves.  Each one of these may also have a given Namespace attached to it, if the imported language supports it, to further categorize them.


Finally we get into the app that is actually generates the pages.  This app has no models, but uses the ones defined in the common and apidocs apps.  This app defines all of the views and templates used by the website’s pages, so no matter what you are working on there’s a good chance you’ll need to make changes in here too. The templates defined here use the ones in ubuntu_website as a base, and then add site and page specific markup for each.

Getting Started

If you’re still reading this far down, congratulations! You have all the information you need to dive in and start turning a boring but functional website into a dynamic, collaborative information hub for Ubuntu app developers. But you don’t need to go it alone, I’m on IRC all the time, so come find me (mhall119) in #ubuntu-website or #ubuntu-app-devel on Freenode and let me know where you want to start. If you don’t do IRC, leave a comment below and I’ll respond to it. And of course you can find the project, file bugs (or pick bugs to fix) and get the code all from the Launchpad project.

Read more
Michael Hall

Quick overview post today, because it’s late and I don’t have anything particular to talk about today.

First of all, the next vUDS was announced today, we’re a bit late in starting it off but we wanted to have another one early enough to still be useful to the Trusty release cycle.  Read the linked mailinglist post for details about where to find the schedule and how to propose sessions.

I pushed another update to the API website today that does a better job balancing the 2-column view of namespaces and fixes the sub-nav text to match the WordPress side of things. This was the first deployment in a while to go off without a problem, thanks to  having a new staging environment created last time.  I’m hoping my deployment problems on this are now far behind me.

I took a task during my weekly Core Apps update call to look more into the Terminal app’s problem with enter and backspace keys, so I may be pinging some of you in the coming week about it to get some help.  You have been warned.

Finally, I decided a few weeks ago to spread out my after-hours community a activity beyond Ubuntu, and I’ve settled on the Debian new maintainers Django website as somewhere I can easily start.  I’ve got a git repo where I’m starting writing the first unit tests for that website, and as part of that I’m also working on Debian packaging for the Python model-mommy library which we use extensively in Ubuntu’s Django website. I’m having to learn (or learn more) Debian packaging, Git workflows and Debian’s processes and community, all of which are going to be good for me, and I’m looking forward to the challenge.

Read more
Michael Hall

We wrapped up the last day of our sprint with a new team photo.  I can honestly say I couldn’t think of a better group of people to be working with.  Even the funny looking guy in the middle.

I mentioned that earlier in the week we decided on naming SDK releases after distro releases, and with that information in hand I spent my last day getting the latest API docs uploaded, so if you’re writing apps for the latest device images, you’ll want to use these:

In the coming week I’ll be working to get the documentation publishing scripts added to the automated build and testing process, so those docs will be continuously updated until the release of Ubuntu 14.04, at which point we’ll freeze those doc pages and start publishing daily updates for 14.10.  Being able to publish  all of those docs in a matter of minutes was a particularly thrill for me, after working for so long to get that feature into production.  It certainly proves that it was the right approach.

Read more
Michael Hall

Last week I posted on G+ about the a couple of new sets of QML API docs that were published.  Well that was only a part of the actual story of what’s been going on with the Ubuntu API website lately.

Over the last month I’ve been working on implementing and deploying a RESTful JSON service on top of the Ubuntu API website, and last week is when all of that work finally found it’s way into production.  That means we now have a public, open API for accessing all of the information available on the API website itself!  This opens up many interesting opportunities for integration and mashups, from integration with QtCreator in the Ubuntu SDK, to mobile reference apps to run on the Ubuntu phone, or anything else your imagination can come up with.

But what does this have to do with the new published docs?  Well the RESTful service also gives us the ability to push documentation up to the production server, which is how those docs got there.  I’ve been converting the old Django scripts that would import docs directly into the database, to instead push them to the website via the new service, and the QtMultimedia and QtFeedback API docs were the first ones to use it.

Best of all, the scripts are all automated, which means we can start integrating them with the continuous integration infrastructure that the rest of Ubuntu Engineering has been building around our projects.  So in the near future, whenever there is a new daily build of the Ubuntu SDK, it will also push the new documentation up, so we will have both the stable release documentation as well as the daily development release documentation available online.

I don’t have any docs yet on how to use the new service, but you can go to to see what URLs are available for the different data types.  You can also append ?<field>=<value> keyword filters to your URL to narrow the results.  For example, if you wanted all of the Elements in the Ubuntu.Components namespace, you can use to do that.

That’s it for today, the first day of my UbBloPoMo posts.  The rest of this week I will be driving to and fro for a work sprint with the rest of my team, the Ubuntu SDK team, and many others involved in building the phone and app developer pieces for Ubuntu.  So the rest of this week’s post may be much shorter.  We’ll see.

Happy Hacking.

Read more

A number of times over the past few years I’ve needed to create some quite complex migrations (both schema and data) in a few of the Django apps that I help out with at Canonical. And like any TDD fanboy, I cry at the thought of deploying code that I’ve just tested by running it a few times with my own sample data (or writing code without first setting failing tests demoing the expected outcome).

This migration test case helper has enabled me to develop migrations test first:

class MigrationTestCase(TransactionTestCase):
    """A Test case for testing migrations."""

    # These must be defined by subclasses.
    start_migration = None
    dest_migration = None
    django_application = None

    def setUp(self):
        super(MigrationTestCase, self).setUp()
        migrations = Migrations(self.django_application)
        self.start_orm = migrations[self.start_migration].orm()
        self.dest_orm = migrations[self.dest_migration].orm()

        # Ensure the migration history is up-to-date with a fake migration.
        # The other option would be to use the south setting for these tests
        # so that the migrations are used to setup the test db.
        call_command('migrate', self.django_application, fake=True,
        # Then migrate back to the start migration.
        call_command('migrate', self.django_application, self.start_migration,

    def tearDown(self):
        # Leave the db in the final state so that the test runner doesn't
        # error when truncating the database.
        call_command('migrate', self.django_application, verbosity=0)

    def migrate_to_dest(self):
        call_command('migrate', self.django_application, self.dest_migration,

It’s not perfect – schema tests in particular end up being quite complicated as you need to ensure you’re working with the correct orm model when creating your test data – and you can’t use the normal factories to create your test data. But it does enable you to write migration tests like:

class MyMigrationTestCase(MigrationTestCase):

    start_migration = '0022_previous_migration'
    dest_migration = '0024_data_migration_after_0023_which_would_be_schema_changes'
    django_application = 'myapp'

    def test_schema_and_data_updated(self):
        # Test setup code


        # Assertions

which keeps me happy. When I wrote that I couldn’t find any other suggestions out there for testing migrations. A quick search now turns up one idea from André (data-migrations only),  but nothing else substantial. Let me know if you’ve seen something similar or a way to improve testing of migrations.

Filed under: django, python, testing

Read more

A number of times over the past few years I’ve needed to create some quite complex migrations (both schema and data) in a few of the Django apps that I help out with at Canonical. And like any TDD fanboy, I cry at the thought of deploying code that I’ve just tested by running it a few times with my own sample data (or writing code without first setting failing tests demoing the expected outcome).

This migration test case helper has enabled me to develop migrations test first:

class MigrationTestCase(TransactionTestCase):
    """A Test case for testing migrations."""

    # These must be defined by subclasses.
    start_migration = None
    dest_migration = None
    django_application = None

    def setUp(self):
        super(MigrationTestCase, self).setUp()
        migrations = Migrations(self.django_application)
        self.start_orm = migrations[self.start_migration].orm()
        self.dest_orm = migrations[self.dest_migration].orm()

        # Ensure the migration history is up-to-date with a fake migration.
        # The other option would be to use the south setting for these tests
        # so that the migrations are used to setup the test db.
        call_command('migrate', self.django_application, fake=True,
        # Then migrate back to the start migration.
        call_command('migrate', self.django_application, self.start_migration,

    def tearDown(self):
        # Leave the db in the final state so that the test runner doesn't
        # error when truncating the database.
        call_command('migrate', self.django_application, verbosity=0)

    def migrate_to_dest(self):
        call_command('migrate', self.django_application, self.dest_migration,

It’s not perfect – schema tests in particular end up being quite complicated as you need to ensure you’re working with the correct orm model when creating your test data – and you can’t use the normal factories to create your test data. But it does enable you to write migration tests like:

class MyMigrationTestCase(MigrationTestCase):

    start_migration = '0022_previous_migration'
    dest_migration = '0024_data_migration_after_0023_which_would_be_schema_changes'
    django_application = 'myapp'

    def test_schema_and_data_updated(self):
        # Test setup code


        # Assertions

which keeps me happy. When I wrote that I couldn’t find any other suggestions out there for testing migrations. A quick search now turns up one idea from André (data-migrations only),  but nothing else substantial. Let me know if you’ve seen something similar or a way to improve testing of migrations.

Filed under: django, python, testing

Read more
Michael Hall

Back in San Francisco, during UDS-Q, we had a discussion about the need for better online documentation for the various APIs that application developers use to write apps for Ubuntu.  The Ubuntu App Showdown and subsequent AppDevUploadProcess spec work has consumed most of my time since then, but I was able to start putting together a spec for such a site.  The App Showdown feedback we got from our developers survey highlighted the need, as lack of good Gtk documentation for Python was one of the most common problems people experienced, giving it a little more urgency.

Fortunately, Alberto Ruiz was at UDS, and told me about a project he had started for Gnome called Gnome Developer Network (GDN for short).  Alberto had already done quite a bit of work on the database models and GObject Instropection parsing needed to populate it.  The plan is to use GDN as the database and import process, and build a user-friendly web interface on top of that, linking in external resources like tutorials and AskUbuntu questions, as well as user submitted comments and code snippets.

Now that the spec is (mostly) done, we need to get together some developers who can implement it.  There will be a lot of front-end work (mostly HTML, CSS and Javascript), but also enough backend work (Python and Django) to keep anybody occupied.  I’ve created a Launchpad project for the site, and a team you can join if you’re interested in helping out.

The GDN code and some very basic template are already available. You can get the code from bzr with bzr branch lp:ubuntu-api-website and following the instructions in the DEVELOPMENT file.  I’ll also be running a live App Developer Q&A Session at 1700 UTC today (September 19th), and would be happy to help anybody get the code up and running during that time.

Read more
Michael Hall

Sweet Chorus

Juju is revolutionizing the way web services are deployed in the cloud, taking what was either a labor-intensive manual task, or a very labor-intensive re-invention of the wheel  (or deployment automation in this case), and distilling it into a collection of reusable components called “Charms” that let anybody deploy multiple inter-connected services in the cloud with ease.

There are currently 84 Juju charms written for everything from game backends to WordPress sites, with databases and cache servers that work with them.  Charms are great when you can deploy the same service the same way, regardless of it’s intended use.  Wordpress is a good use case, since the process of deploying WordPress is going to be the same from one blog to the next.

Django’s Blues

But when you go a little lower in the stack, to web frameworks, it’s not quite so simple.  Take Django, for instance.  While much of the process of deploying a Django service will be the same, there is going to be a lot that is specific to the project.  A Django site can have any number of dependencies, both common additions like South and Celery, as well as many custom modules.  It might use MySQL, or PostgreSQL, or Oracle (even SQLite for development and testing).  Still more things will depend on the development process, while WordPress is available in a DEB package, or a tarball from the upstream site, a Django project may be anywhere, and most frequently in a source control branch specific to that project.  All of this makes writing a single Django charm nearly impossible.

There have been some attempts at making a generic, reusable Django charm.  Michael Nelson made one that uses Puppet and a custom config.yaml for each project.  While this works, it has two drawbacks: 1) It requires Puppet, which isn’t natural for a Python project, and 2) It required so many options in the config.yaml that you still had to do a lot by hand to make it work.  The first of these was done because ISD (where Michael was at the time) was using Puppet to deploy and configure their Django services, and could easily have been done another way.  The second, however, is the necessary consequence of trying to make a reusable Django charm.

Just for Fun

Given the problems detailed above, and not liking the idea of making config options for every possible variation of a Django project, I recently took a different approach.  Instead of making one Django Charm to rule them all, I wrote a small Django App that would generate a customized Charm for any given project.  My goal is to gather enough information from the project and it’s environment to produce a charm that is very nearly complete for that project.  I named this charming code “Naguine” after Django Reinhardt’s second wife, Sophie “Naguine” Ziegler.  It seemed fitting, since this project would be charming Django webapps.

Naguine is very much a JFDI project, so it’s not highly architected or even internally consistent at this point, but with a little bit of hacking I was able to get a significant return. For starters, using Naguine is about as simple as can be, you simply install it on your PYTHONPATH and run:

python charm --settings naguine

The –settings naguine will inject the naguine django app into your INSTALLED_APPS, which makes the charm command available.

This Kind of Friend

The charm command makes use of your Django settings to learn about your other INSTALLED_APPS as well as your database settings.  It will also look for a requirements.txt and, inspecting each to learn more about your project’s dependencies.  From there it will try to locate system packages that will provide those dependencies and add them to the install hook in the Juju  charm.

The charm command also looks to see if your project is currently in a bzr branch, and if it is it will use the remote branch to pull down your  project’s code during the install.  In  the future I hope to also support git and hg deployments.

Finally the command will write hooks for linking to a database instance on another server, including running syncdb to create the tables for your models, adding a superuser account with a randomly generated password and, if you are using South, running any migration scripts as well. It also writes some metadata about your charm and a short README explaining how to use it.

All that is left for you to do is review the generated charm, manually add any dependencies Naguine couldn’t find a matching package for, and manually add any install or database initialization that is specific to your project.  The amount of custom work needed to get a charm working is extremely minor, even for moderately complex projects.

Are you in the Mood

To try Naguine with your Django project, use the following steps:

  1. cd to your django project root (where your is)
  2. bzr branch lp:naguine
  3. python charm –settings naguine

That’s all you need.  If your django project lives in a bzr branch, and if it normally uses, you should have a directory called ./charms/precise/ that contains an almost working Juju charm for your project.

I’ve only tested this on a few Django project, all of which followed the same general conventions when it came to development, so don’t be surprised if you run into problems.  This is still a very early-stage project after all.  But you already have the code (if you followed step #2 above), so you can poke around and try to get it working or working better for your project.  Then submit your changes back to me on Launchpad, and I’ll merge them in.  You can also find me on IRC (mhall119 on freenode) if you get stuck and I will help you get it working.

(For those who are interested, each of the headers in this post is the name of a Django Reinhardt song)

Read more
Michael Hall

Back when I first started writing Unity lenses, I lamented the complexity required to get even the most basic Lens written and running.  I wrote in that post about wanting to hide all of that unnecessary complexity.  Well now I am happy to announce the first step towards that end: Singlet.

In optics, a “singlet” is a very simple lens.  Likewise, the Singlet project aims to produce simple Unity lenses.  Singlet targets opportunistic programmers who want to get search results into Unity with the least amount of work.  By providing a handful of Python meta classes and base classes, Singlet lets you write a basic lens with a minimal amount of fuss. It hides all of the boilerplate code necessary to interface with GObject and DBus, leaving the developer free to focus solely on the purpose of their lens.  With Singlet, the only thing a Lens author really needs to provide is a single search function.

Writing a Singlet

So what does a Singlet Lens look like?  Here is a sample of the most basic lens, which produced the screenshot above:

#! /usr/bin/python

from singlet.lens import SingleScopeLens, IconViewCategory, ListViewCategory
from singlet.utils import run_lens

class TestLens(SingleScopeLens):

    class Meta:
        name = 'test'

    cat1 = IconViewCategory("Cat One", "stock_yet")

    cat2 = ListViewCategory("Cat Two", "hint")

    def search(self, phrase, results):
        results.append('' % phrase,
                             phrase, phrase, '')

        results.append('' % phrase,
                             phrase, phrase, '')

if __name__ == "__main__":
    import sys
    run_lens(TestLens, sys.argv)

As you can see, there isn’t much to it.  SingleScopeLens is the first base class provided by Singlet.  It creates an inner-scope for you, and connects it to the DBus events for handling a user’s search events.  The three things you need to do, as a Lens author, is give it a name in the Meta  class, define at least one Category, and most importantly implement your custom search(self, phrase, results) method.

Going Meta

Django developers will notice a similarity between Singlet Lenses and Django Models in their use of an inner Meta class.  In fact, they work exactly the same way, though with different properties.  At a minimum, you will need to provide a name for your lens.  Everything else can either use default values, or will be extrapolated from the name.  Everything in your Meta class, plus defaults and generated values, will be accessible in <your_class>._meta later on.

Categorically easy

Again borrowing from Django Models, you add categories to your Singlet Lens by defining it in the Class’s scope itself, rather than in the  __init__ method.  One thing that I didn’t like about Categories when writing my previous lenses was that I couldn’t reference them when adding search results to the result model.  Instead you have to give the numeric index of the category.  In Singlet, the variable name you used when defining the category is converted to the numeric index for that category, so you easily reference it again when building your search results.  But don’t worry, you category objects are still available to you in <your_class>._meta.categories if you want them.

The search is on

The core functionality of a Lens is the search.  So it makes sense that the majority of your work should happen here.  Singlet will call your search method, passing in the current search phrase and an empty results model.  From there, it’s up to you to collect data from whatever source you are targeting, and start populating that results model.

You can handle the URI

Unity knows how to handle common URIs in your results, such as file:// and http:// uris.  But often times your lens isn’t going to be dealing with results that map directly to a file or website.  For those cases, you need to hook into DBus again to handle the URI of a selected result item, and return a specifically constructed GObject response.  With Singlet, all you need to do is define a handle_uri method on your Lens, and it will take care of hooking it into DBus for you.  Singlet also provides a couple of helper methods for your return value, either hide_dash_response to hide the dash after you’ve handled the URI, or update_dash_response if you want to leave it open.

Make it go

Once you’ve defined your lens, you need to be able to initialize it and run it, again using a combination of DBus and GObject.  Singlet hides all of this behind the run_lens function in singlet.utils, which you should call at the bottom of your lens file as shown in the above snippet.

Lord of the files

There’s more to getting your Lens working that just the code, you also need to specify a .lens file describing your lens to Unity, and a .service file telling dbus about it.  Singlet helps you out here too, by providing command line helpers for generating and installing these files.  Suppose the code snippet above was in a file called, once it’s written you can run “python make” and it will write test.lens and unity-test-lens.service into your current directory.  The data in these files comes from <your_lens>._meta, including the name, dbus information, description and icon.  After running make you can run “sudo python install”, this will copy your code and config files to where they need to be for Unity to pick them up and run them.

More to come

You can get the current Singlet code by branching lp:singlet.  I will be working on getting it built and available via PyPi and a PPA in the near future, but for now just having it on your PYTHONPATH is enough to get started using it.  Just be aware that if you make/install a Singlet lens, you need to make the Singlet package available on Unity’s PYTHONPATH as well or it won’t be able to run.  I’ve already converted my Dictionary Lens to use Singlet, and will work on others while I grow the collection of Singlet base classes.  If anybody has a common yet simple use case they would like me to target, please leave a description in the comments.

Read more

After experimenting with juju and puppet the other week, I wanted to see if it was possible to create a generic juju charm for deploying any Django apps using Apache+mod_wsgi together with puppet manifests wherever possible. The resulting apache-django-wsgi charm is ready to demo (thanks to lots of support from the #juju team), but still needs a few more configuration options. The charm currently:

  1. Enables the user to specify a branch of a Python package containing the Django app/project for deploy. This python package will be `python install`’d on the instance, but it also
  2. Enables you to configure extra debian packages to be installed first so that your requirements can be installed in a more reliable/trusted manner, along with the standard required packages (apache2, libapache2-mod-wsgi etc.). Here’s the example charm config used for,
  3. Creates a django.wsgi and httpd.conf ready to serve your app, automatically collecting all the static content of your installed Django apps to be served separately from the same Apache virtual host,
  4. When it receives a database relation change, it creates some local settings, overriding the database settings of your branch, sync’s and migrates the database (a noop if it’s the second unit) and restarts apache (See the database_settings.pp manifest for more details).

Here’s a quick demo which puts up a postgresql unit and two app servers with these commands:

$ juju deploy --repository ~/charms local:postgresql
$ juju deploy --config ubuntu-app-dir.yaml --repository ~/apache-django-wsgi/ local:apache-django-wsgi
$ juju add-relation postgresql:db apache-django-wsgi
$ juju add-unit apache-django-wsgi

Things that I think need to be improved or I’m uncertain about:

  1. `gem install puppet-module` is included in the install hook (a 3rd way of installing something on the system :/). I wanted to use the vcsrepo puppet module to define bzr resource types and puppet-module-tool seems to be the way to install 3rd-party puppet modules. Using this resource-type enables a simple initial_state.pp manifest. Of course, it’d be great to have ‘necessary’ tools like that in the archive instead.
  2. The initial_state.pp manifest pulls the django app package to /home/ubuntu/django-app-branch and then pip installs it on the system. Requiring the app to be a valid python package seemed sensible (in terms of ensuring it is correctly installed with its requirements satisfied) while still allowing the user to go one step further if they like and provide a debian package instead of a python package in a branch (which I assume we would do ultimately for production deploys?)
  3. Currently it’s just a very simple apache setup. I think ideally the static file serving should be done by a separate unit in the charm (ie. an instance running a stripped down apache2 or lighttpd). Also, I would have liked to have used an ‘official’ or ‘blessed’ puppet apache module to benefit from someone else’s experience, but I couldn’t see one that stood out as such.
  4. Currently the charm assumes that your project contains the configuration info (ie. a, etc.), of which the database settings can be simply overridden for deploy. There should be an additional option to specify a configuration branch (and it shouldn’t assume that you’re using django-configglue), as well as other options like django_debug, static_url etc.
  5. The charm should also export an interface (?) that can be used by a load balancer charm.

Filed under: django, juju

Read more
Michael Hall

In a previous post a commentator was explaining his typical web-stack deployment, and boasting about how he “can roll this out on Debian in less than 4 hours”.  Now he’s talking about provisioning, installing the OS, installing the services, and configuring everything.  That’s easily a day’s work for a capable sysadmin in a corporate environment. At least, it is in my experience.  So 4 hours sounds pretty good, doesn’t it?  I tell you what, go find your nearest admin and ask them how long it would take for them to get a new WordPress site up and ready for you to start posting to.  No really, I mean it, go ask.  I’ll wait.

Back?  Good.  I’m betting that, for many of you, the answer was a day or less, depending on their backlog of work.  A lucky few would be able to get their new site in an hour or two.  Not bad. But if you were using Ubuntu and cloud technology, you would have had your new site ready in the time that it took you to get that answer.

You think I’m exaggerating?  You can get a new world-accessible WordPress site up and running in less than 10 minutes.  That’s not marketing hype, Canonical is quite literally putting it’s money where it’s mouth is, by paying for an hour of Ubuntu Server on Amazon’s cloud service.  Go to and follow along, and you’ll have a WordPress site up and running in less time than it’ll take you to finish reading this article.

Welcome to the Try Ubuntu project.  Yes, we really are footing the bill for this, it won’t cost you a dime.  Click that large inviting “Let’s go to the cloud” button to start your adventure in cloud computing.

You will need to sign into Ubuntu SSO, because while we’re perfectly happy to pay for one hour, we do need to prevent abuse.  Requiring a login lets us limit this to once per user.  If you don’t have an SSO account you can create one now, and really if you’re going to spend any time at all with Ubuntu or the community, you’re going to want one soon or later anyway.

Ubuntu SSO lets you decide what detailed information to send back to any requesting web service.  You don’t have to send any of these details back to use this service, but if you have a Launchpad profile and uploaded SSH keys, you’ll get a better experience if you send at least your username.

Next you’ll get to choose what you want running on your trial instance.  You’ll also have to agree not to be abusive with your instance, do anything illegal, or generally cause other people problems as a result of our generosity.  Seriously, just don’t do it.

Currently we offer a base Ubuntu Server, running just the default installation, as well as the base server plus WordPress, Drupal or MoinMoin.  These aren’t pre-made images, they’ll be installed after the new server is provisioned, just like you would do manually, only thanks to cloud-init, we have it all automated.  You will use the same Ubuntu Server AMI regardless of which service you choose.

That’s all you need to do!  Now click the “Launch” button and the website will ask for a new m1.small instance through Amazon’s EC2 API.  This API is available to anyone, by the way, so you can script your own cloud deployments in exactly the same way.  We are using the boto python library to access the API from within Django.

After we get an instance reserved, we have to wait for Amazon to start it.  It only takes about a minute for Amazon to start up your new instance. In the mean time, this page will periodically refresh itself, and will let you know once your instance has started.  Did you notice that “View Cloud-Config” link?

This is what will be run on your new instance as soon as it’s ready.  You can copy this script, and use it later to start up your own permanent Ubuntu Server instances on AWS or any other EC2 compatible cloud host.  It is this script that will install Apache, MySQL and WordPress, configure them all properly, and get them all running.

Now your instance has started, and so has your countdown clock. At this point Ubuntu is fully booted, and your cloud-init script is busy installing all the packages and dependencies needed to run WordPress, all you have to do is sit back and wait.  Deploying is hard work huh?.  Don’t worry, I won’t tell your boss.

Hope you didn’t get too comfy, because 3 minutes later and we’re done!  Here you’ve given the command to SSH into your new instance.  If you’re running this from Windows, you’ll need to get something like PuTTY, because last time I checked Redmond still thinks that unencrypted Telnet is a good idea.  Trust me, you want SSH.

So fire up a terminal (yes, a terminal, you’re in server-land now buddy) and copy/paste the ssh command to get connected to your new instance.

Next you will be prompted for your password, it’s back on the web page, just copy/paste it into the terminal.  Remember how I said earlier that having a Launchpad profile with uploaded SSH keys would give a better experience?  Well if you’ve got all that, then you won’t see this step.  You see, part of the initialization that happened when your new instance started was to download your public SSH keys from Launchpad, allowing you to use your private SSH keys for authentication.  Nice huh?

So now you’re connected to your new instance.  But what’s all that stuff at the bottom of your terminal now?  That my friend, is Byobu, Ubuntu’s highly customized profile for GNU Screen.   Describing all of it’s wonderful goodness would take another full blog post, so I’ll just point out the highlights.  F2 creates a new “tab”, and you can switch between them with F3/F4.  Down at the bottom are some system monitoring widgets for things like load and memory usage.  In green characters is a special widget for Amazon EC2 that gives you an estimate of how much your instance is costing you (a whopping $0.09 USD at this point), and in blue characters is a clock showing how long your trial has been running (so you’ll know how much of your 55 minutes you have used).

Alright, your instance is running and configured, now what?  Well, did you  see that link on the web page?

That one, next to “Try going to”.  Click on it.

Well look at that, it’s your new WordPress site, just waiting for you to give it a name (and also username and password).  No unzipping, copying, apache configs, database setup, nothing.  We literally can’t make it any easier.

So pick a name, give it a password and email, and you’re all set!  Yes, I know my password was weak.  Actually my password was ‘password’.  Hey, it was only up for 55 minutes, I’m not going to spend extra time thinking of a secure password.  Come on, we’ve got a WordPress site to play with!


There you are, your new WordPress site is deployed.  And how are we doing on time?  Well if you hadn’t spent so much time reading along with this article, you’d have at least 45 minutes left out of the original 55.  Heck, I wasted a bunch of time taking screenshots along the way, and I still had more than 40 remaining.  How far do you think your sysadmin would have gotten in this same amount of time?  He probably just got back form refilling his coffee (which, to his credit, really is necessary before attempting a deployment the old fashioned way).

After a while, as you get close to the end of your trial period, you’ll get these helpful messages in your terminal session, letting you know how much longer you have.  And before you start thinking that you can use your mighty sudo powers to stop your instance’s termination, sorry pal, but we keep track of them on our end too, and your instance will be killed through the same EC2 API that launched it.  But I sure hope you had fun.


So now you’ve seen how fast and easy it is to deploy not just Ubuntu Server in the cloud, but actual, useful services running on top of it.  We offer you three popular software packages for websites, but those are only the tip of the iceberg.  You can write cloud-init scripts for anything you want to deploy on Ubuntu, even your own in-house build applications.  Then you too can deploy into the cloud with the click of a button.

What’s that you say?  You don’t have a handy dandy webapp for one-click deployments into the cloud?  Oh but you do!  You see, everything you just saw is open source, you can download it from lp:awstrial on Launchpad.  Use it to run your own trials, or just to learn how we did it so you can write your own internal provisioning service.  It defaults to using Amazon’s EC2 cloud, but you can point it to any EC2 compatible cloud.  We ran it internally against an OpenStack EC2 cloud during development and testing.

Did you enjoy your trial?  Leave us some feedback on what you liked, what you didn’t, and what you want us to offer in the future.

Read more
Michael Hall

We’re less than a month away from the start of UDS-P, which means we’re winding down the development pace on the Summit project. It’s been a very, very busy 6 months for us, we’ve done more work on Summit this cycle than I think it has ever seen before.  As I mentioned in previous posts, our main focus this cycle has been on stabilizing both the code, and the development process, and I think we’ve done an excellent job of meeting those goals.

During this cycle we have developed an easy to reproduce development environment that allows new contributors to get started hacking on Summit much faster than the could previously.  At the same time we’ve implemented stricter code reviews, requiring accompanying test cases in most instances, and have configured Tarmac to help us keep approved branches landing without delay.  We have also handed the task of maintaining our production environment and production deployments to Canonical’s IS team so that Summit will be managed by the same professional team as any other Ubuntu website.  We have also added 3 new contributors during this cycle, and hope to add more even more in the next.

From the end of UDS-O to the time of this writing, the Summit developers have closed 37 bugs, landed 89 merge proposals, and added 118 test cases (which is 118 more than we had before).  I can’t even begin to say how proud I am of the team of developers that have contributed to this project, and the amazing results that we have achieved in so short a time.  It is even more incredible because this is truly a community project, we all contribute to it in our spare time.  So thank you to everybody who has contributed to the success of the Summit project this cycle.

Below is the full list of bugs, branch merges and test cases from this cycle.


.Bugs TD { spacing: 0px; padding: 2px; border: #000000 1px solid; border-collapse: collapse; } .Bugs .Critical { background-color: #FFDDDD; } .Bugs .High { background-color: #FFFFDD; } .Bugs .Medium { background-color: #DDFFDD; } .Bugs .Low { background-color: #EEEEEE; } .Bugs .Wishlist { background-color: #DDDDFF; }
Bug # Title Priority
855826 session slugs containing + have broken etherpad links Critical
854709 URL encoding Launchpad links breaks them Critical
849078 Stop displaying the track name in the room name when there is a single track Medium
781693 Rooms: Add boolean for if a room has dial-in Medium
793018 Pull the summary from the launchpad blueprint and push it out via the iCal to Guidebook High
779833 Automatically clear cache when the data it contains changes Critical
853991 Plus sign in meeting name breaks url lookup Critical
815196 meeting import failed for lp update High
777171 Percent signs in the wiki field break summit Critical
766392 Pull Real Names from LP for use as “crew” Wishlist
765031 Support for private rooms and private meetings in those rooms Wishlist
793019 Make the colors for the track a database field instead of in the css High
647131 Don’t depend on a room being declared “plenary” Critical
849331 Needs to send no-cache headers when requesting +temp-meeting-export High
780342 Logging in next= is broken again Medium
835955 Sanitize input! Undecided
779884 autoscheduler should never schedule sessions at times in the past Critical
831311 Internal error when trying to delete a duplicate sponsoree High
814375 meeting link breaks on non-unique or missing meeting name Critical
813531 logo should point to High
829529 Summit barfs if lp id doesn’t exist and you submit it for sponsorship. High
793021 Add a today link to the topnav Medium
798826 Name fields throw confusing error Low
781137 Need more space between QR code and Day/Room name Low
781117 Change /today to be /xx/today Low
780969 Enable 404 page instead of showing a debug Low
779769 Remove ‘Attendees’ from meeting page if it is a plenary Low
783291 Brainstorm should be removed from summit Medium
668542 Don’t reschedule events/days that have already happened Critical
790675 Stop screen scraping launchpad for information. Use the API instead. High
782062 When importing from Launchpad, the blueprint name should be cleaned before being used High
665589 Importing blueprints unreliable High
798822 initslots doesn’t give feedback when done Low
793020 Match and main-nav Medium
783030 Change to room.title on the next sessions page Low
783029 Add link to meeting page in iCal Wishlist
664879 “previous day” and “next day” links on schedule would be nice Low


Branch Merges

193: Michael Hall 2011-10-02 [merge] [r=james-w] Optimizations to reduce the number of database queries on the summit and schedule pages.
192: James Westby 2011-09-22 [merge] [r=mhall119] Apply the same transform as etherpad to meeting names when generating pad urls.
191: Michael Hall 2011-09-22 [merge] [r=mhall119] Allow linaro tracks to be scheduled in adjacent slots in the same room.
190: James Westby 2011-09-21 [merge] [r=mhall119] Don’t escape URLs before putting them in the HTML.
189: Michael Hall 2011-09-21 [merge] [r=james-w] Change user_private_ical to use Schedule.from_request, add test case for private ical.
188: James Westby 2011-09-21 [merge] [r=mhall119] Allow + in track, room and attendee names without causing url lookup errors.
187: James Westby 2011-09-21 [merge] [r=mhall119] Fix the 500 error pages to not crash when displayed.
186: Michael Hall 2011-09-19 [merge] [r=james-w] Exclude attendee secret key from the API
185: James Westby 2011-09-19 [merge] [r=mhall119] Allow “+” in a meeting name without crashing on the url lookup.
184: James Westby 2011-09-17 [merge] Make it a single query, rather than doing a few queries per meeting.
183: Michael Hall 2011-09-18 [merge] [r=nigelbabu] Fixes problem with old cache on track view schedule
182: James Westby 2011-09-18 [merge] [r=mhall119] Add some tests for constructing a Schedule object.
181: Nigel Babu 2011-09-17 [merge] [r=james-w] Initial run at pep8 and pyflakes complaints fixing.
180: Michael Hall 2011-09-17 [merge] [r=james-w] Adds read-only REST/JSON API to the schedule data
179: Michael Hall 2011-09-17 [merge] [r=james-w] Adds the ability to download or subscribe to an ical containing your public *and* private meetings
178: Michael Hall 2011-09-17 [merge] [r=james-w,nigelbabu] Fix tests that were looking for a hard-coded SITE_ROOT in urls
177: Jamal Fanaian 2011-09-16 [merge] [r=mhall119] Updating the description of a meeting from the LP blueprint.
176: James Westby 2011-09-16 [merge] The autoscheduler will now reliably not require someone to be in two places at once.
175: Michael Hall 2011-09-16 [merge] [r=nigelbabu] In unit tests, specify that all Meeting instances are requires_dial_in=False unless explicitly testing that functionality.
174: James Westby 2011-09-16 [merge] [r=mhall119] Add fields on rooms and meetings for dial-in.
173: James Westby 2011-09-16 [merge] [r=mhall119] Remove the code to display the track in the room title if there is only one.
172: James Westby 2011-09-16 [merge] [r=mhall119] Add tests for the reschedule command, and make it do something again.
171: James Westby 2011-09-16 [merge] [r=mhall119] Fix a javascript error when a meeting has no participants.
170: James Westby 2011-09-16 [merge] [r=mhall119] Revert r50 which was a band-aid to fix a bug that can no longer be reproduced.
169: James Westby 2011-09-15 [merge] Fix percent sign escaping in
168: James Westby 2011-09-15 [merge] Add support for using multiple Launchpad sprints to populate a single Summit, which will allow separate UDS and Linaro Connect sprints
167: Michael Hall 2011-09-14 [merge] [r=james-w] Adds test cases to make sure % signs are being properly escaped in
166: James Westby 2011-09-14 [merge] [r=mhall119] Update the location of the linaro theme branch.
165: James Westby 2011-09-13 [merge] Add headers to avoid caches on the +temp-meeting-export fetch.
164: Michael Hall 2011-09-14 [merge] [r=james-w] Fix to allow periods in records names
163: James Westby 2011-09-13 [merge] Stops the auto-scheduler from acting on private meetings.
162: Chris Johnston 2011-09-13 [merge] [r=james-w] Updates linaro link to match uds.u.c
161: Michael Hall 2011-09-13 [merge] [r=nigelbabu] Fixes 2 typos in the get_edit_link_to_pad method
160: Michael Hall 2011-09-13 [merge] [r=james-w] Fix for the PrivateSchedulingTestCase
159: Michael Hall 2011-09-12 [merge] [r=nigelbabu] Adds a new TestCase for building tests of the schedule conflict resolution.
158: Michael Hall 2011-09-11 [merge] [r=chrisjohnston] Adds a new ‘color’ field to the Track record, this contains a 6-char hex color code that will be used as the background for meeting blocks on the schedule.
157: Michael Hall 2011-09-02 [merge] [r=chrisjohnston] Check that at least one plenary room exists before trying to use 156: Michael Hall 2011-09-02 Add back in what was lost on rebase
155: Michael Hall 2011-09-02 Add back in what was lost on rebase
154: Chris Johnston 2011-09-02 Adds ability for schedulers to schedule private rooms.
153: Chris Johnston 2011-09-02 Adds private rooms to edit page
152: Chris Johnston 2011-09-02 Adds display of private rooms on UDS page for staff
151: Chris Johnston 2011-09-02 Adds def private_rooms
150: Jamal Fanaian 2011-09-02 [merge] [r=mhall119] Created a method to get an attendee’s full name. Showing crew
149: Chris Johnston 2011-09-02 [merge] [r=mhall119] Removes stray }
148: Michael Hall 2011-08-27 [merge] More XSS fixes
147: Michael Hall 2011-08-27 [merge] Fix XSS vulnerability
147: Michael Hall 2011-08-22 [merge] Check with launchpad to see if an entered username is valid on sponsorship suggestion form
147: Michael Hall 2011-08-22 [merge] Fix errors when converting sponsorship scores to unicode strings
146: Chris Johnston 2011-08-22 [merge] [r=nigelbabu,mhall119] Updates the version of light-django-theme and fixes bzr apps after an update.
145: Chris Johnston 2011-08-22 [merge] [r=nigelbabu] Moves admin link to masthead due to wrapping in main-nav
144: Chris Johnston 2011-08-19 [merge] [r=nigelbabu] Changes Linaro link to match uds.u.c
143: Chris Johnston 2011-08-14 [merge] [r=nigelbabu] This will set debug to true when running locally.
142: Chris Johnston 2011-08-12 [merge] [r=nigelbabu] Switches summit to using bzr_apps by running init-summit.
141: Chris Johnston 2011-08-12 [merge] [r=nigelbabu] Adds a link to the page for editing an etherpad
140: Chris Johnston 2011-08-01 [merge] Adds today link to main-nav
140: Michael Hall 2011-07-22 [merge] Adds the meeting id to the meeting_page_url, and uses only that as the lookup parameter
139: Nigel Babu 2011-07-30 [merge] [r=mhall119] Reset the theme of the documentation to default.
138: Chris Johnston 2011-07-30 [merge] [r=nigelbabu][] Remove translation tags, summit is not translated.
137: Nigel Babu 2011-07-30 [merge] [r=][] Create a docs so that it shows up in
136: Chris Johnston 2011-07-25 [merge] [r=mhall119][] Removes old migrations and adds new initial migration
137: Michael Hall 2011-07-20 [merge] Adds 960px style to all pages except the wide schedule
134: Chris Johnston 2011-07-20 [merge] Fixes lpupdate. Props mhall119
134: Michael Hall 2011-07-20 Make sure we don’t have periods in meeting names when trying to form the meeting page url
133: Michael Hall 2011-07-20 [merge] point logo link to
133: Michael Hall 2011-07-20 If meeting has no name, we can’t form a proper URL for it, fallback to returning no URL
132: Chris Johnston 2011-07-19 [merge] [r=mhall119][] Adds update-openids script to summit to fix usernames.
131: Chris Johnston 2011-07-19 [merge] [r=mhall119][765031] Adds private room as an option for a room status.
130: Chris Johnston 2011-07-19 [merge] [r=mhall119][793020] Modifies links to match
129: Michael Hall 2011-07-19 [merge] [r=mhall119][781117] Changes /today url to /summit_name/today
128: Chris Johnston 2011-07-18 [merge] [r=chrisjohnston][] Fixes a minor spelling issue in
127: Chris Johnston 2011-07-18 [merge] [r=][] Updates required version of south
126: Chris Johnston 2011-07-11 [merge] [r=][] Fixes minor spelling error
125: Chris Johnston 2011-06-26 [merge] [r=nigelbabu][chrisjohnston][798822] Adds a print statement to provide feedback
124: Chris Johnston 2011-06-17 [merge] [r=mhall119][chrisjohnston][798826] Displays information in the name field error message more clearly.
123: Maris Fogels 2011-06-17 [merge] [r=nigelbabu][mars][] Added a module and an in-memory sqlite database for running the test suite.
122: Nigel Babu 2011-06-12 [merge] [r=james-w][nigelbabu][782062] Remove the ‘.’ from the name of the meeting and replace it with ‘-’
121: Nigel Babu 2011-06-10 [merge] [r=james-w][nigelbabu][790675] Stop the screen scape and use the json API instead.
120: Nigel Babu 2011-06-11 [merge] [r=chrisjohnston,james-w][nigelbabu][783291] Removed the brainstorm code out of
119: Penelope Stowe 2011-06-11 [merge] [r=james-w,nigelbabu][Penelope Stowe] try-catching the launchpad requests with 5
118: Nigel Babu 2011-06-09 [merge] Removes Attendees list from the meeting page if the session is a plenary. Props Nigel Babu
117: Nigel Babu 2011-06-09 [merge] Adds spacing around the QR code to avoid overlap. Props Nigel Babu
116: Nigel Babu 2011-05-22 [merge] Fix authschedule to not modify anything in the past
115: Chris Johnston 2011-05-22 [merge] Add prev/next day links to schedule view
114: Michael Hall 2011-05-22 [merge] Add full path to meeting page in the ical feed
113: Michael Hall 2011-05-22 Fix errors in template, add settings option to have django serve media files even when DEBUG=False
112: Chris Johnston 2011-05-22 [merge] Disable debug mode and give better error messages
111: Nigel Babu 2011-05-22 [merge] Fixes to reschedule command
110: Michael Hall 2011-05-22 [merge] Updated and requirements.txt to match current production environment




Read more
Michael Hall

Summit is the code that runs the session scheduler for the Ubuntu Developer Summit (UDS) and, as of last cycle, the Linaro Summit as well.  Summit has had a rather troubled past, being passed from one maintainer to another, evolving organically as it went.  But during UDS-N, it started gaining a team of community contributors, specifically Chris Johnston and I.  This expanded further for UDS-O, when Nigel Babu took the helm as the project manager.  We were also joined by Linaro developers who wanted to make Summit support two simultaneous events, using the same schedule, the same rooms and the same attendees.

Many changes were made in the run-up to UDS-O, and by “run-up” I mean all the way up to the first day of sessions.  Unfortunately, nowhere along Summit’s organic growth did it gain the proper test suite and deployment processes that are a necessity for a project of this size.  In fact, one of the bugs that was discovered during UDS-O was a script running on the server that wasn’t even part of Summit’s revision control tree!

Well this part of Summit’s history is coming to an end.  After UDS-O, the community developers started to plan out how to stabilize Summit, both it’s code base by adding testing, and also the deployment process by strictly managing how new code gets into production.


Bug Fixes

The bug fixing started early this cycle.  Nigel was submitting merge proposals by the end of the week of UDS-O, and Chris and I were pair-programming on the flight from JFK back to Orlando.  So far there have been 30 branch merges into the summit tree and fixes for 20 bug reports.  Nigel gives the full list over at his blog.


Setup and Development

Summit can now be easily setup for development using Virtualenv, which makes getting started with development significantly easier.  LoCo Directory recently gained a script that fully automated the setup of a development environment, and this will soon be coming to the Summit code.  At the time I’m writing this post, Jorge Castro has even begun work on an Ensemble formula, that will make deploying a fully configured instance of Summit on Amazon’s EC2 platform a matter of a few simple commands.

Making development setup easier lowers the barrier to new contributors, and we hope this will encourage more community members to get involved in such a fun and important project.  Making sure we’re all using the same development environment, and having it easily replicated for others to develop and test, will help improve the accessibility and stability of our code.



During UDS-O we got some help setting up and writing the very first testing code for Summit.  From now on, writing test cases for new features or bug fixes will become a normal part of our development process.  We recently held an online classroom session about how to write test code for Summit (and LoCo Directory too).  There is still a lot of Summit code that needs tests written for it, but we’re going to cover as much of that as we can while continuing to move forward with development.  More than any other change this cycle, I’m excited about the huge improvements to stability that we can gain through aggressively testing our code.


Branch based deployments

Summit has always used branch-based deployments, that is our production server has a copy of our bzr tree that it runs from, instead of a package that gets installed.  Unfortunately, up until last week the only branch we really had was trunk, which made it harder to properly track emergency fixes when we already had revisions committed to trunk that weren’t ready to be deployed.  To fix this we’ve split off a production branch, which is the only branch we will deploy from, and will always have a copy of the exact code that is running in production.

We will also, for the short term, have two branches for development.  The 1.x branch is our “stable” tree, that’s where we will make any changes that will be ready to deploy in the coming days or weeks.  This means that we can use our trunk branch for long-term development, where we can perform some much-needed refactoring and code cleanup, without worrying about blocking deployments while these changes settle into place.  There are some major and necessary changes coming to parts of the Summit code, and this development setup will let us start landing those quickly so that we can test them and build off them, without destabilizing the currently used code tree or blocking minor fixes from being deployed.


Ubuntu Website integration

If you visit the Summit website today, you’ll already see some of our recent changes.  To better integrate with the WordPress instance running, we have changed our main navigation and 960px width to match. Once the WordPress theme updates are rolled out, both sites will have the new community top navigation bar too.  No longer will it feel like you’re being thrown from one site to another without a means of getting back.  This should lead to a less confusing user experience for both sites, and much happier UDS attendees all around.


Read more

What is the dream setup for developing and deploying Django apps? I’m looking for a solution that I can use consistently to deploy apps to servers where I may or may not have the ability to install system packages, or where I might need my app temporarily to use a newer version of a system-installed package while giving other apps running on the same server breathing space to update (think: updating a system-installed Django package on a server running four independent apps).

Specifically, the goals I have for this scenario are:

  • It should be easy to use for both development and deployment (using standard tools and locations so developers don’t need to learn the environment),
  • Updating any virtualenv environment should be automatic, but transparent (ie. if the pip requirements.txt changes, the next time I run tests, devserver or deployed server, it’ll automatically ensure the virtualenv is correct),
  • I shouldn’t have to wait unnecessarily for virtualenvs to be created (ie. if I make a change to the requirements to try a new version of a package, and then change it back, I don’t want to re-create the original virtualenv). Similarly, if I revert a deployment to a previous version, the previous virtualenv should still be available.
  •  For deployment, the virtualenv shouldn’t unnecessarily replace system python packages, but allow this as an option (ie. not a –no-site-packages virtualenv).

There are a lot of virtualenv/fabric posts out there for both development and deployment, and using a SHA of the requirements.txt seems an obvious way to go. What I ended up with for my project was this develop and deploy with virtualenv snippet which so far is working quite well (although I’m yet to try a deploy where I override system packages). If the deployed version is using virtualenv exclusively, the requirements.txt file can be shared, but otherwise it would just be a matter of including the requirements.txt for the deploy with the other configuration data ( etc.).

If you can see any reasons why this is not a good idea, or improvements, please let me know!

Filed under: django, python

Read more
Michael Hall

This past weekend was Ubuntu Global Jam, where Ubuntu users and contributors all over the world get together to work on improving the project.  Jams come in many forms, code hacking, bug triaging, translating, documenting, or even just promoting Ubuntu in their community.  In my own corner of the Ubuntu community, a few of us got to together to work on improving the Summit project

This is the code behind, which provides the UDS scheduler and sponsorship application forms.  Summit is a Django application, released under the AGPLv3 license, and is primarily developed by community members.  Joining me were Chris Johnston,  a frequent community contributor who I’ve also worked with in LoCo Directory and other projects, and Elliot Murphy, my 3rd-level boss as Canonical (no pressure there!).

Here’s a list of what we managed to accomplish:

Switch to the new ubuntu-community-webthemes, which will give us the “mothership” top-navigation links as seen on and

Started work on integrating Summit with Django testing framework.

Bug #643012: Register Interest should only show currently available tracks

Currently when you register your interest in a track, the form shows tracks for previous summits.  This will restrict it to just the tracks for the summit you’re registering for.

Bug #668532: /today page to display current day’s schedule

A new, permanent URL which will show the current day’s schedule, so you can bookmark it once and re-use it for each day of the summit, and even future summits!

Bug #745378: Empty sub-nav exists on sponsorship page

Removes the gray sub-navigation bar from pages where there aren’t any linkes in it.

Bug #462793: Add slots for videographers

Up to two videographers can not be assigned to a UDS session and their names will appear on the schedule.

Bug #747296: Add plenary flag to iCal feed for

We have been working with the makers of Conventionist, a convention management application, which will allow you to track your session schedule on your Android or iPhone, even getting directions to the correct room.  This fix was necessary for them to distinguish plenary sessions from regular ones.

Bug #747301: Add daily Crew list

Allows Summit to schedule which UDS attendees are willing to act as event crew, with the current day’s crew assignments listed on the daily schedule which is displayed on the large monitors during the event.

Bug #747303: Auto-add slots to schedule

This solved an administrative headache for those organizing the summit.  For past events, every available time slot had to be entered manually, which was a very time consuming task.  This provides them a quick way to pre-populate the time slots, with the ability to fine-tune just the ones that need it.

Bug #747419: Fix login redirect

Several features of Summit require that you log in using your SSO/Launchpad account.  However, after login you are currently redirected back to the main Summit page instead of the page you left.  This sends your current page URL as the path to redirect to after a successful login, so you no longer have to go find that page again.


Read more
Anthony Lenton

The great Django command-extensions app gives you an easy way to integrate the Werkzeug debugger into your Django site.

For if you’ve never used this debugger before, imagine the regular Django technical details debugging page, with all same super useful information — environment variables, sql queries, and a full traceback with even the local variables and a bit of code context for each frame.
Now imagine you could have a pdb prompt running within each frame of the traceback, actually running interactively via ajax within the frames that you see.  That’s the kind of raw power Werkzeug gives you.

A big disclaimer though: this is not for running on production sites.  You’ll be giving anybody that happens to stumble upon a failure page an interactive (unrestricted) python terminal running with the web server’s permissions on your box.

So, this is just what out the box django-command-extension’s runserver_plus gives you.  With a bit of tweaking, here are a couple of other neat things you can do:

Make your server multithreaded

There are times when you need a multithreaded server in your development environment.  I’ve seen arguments about this being a smell for bad design, so here ar two reasonable scenarios I’ve come across:

  • Developing a web API, you might have a view that provides a test consumer for the API, not intended to be available on production.  The test consumer view will make a (blocking) call out to the API that’s provided by the same server, hanging everything if you don’t have multithreadedness.
  • Or, in larger systems where you have more than one application server, you might have a view or two that do a bit of communication between servers (to see if everything’s ok, to provide a neat server status screen), that you’ll need to test calling out to the same server during development.

For whatever reason it may be, if you need to enable threading, you can just set threaded=True when you call Werkzeug’s run_simple function:

Debug your whole wsgi app instead of just Django

If you’re running Django as part of a larger wsgi stack (a non-Django web api, or some authentication wsgi middleware), you probably would love to be able to run the whole stack in your development server, and even have those same great debugging features available for everything, not only Django.
You can do this by modifying the wsgi app that Werkzeug debugs:

It’s something with raw power that keeps getting better the more you look…

Read more
Michael Foord

After the release of Ubuntu Maverick Meerkat we have a brief time of housekeeping in the ISD team, where we work on clearing our technical debt and implementing improvements to our development processes. One of these improvements has been getting continuous integration setup for some of our projects. Continuous integration means not just having an automated test suite, but having an automated system for running the tests (continuously)

We have settled on Hudson as our CI server. Despite being a Java based tool it is a popular choice in the Python world, mainly because Hudson is both easy to install / configure and provides an attractive and powerful web based interface out-of-the-box.

You can use Hudson to report test run pass or fail, and view the console output from a test run, with virtually no work at all. Simply provide a shell command for running your tests and off you go. Hudson works best when your test run generates an XML description of the test run in a poorly specified (but widely implemented) format called JUnit-XML. This is the same JUnit that was the original inspiration for the Python unittest testing library.

With JUnit XML describing your test run you can use the Hudson UI to view individual test failures and generate pretty graphs for how the time taken by tests changes over successive builds.

Our projects are based on Django, which in turn uses the standard unittest test runner for running tests. Python has a wealth of different choices for coaxing JUnit-XML out of a unittest test run. As we’re deploying on Ubuntu Lucid servers we needed a solution easy to integrate with the Lucid distribution of Django (approximately version 1.1.2).

After trying several alternatives we settled on the pyjunitxml package, which just happens to be the creation of Robert Collins, a fellow Canonical developer.

For a suite (unittest terminology for a collection) of tests, getting a junit-xml report from pyjunitxml is gloriously easy. Here’s the code:

    import junitxml
    with open('xmlresults.xml', 'w') as report:
        result = junitxml.JUnitXmlResult(report)

If you’re familiar with unittest code you may be surprised that there is no test runner involved. The unittest TextTestRunner class is useful for generating console output of a test run, but as this isn’t needed for a run under continuous integration we only need a test suite and the test result object from junitxml.

To integrate this into a standard django test run we had to copy the run_tests function from the django.test.simple module and add a parameter to use this code when running under Hudson.

Unsurprisingly our projects are managed on Launchpad and use Bazaar for version control. Although not enabled by default Hudson ships with a plugin for Bazaar integration. Here’s a guide for setting up Hudson for a Launchpad / Bazaar based project. It assumes you have a script “” for running your test suite with the junitxml code active:

First install hudson. For debian and Ubuntu this link gives the details:

(You will also need a version of Java and the jre installed which I don’t think that guide covers.)

Once installed Hudson will start on boot. Immediately after install you may need to start it manually:

sudo /etc/init.d/hudson restart

You can view the locally running Hudson from this URL:


You need to enable the bzr plugin in Hudson. You can do this through the Hudson plugin manager:


Find it in the ‘available’ tab of the plugin manager. You will need to restart Hudson after installing the plugin.

Next you need to give the Hudson user access to your repository. You can this step if your repository is public and can be fetched by an anonymous user.

  • Switch to the Hudson user: sudo su – hudson
  • Tell bzr your launchpad user: bzr lp-login your-username
  • Generate ssh keys without a passphrase: ssh-keygen -t rsa
  • Add the public key to your user on launchpad.

Next create a new Hudson job, with whatever name you want.

  • Select Build a free-style software project
  • Source code management: Bazaar
  • Repository URL: lp:~your-username/project-name/branch
  • Add build step -> Execute Shell:
  • Post-build action: Check Publish JUnit test result report
  • Test report XMLs: xmlresults.xml

Finally: Build Now

You can watch the build in progress via the console output.

There is no console output during the test run itself, the results go to the junit xml file used by Hudson to report the results. The console output is still useful as any build errors or unhandled exceptions will show up there.

At the moment Hudson is happily running scheduled builds on my local machine. The next step is to run it on our private cloud and decide whether or not to use a commit hook to trigger the builds. I have a personal preference for scheduled builds (as close to back-to-back as possible). Multiple runs per revision gives you the opportunity to shake out fragilities in your tests and collect better data (more data == better data) for performance tests.

Read more