Canonical Voices

What Something driven development talks about

Michael

Just documenting for later (and for a friend and colleague who needs it now) – my notes for setting up openstack swift using juju. I need to go back and check whether keystone is required – I initially had issue with the test auth so switched to keystone.

First, create the config file to use keystone, local block-devices on the swift storage units (ie. no need to mount storage), and using openstack havana:

cat >swift.cfg <<END
swift-proxy:
    zone-assignment: auto
    replicas: 3
    auth-type: keystone
    openstack-origin: cloud:precise-havana/updates
swift-storage:
    zone: 1
    block-device: /etc/swift/storagedev1.img|2G
    openstack-origin: cloud:precise-havana/updates
keystone:
    admin-token: somebigtoken
    openstack-origin: cloud:precise-havana/updates
END

Deploy it (this could probably be replaced with a charm bundle?):

juju deploy --config=swift.cfg swift-proxy
juju deploy --config=swift.cfg --num-units 3 swift-storage
juju add-relation swift-proxy swift-storage
juju deploy --config=swift.cfg keystone
juju add-relation swift-proxy keystone

Once everything is up and running, create a tenant and user with the user having admin rights for the tenant (using your keystone unit’s IP address for keystone-ip). Note, below I’m using the names of tenant, user and role – which works with keystone 0.3.2, but apparently earlier versions require you to use the uuids instead. Check with `keystone help user-role-add`).

$ keystone --endpoint http://keystone-ip:35357/v2.0/ --token somebigtoken tenant-create --name mytenant
$ keystone --endpoint http://keystone-ip:35357/v2.0/ --token somebigtoken user-create --name myuser --tenant mytenant --pass userpassword
$ keystone --endpoint http://keystone-ip:35357/v2.0/ --token somebigtoken user-role-add --tenant mytenant --user myuser --role Admin

And finally, use our new admin user to create a container for use in our dev environment (specify auth version 2):

$ export OS_REGION_NAME=RegionOne
$ export OS_TENANT_NAME=mytenant
$ export OS_USERNAME=myuser
$ export OS_PASSWORD=userpassword
$ export OS_AUTH_URL=http://keystone-ip:5000/v2.0/
$ swift -V 2 post mycontainer

If you want the container to be readable without auth:

$ swift -V 2 post mycontainer -r '.r:*'

If you want another keystone user to have write access:

$ swift -V 2 post mycontainer -w mytenant:otheruser

Verify that the container is ready for use:
$ swift -V 2 stat mycontainer

Please let me know if you spot any issues (these notes are from a month or two ago, so I haven’t just tried this).


Filed under: Uncategorized

Read more
Michael

After working with InformatiQ to setup a new charm using the ansible support (and ironing out a few issues), it made sense to capture the process…

The README at charm-bootstrap-ansible has the details, but the branch will pull in the required charm-helpers library and run the tests, leaving you ready to deploy and explore.

Hopefully I can get this into the charm-create tool eventually.


Filed under: bzr, juju

Read more
Michael

I’ve been working on some more support for ansible in the juju charm-helpers package recently [1], which has effectively transformed my juju charm’s hooks.py to something like:

# Create the hooks helper, passing a list of hooks which will be
# handled by default by running all sections of the playbook
# tagged with the hook name.
hooks = charmhelpers.contrib.ansible.AnsibleHooks(
    playbook_path='playbooks/site.yaml',
    default_hooks=['start', 'stop', 'config-changed',
                   'solr-relation-changed'])

@hooks.hook()
def install():
    charmhelpers.contrib.ansible.install_ansible_support(from_ppa=True)

And that’s it.

If I need something done outside of ansible, like in the install hook above, I can write a simple hook with the non-ansible setup (in this case, installing ansible), but the decorator will still ensure all the sections of the playbook tagged by the hook-name (in this case, ‘install’) are applied once the custom hook function finishes. All the other hooks (start, stop, config-changed and solr-relation-changed) are registered so that ansible will run the tagged sections automatically on those hooks.

Why am I excited about this? Because it means that practically everything related to ensuring the state of the machine is now handled by ansibles yaml declarations (and I trust those to do what I declare). Of coures those playbooks could themselves get quite large and hard to maintain, but ansible has plenty of ways to break up declarations into includes and roles.

It also means that I need to write and maintain fewer unit-tests – in the above example I need to ensure that when the install() hook is called that ansible is installed, but that’s about it. I no longer need to unit-test the code which creates directories and users, ensures permissions etc., or even calls out to relevant charm-helper functions, as it’s all instead declared as part of the machine state. That said, I’m still just as dependent on integration testing to ensure the started state of the machine is what I need.

I’m pretty sure that ansible + juju has even more possibilities for being able to create extensible charms with plugins (using roles), rather than forcing too much into the charms config.yaml, and other benefits… looking forward to trying it out!

[1] The merge proposal still needs to be reviewed, possibly updated and landed :)


Filed under: Uncategorized

Read more
Michael

logo-jujuHave you ever wished you could just declare the installed state of your juju charm like this?

deploy_user:
    group.present:
        - gid: 1800
    user.present:
        - uid: 1800
        - gid: 1800
        - createhome: False
        - require:
            - group: deploy_user

exampleapp:
    group.present:
        - gid: 1500
    user.present:
        - uid: 1500
        - gid: 1500
        - createhome: False
        - require:
            - group: exampleapp


/srv/{{ service_name }}:
    file.directory:
        - group: exampleapp
        - user: exampleapp
        - require:
            - user: exampleapp
        - recurse:
            - user
            - group


/srv/{{ service_name }}/{{ instance_type }}-logs:
    file.directory:
        - makedirs: True

While writing charms for Juju a long time ago, one of the things that I struggled with was testing the hook code – specifically the install hook code where the machine state is set up (ie. packages installed, directories created with correct permissions, config files setup etc.) Often the test code would be fragile – at best you can patch some attributes of your module (like “code_location = ‘/srv/example.com/code’”) to a tmp dir and test the state correctly, but at worst you end up testing the behaviour of your code (ie. os.mkdir was called with the correct user/group etc.). Either way, it wasn’t fun to write and iterate those tests.

But support has improved over the past year with the charmhelpers library. And recently I landed a branch adding support for declaring saltstack states in yaml, like the above example. That means that the install hook of your hooks.py can be reduced to something like:

import charmhelpers.core.hookenv
import charmhelpers.payload.execd
import charmhelpers.contrib.saltstack


hooks = charmhelpers.core.hookenv.Hooks()


@hooks.hook()
def install():
    """Setup the machine dependencies and installed state."""
    charmhelpers.contrib.saltstack.install_salt_support()
    charmhelpers.contrib.saltstack.update_machine_state(
        'machine_states/dependencies.yaml')
    charmhelpers.contrib.saltstack.update_machine_state(
        'machine_states/installed.yaml')


# Other hooks...

if __name__ == "__main__":
    hooks.execute(sys.argv)

…letting you focus on testing and writing the actual hook functionality (like relation-set’s etc. I’d like to add some test helpers that will automatically check the syntax of the state yaml files and template rendering output, but haven’t yet).

Hopefully we can add similar support for puppet and Ansible soon too, so that the charmer can choose the tools they want to use to declare the local machine state.

A few other things that I found valuable while writing my charm:

  • Use a branch for charmhelpers – this way you can make improvements to the library as you develop and not be dependent on your changes landing straight away to deploy (thanks Sidnei – I think I just copied that idea from one of his charms). The easiest way that I found for that was to install the branch into mycharm/lib so that it’s included in both dev and when you deploy (with a small snippet in your hooks.py.
  • Make it easy to deploy your local charm from the branch… the easiest way I found was a link-test-juju-repo make target – I’m not sure what other people do here?
  • In terms of writing actual hook functionality (like relation-set events etc), I found the easiest way to develop the charm was to iterate within a debug-hook session. Something like:
    1. write new test+code then juju upgrade-charm or add-relation
    2. run the hook and if it fails…
    3. fix and test right there within the debug-hook
    4. put the code back into my actual charm branch and update the test
    5. restore the system state in debug hook
    6. then juju upgrade-charm again to ensure it works, if it fails, iterate from 3.
  • Use the built-in support of template rendering from saltstack for rendering any config files that you need.

I don’t think I’d really appreciated the beauty of what juju is doing until, after testing my charm locally together with a gunicorn charm and a solr backend, I then setup a config using juju-deployer to create a full stack with an Apache front-end, a cache load balancer for multiple squid caches, as well as a load balancer in front of potentially multiple instances of my charms wsgi app, then a back-end loadbalancer in between my app and the (multiple) solr backends… and it just works.


Filed under: juju, python, testing

Read more
Michael

A number of times over the past few years I’ve needed to create some quite complex migrations (both schema and data) in a few of the Django apps that I help out with at Canonical. And like any TDD fanboy, I cry at the thought of deploying code that I’ve just tested by running it a few times with my own sample data (or writing code without first setting failing tests demoing the expected outcome).

This migration test case helper has enabled me to develop migrations test first:

class MigrationTestCase(TransactionTestCase):
    """A Test case for testing migrations."""

    # These must be defined by subclasses.
    start_migration = None
    dest_migration = None
    django_application = None

    def setUp(self):
        super(MigrationTestCase, self).setUp()
        migrations = Migrations(self.django_application)
        self.start_orm = migrations[self.start_migration].orm()
        self.dest_orm = migrations[self.dest_migration].orm()

        # Ensure the migration history is up-to-date with a fake migration.
        # The other option would be to use the south setting for these tests
        # so that the migrations are used to setup the test db.
        call_command('migrate', self.django_application, fake=True,
                     verbosity=0)
        # Then migrate back to the start migration.
        call_command('migrate', self.django_application, self.start_migration,
                     verbosity=0)

    def tearDown(self):
        # Leave the db in the final state so that the test runner doesn't
        # error when truncating the database.
        call_command('migrate', self.django_application, verbosity=0)

    def migrate_to_dest(self):
        call_command('migrate', self.django_application, self.dest_migration,
                     verbosity=0)
 

It’s not perfect – schema tests in particular end up being quite complicated as you need to ensure you’re working with the correct orm model when creating your test data – and you can’t use the normal factories to create your test data. But it does enable you to write migration tests like:

class MyMigrationTestCase(MigrationTestCase):

    start_migration = '0022_previous_migration'
    dest_migration = '0024_data_migration_after_0023_which_would_be_schema_changes'
    django_application = 'myapp'

    def test_schema_and_data_updated(self):
        # Test setup code

        self.migrate_to_dest()

        # Assertions

which keeps me happy. When I wrote that I couldn’t find any other suggestions out there for testing migrations. A quick search now turns up one idea from André (data-migrations only),  but nothing else substantial. Let me know if you’ve seen something similar or a way to improve testing of migrations.


Filed under: django, python, testing

Read more
Michael

Android UI Fragments look like a great way to build re-usable elements within an app, but they don’t work exactly as expected out of the box (well, exactly is I expected – but that could be a lack of experience with android):

After defining an onClick event on a button within my fragment:

  <Button android:id="@+id/add_goal_button"
      android:layout_width="wrap_content"
      android:layout_height="wrap_content"
      android:text="@string/button_create"
      android:onClick="addGoal" />

I’d expected this event to be routed directly to my fragment class without the containing Activity class needing to know about it – but instead, the addGoal() method is expected on the containing Activity instead.

To connect the fragment event directly to a click handler on the fragment class (so the view doesn’t need to be handling the fragment events, you can do the following instead (thanks Brill Pappin):

public class NewGoalFragment extends Fragment {
	
	...
	
	@Override
	public View onCreateView(LayoutInflater inflater, ViewGroup container,
			                 Bundle savedInstanceState) {
		final View fragmentView = inflater.inflate(R.layout.fragment_new_goal,
							   container, false);
		...
		Button addGoalButton = (Button) fragmentView.findViewById(R.id.add_goal_button);
		addGoalButton.setOnClickListener(new OnClickListener() {
			@Override
			public void onClick(final View v) {
				// Pass the fragmentView through to the handler
				// so that findViewById can be used to get a handle on
				// the fragments own views.
				addGoal(fragmentView);
			}
		});
		return fragmentView;
	}
	
    public void addGoal(View view) {    	
    	EditText newGoal = (EditText) view.findViewById(R.id.new_goal);
	...
    }

Filed under: android

Read more
Michael

The last exercise in the Go Tour – parallelizing a web crawler – turned out to be quite a bit more interesting than I’d expected. If anyone has suggested improvements from which I can learn a bit more, or there own solutions posted, let me know – my exercise solution is on github. I’ve tried to stick to the tour content (ie. only using channels rather than the sync package for accessing shared data).

Spoiler Alert: If you are learning Golang and haven’t yet worked through the Go-Tour, go and do so now. If you get stuck, keep struggling, take a break, try again in a few days etc., before looking at other peoples’ solutions.

The solution I ended up with has a Crawl() function very similar to the original, just with two extra function parameters:

func Crawl(url string, depth int, fetcher Fetcher,
	startCrawl func(string) bool, crawlComplete chan string) {

	if depth <= 0 {
		crawlComplete <- url
 		return
	}

	body, urls, err := fetcher.Fetch(url)
	if err != nil {
		fmt.Println(err)
		crawlComplete <- url
 		return
	}

	fmt.Printf("found: %s %q\n", url, body)
	for _, u := range urls {
		if startCrawl(u) {
			go Crawl(u, depth-1, fetcher, startCrawl, crawlComplete)
		}
	}
	crawlComplete <- url
}

The two parameters are:

  • startCrawl func(url string) bool – used as a check before spawning a new ‘go Crawl(url)’ to ensure that we don’t crawl the same url twice.
  • crawlComplete chan string – used to signal that the Crawl function has fetched the page and finished spawning any child go-routines.

These two resources are created and passed in to the initial Crawl() call in the main() function:

func main() {
	startCrawl := make(chan StartCrawlData)
	crawlComplete := make(chan string)
	quitNow := make(chan bool)
	go processCrawls(startCrawl, crawlComplete, quitNow)

	// Returns whether a crawl should be started for a given
	// URL.
	startCrawlFn := func(url string) bool {
		resultChan := make(chan bool)
		startCrawl <- StartCrawlData{url, resultChan}
		return <-resultChan
	}

	Crawl("http://golang.org/", 4, fetcher, startCrawlFn,
		crawlComplete)

	<-quitNow
}

Access to the shared state of which urls have been crawled and when all Crawls() have finished etc., is managed via those channels in the processCrawls() go-routine, so that the main() can simply call the first Crawl() and then wait to quit. I want to check how cheap the temporary creation of a channel is (for the return value of the startCrawlFn above) – I think I saw this method on an earlier GoLang tutorial example, but otherwise I’m happy with the solution :-).

Other solutions to learn from:


Filed under: golang

Read more
Michael

Kanban for kids

Facing the arrival of another child, I set out a few months ago to experiment with ways to help each other get things done together as a family without becoming a task-master of a father. A very simplified Kanban board seemed like a good fit for our kitchen wall, as it visualises what needs doing or other fun things that are planned, helps us to help each other when needed, and enables the kids to take part in the planning and organising of the day.

The photo of our board above is what we’ve ended up with after a few months. It is really a habit tracker – with most of the items on the board being things that we do daily, every few days, weekly or fortnightly. Some points that we’ve found helpful:

  • Items specific to the kids are colour coded (ie. we have a “Put clothes away” item for each child in their colour).
  • We have some incentives – each item has a number of associated “service points” [1]. For eg. 5 for putting away clothes, 15 for taking the dog out for a walk etc. (agreed with the kids and adjusted on feedback).
  • We have incentives to have fun doing items together – if you complete an item together with someone else, you get 50% extra service points, simply to encourage having fun doing things together (we put a magnet on the item to signify this).
  • We very rarely tell the kids to do something – in extreme circumstances I’ve told my daughter I need her to take the dog out now as I can’t do it myself, but it’s something we generally avoid, in fact…
  • The kids generally choose what item they want to do next and when they want to do it (ie. “You’d like to help daddy set the table for lunch? Shall we do it now or after playing Lego for a while?”).
  • Each item has a drawing and is laminated with a small magnet, so that we can stack certain items together (for eg, we have a stack of cards related to the dog which are done in order each day – dog breakfast, walk 7am-ish, Walk 11am-ish, etc).
  • We start each day by pulling things that we want to do into the backlog (in addition to the daily stuff). Some items are weekly, which we store in a stack for each day on the top-left. Other items are ad-hoc which we pull in from the mess on the bottom left.
  • We limit the number of cards in the TODO and Doing lanes to 4, to help us help each flow (or review whether we really want to do something).
  • We don’t put normal stuff like playing lego or dressups there – the kids can play whenever they want without feeling they need to have an item on the board :P

So far it’s been a great help, but as always, a continual learning process. Each child is motivated by different things, and yet the end goal is that we’d all enjoy helping each other without extrinsic motivation. Interestingly, one of the most freeing things has been for the kids (and my wife) to get used to the fact that it’s OK to say, no, I don’t feel like doing that today, I’m just going to move the card over to Thursday for the moment. That is, the board is something we control – a helpful way to organise and communicate – it does not tell us what we have to do.

[1] Service points – The kids can use the service points at the end of the day in ways such as staying up a bit later (1minute per point, within reason), or going out on their own with mum or dad for an ice-cream desert (50 points). We haven’t yet found a good solution for left over points, and are currently converting to Euro-cents as pocket money at the end of the week which works well in terms of value, but I don’t like the direct connection of being paid money to do stuff around the house. Another option I’m trying at the moment is using saved service points as bargaining power for larger items that the kids need (like a new school bag). I’d be glad to hear of other ideas.


Filed under: Uncategorized

Read more
Michael

I’ve spent a few evenings this week implementing a derbyjs version of the Todo spec for the TodoMVC project [1] – and it was a great way to learn more about the end-to-end framework, and appreciate how neat the model-view bindings really are. Here’s a 2 minute demo showing the normal TodoMVC functionality as well as the collaborative editing which Derby brings out of the box:

 

It’s amazing how simple Derby’s model-view bindings enable the code to be. It’s really just two files containing the functionality:

The other files are just setup (define which queries are allowed, define the express server, and some custom style on top of the base.css from TodoMVC). Well done Nate and Brian, and the DerbyJS community (which seems to be growing quite a bit over the last few weeks)!

[1] I’ve still got a few things todo before I can submit a pull-request to get this added to TodoMVC.


Filed under: javascript

Read more
Michael

Over the last week or so I’ve spent a few hours learning a bit about DerbyJS – an all-in-one app framework for developing collaborative apps for the web [1]. You can read more about DerbyJS itself at derbyjs.com, but here are six highlights that I’m excited about (text version below the video):

 

1. The browser reflects dev changes immediately. While developing, derbyjs automatically reflects any changes you make to styles, templates (and scripts?) as soon as you save. No need to switch windows and refresh, instead they’re pushed out to your browser(s).

2. Separation of templates (views) and controllers. Derbyjs provides a model-view-controller framework that we’ve come to expect, with Handlebars-like templates and trivial binding to any event handlers defined in your controller. Derby also provides standard conventions for file locations and bundles your files for you.

3. Model data is bound to the view – derbyjs automatically updates other parts of your templates that refer to any data which the user changes, but that’s not all…

4. Model data is synced real-time (as you type/edit) – updating the data you are changing in all browsers viewing the same page. The data just synchronises (and resolves conflicts) without me caring how. (OK, well I really do care how, but I don’t *need* to care).

5. The same code runs on both the (node) server and the browser. The code that renders a page after a server request (great for initial page load and search indexing) is one and the same code that renders pages in the browser without (necessarily) hitting the server.

6. My app works offline out of the box. That’s right – as per the demo – any changes made while offline are automatically synced back to the server and pushed out to other clients as soon as a connection is re-established.

It’s still early days for DerbyJS, but it looks very promising – opening up the doors to loads of people with great ideas for collaborative apps who don’t have the time to implement their own socket.io real-time communication or conflict resolution. Hats off to the DerbyJS team and community!

[1] After doing similar experiments in other JS frameworks (see here and here), and being faced with the work of implementing all the server-sync, sockets, authentication etc. myself, I went looking and found both meteorjs and derbyjs. You can read a good comparison of meteorjs and derbyjs by the derbyjs folk (Note: both are now MIT licensed).


Filed under: javascript

Read more
Michael

After experimenting recently with the YUI 3.5.0 Application framework, I wanted to take a bit of time to see what other HTML5 app development setups were offering while answering the question: “How can I make HTML5 app development more fun on Ubuntu” – and perhaps bring some of this back to my YUI 3.5 setup.

I’m quite happy with the initial result – here’s a brief (3 minute) video overview highlighting:

  • Tests running not only in a browser but also automatically on file save without even a headless browser (with pretty Ubuntu notifications)
  • Modular code in separate files (including html templates, via requirejs and its plugins)

 

 

Things that I really like about this setup:

  • requirejs - YUI-like module definitions and dependency specification makes for very clear code. It’s an implementation of the Asynchronous Module Definition “standard” which allows me to require dependencies on my own terms, like this:
    require(["underscore", "backbone"], function(_, Backbone) {
        // In here the underscore and backbone modules are loaded and
        // assigned to _ and Backbone respectively.
    })

    There’s some indication that YUI may also implement AMD in its loader also. RequireJS also has a built in optimiser to combine and minify all your required JS during your build step. With two plugins for RequireJS I can also use CoffeeScript instead of Javascript, and load my separate HTML templates as resources into my modules (no more stuffing them all into your index.html.

  • mocha tests running on nodejs or in the browser – as shown in the above screencast. Once configured, this made it pretty trivial to add a `make watch` command to my project which runs tests automatically (using nodejs’ V8 engine) when files change, displaying the results using Ubuntu’s built-in notification system. (Mocha already has built in growl support for Mac users, it’d be great to get similar OSD notifications built in too).

The setup wasn’t without its difficulties [1], but the effort was worth it as now I have a fun environment to start building my dream app (we’ve all got one right?) and continue learning. I think it should also be possible for me to go back and re-create this nodejs dev environment using YUI also – which I’m keen to try if someone hasn’t already done something similar – or even possibly without needing nodejs? I think the challenge for YUI will be if and when most other modules can be loaded via AMD why, as an app developer, would I want to commit to one monolithic framework release when I can cleanly pick-n-chose the specific versions of small tightly-focused modules that I need (assuming my tests pass). Or perhaps YUI will join in and begin versioning modules (and module dependencies) rather than the one complete framework so that they were available via any AMD loader – that would rock!

Thanks to James Burke (author of RequireJS) and the brunch team for their help!

[1] For those interested, things that were difficult getting this setup were:

  • Many JS libraries are not yet AMD ready (or yet giving support), which means adding shims to load them correctly (or using the use plugin in some cases). And sometimes this gets complicated (as it did for me with expect.js). I don’t know if AMD will get widespread adoption, who knows? A result of this is that many JS libraries are designed to work within the browser or node only (ie. they assume that either window or module/exports will be available globally).
  • Using the coffeescript plugin is great for looking at the code, but the errors displayed when running tests that result from  coffeescript parse errors are often hard to decipher (although I could probably use an editor plugin to check the code on save and highlight errors).
  • A recent nodejs version isn’t yet available in the ubuntu archives for Precise. It wasn’t difficult, but I had to install Node 0.6.12 to my home directory and put its bin directory on my path before I could get started.

If you want to try it out or look at the code, just make sure NodeJS 0.6.12 is available on your path and do:

tmp$ bzr branch lp:~michael.nelson/open-goal-tracker/backbone-test/
Branched 42 revisions. 
tmp$ cd backbone-test/
backbone-test$ make
[snip]
backbone-test$ make test
...........
? 12 tests complete (47ms)

Filed under: javascript, open-goal-tracker, ubuntu

Read more
Michael

I had a few hours recently to try updating my Open Goal Tracker javascript client prototype to use jQuery Mobile for the UI… and wow – it is so nice, as a developer with an idea, not having to think about certain UI issues (such as a touch interface, or just basic widget design). I can see now how ugly my previous play-prototype looked. Here’s a brief demo of the jQueryMobile version (sorry for the mumbling):

 

That’s using jQuery Mobile 1.01 for the UI and YUI 3.5.0PR2 for the MVC client-side framework, although I’m tempted to move over to backbone.js (which is what the YUI application framework is based on, it seems). Backbone.js has beautifully annotated source and a book – Developing backbone applications - which so far seems like very worthwhile reading material.

The prototype can be played with at http://opengoaltracker.org/prototype_fun/


Filed under: javascript, jquery, open-goal-tracker, yui

Read more
Michael

More YUI and html5 fun

Continuing on my YUI MVC experiment, I got to spend a bit of time last week playing with my prototype for Open Goal Tracker, and was blown away by the simplicity of two new html attributes: x-webkit-speech (just ‘speech’ when standardised) and contenteditable. Here’s a 2min video showing how I might use both of them in practise (in dire need of styling as usual):

 

 

Behind the scenes I’m still loving Yahoo User Interfaces application framework together with the unit-test api, which allow me to create tests like “creating the view should render stats for existing goals”.

Next week I hope to start implementing the server-sync and investigating integration with Ubuntu’s notification system. Once I’ve sorted out the application-cache and build/packaging etc. for both Ubuntu and Android, I want to do a initial dummy release and then start iterating the features that will make Open Goal Tracker actually useful for facilitating social-yet-individual learning, both within and without of classrooms.

You can play with the little open-goal-tracker prototype (I’ve only tried webkit so far) or run the tests yourself.


Filed under: Uncategorized

Read more
Michael

After experimenting with juju and puppet the other week, I wanted to see if it was possible to create a generic juju charm for deploying any Django apps using Apache+mod_wsgi together with puppet manifests wherever possible. The resulting apache-django-wsgi charm is ready to demo (thanks to lots of support from the #juju team), but still needs a few more configuration options. The charm currently:

  1. Enables the user to specify a branch of a Python package containing the Django app/project for deploy. This python package will be `python setup.py install`’d on the instance, but it also
  2. Enables you to configure extra debian packages to be installed first so that your requirements can be installed in a more reliable/trusted manner, along with the standard required packages (apache2, libapache2-mod-wsgi etc.). Here’s the example charm config used for apps.ubuntu.com,
  3. Creates a django.wsgi and httpd.conf ready to serve your app, automatically collecting all the static content of your installed Django apps to be served separately from the same Apache virtual host,
  4. When it receives a database relation change, it creates some local settings, overriding the database settings of your branch, sync’s and migrates the database (a noop if it’s the second unit) and restarts apache (See the database_settings.pp manifest for more details).

Here’s a quick demo which puts up a postgresql unit and two app servers with these commands:

$ juju deploy --repository ~/charms local:postgresql
$ juju deploy --config ubuntu-app-dir.yaml --repository ~/apache-django-wsgi/ local:apache-django-wsgi
$ juju add-relation postgresql:db apache-django-wsgi
$ juju add-unit apache-django-wsgi

Things that I think need to be improved or I’m uncertain about:

  1. `gem install puppet-module` is included in the install hook (a 3rd way of installing something on the system :/). I wanted to use the vcsrepo puppet module to define bzr resource types and puppet-module-tool seems to be the way to install 3rd-party puppet modules. Using this resource-type enables a simple initial_state.pp manifest. Of course, it’d be great to have ‘necessary’ tools like that in the archive instead.
  2. The initial_state.pp manifest pulls the django app package to /home/ubuntu/django-app-branch and then pip installs it on the system. Requiring the app to be a valid python package seemed sensible (in terms of ensuring it is correctly installed with its requirements satisfied) while still allowing the user to go one step further if they like and provide a debian package instead of a python package in a branch (which I assume we would do ultimately for production deploys?)
  3. Currently it’s just a very simple apache setup. I think ideally the static file serving should be done by a separate unit in the charm (ie. an instance running a stripped down apache2 or lighttpd). Also, I would have liked to have used an ‘official’ or ‘blessed’ puppet apache module to benefit from someone else’s experience, but I couldn’t see one that stood out as such.
  4. Currently the charm assumes that your project contains the configuration info (ie. a settings.py, urls.py etc.), of which the database settings can be simply overridden for deploy. There should be an additional option to specify a configuration branch (and it shouldn’t assume that you’re using django-configglue), as well as other options like django_debug, static_url etc.
  5. The charm should also export an interface (?) that can be used by a load balancer charm.

Filed under: django, juju

Read more
Michael

I’ve been playing around wit h YUI’s new application framework over the past few weeks, basically building on the ToDo list example app, extending it for a basic interface for the Open Goal Tracker project. Here’s a 2 min demo:

I’m loving the MVC separation in the client code, separation of templates, ability to use real links for navigating within the client (which will also work when copy-n-pasted to hit the server) etc.

If anyone is interested, you can play with the code yourself:

$ bzr branch lp:~michael.nelson/open-goal-tracker/html5-client-play
$ cd html5-client-play
$ fab test

which will setup the dev environment and run the django tests… followed by

$ fab runserver

To play with the demo html5 client: http://localhost:8000/static/index.html
To run the javascript unit tests: http://localhost:8000/static/app/tests/test_models.html

or find out more about the purpose of the app with the first 2 mins of this video from a year ago outlining where i hope to go with the application: http://www.youtube.com/watch?v=OT7P-u-86sM (when time permits :) ).


Filed under: javascript, yui

Read more
Michael

I’ve been playing with juju for a few months now in different contexts and I’ve really enjoyed the ease with which it allows me to think about services rather than resources.

More recently I’ve started thinking about best-practices for deploying services using juju, while still using puppet to setup individual units. As a simple experiment, I wrote a juju charm to deploy an irssi service [1] to dig around. Here’s what I’ve found so far [2]. The first is kind of obvious, but worth mentioning:

Install hooks can be trivial:

#!/bin/bash
sudo apt-get -y install puppet

juju-log "Initialising machine state."
puppet apply $PWD/hooks/initial_state.pp

Normally the corresponding manifest (see initial_state.pp) would be a little more complicated, but in this example it’s hardly worth mentioning.

Juju config changes can utilise Puppet’s Facter infrastructure:

This enables juju config options to be passed through to puppet, so that config-changed hooks can be equally simple:

#!/bin/bash
juju-log "Getting config options"
username=`config-get username`
public_key=`config-get public_key`

juju-log "Configuring irssi for user"
# We specify custom facts so that they're accessible in the manifest.
FACTER_username=$username FACTER_public_key=$public_key puppet apply $PWD/hooks/configured_state.pp

In this example, it is the configured state manifest that is more interesting (see configured_state.pp). It adds the user to the system, sets up byobu with an irssi window ready to go, and adds the given public ssh key enabling the user to login.

The same would go for other juju hooks (db-relation-changed etc.), which is quite neat – getting the best of both worlds: the charm user can still think in terms of deploying services, while the charm author can use puppets declarative syntax to define the machine states.

Next up: I hope to experiment with an optional puppet master for a real project (something simple like the Ubuntu App directory), so that

  1. a project can be deployed without the (probably private) puppet-master to create a close-to-production environment, while
  2. configuring a puppet-master in the juju config would enable production deploys (or deploys of exact replicas of production to a separate environment for testing).

If you’re interested in seeing the simple irssi charm, the following 2min video demos:

# Deploy an irssi service
$ juju deploy --repository=/home/ubuntu/mycharms  local:oneiric/irssi
# Configure it so a user can login
$ juju set irssi username=michael public_key=AAAA...
# Login to find irssi already up and running in a byobu window
$ ssh michael@new.ip.address

and the code is on Launchpad.

[1] Yes, irssi is not particularly useful as a juju service (as I don’t want multiple units, or relating it to other services etc.), but it suited my purposes for a simple experiment that also automates something I can use for working in the cloud.

[2] I’m not a puppet or juju expert, so if you’ve got any comments or improvements, don’t hesitate.


Filed under: juju, puppet, ubuntu

Read more
Michael

Working in the cloud

For a while now I’ve done all my development work and irc communication via SSH and byobu – I love the freedom of just grabbing my netbook (ie. switching computers) and heading to a cafe and picking up right where I left off (or even letting tests run while I walk).

The one pain-point was that I SSH’d into my own box at home – and if my home connection went (anything from a thunderstorm to a curious kid pulling a chord), I had to then work on my much slower netbook.

Now, after reading on an email thread that Aaron was doing something similar, I’m running a tiny cloud instance for my irc and development work, and documenting everything that I’m doing to get my dev environment just right – so that I can automate the setup (not really something for an ensemble formula – just a simple script).

That, and having all my music in the cloud for the past 6 months make switching computers and/or physical locations a non-obtrusive part of my work-day.

If anyone has small tips that make their own dev environment enjoyable, let me know!


Filed under: Uncategorized

Read more
Michael

What is the dream setup for developing and deploying Django apps? I’m looking for a solution that I can use consistently to deploy apps to servers where I may or may not have the ability to install system packages, or where I might need my app temporarily to use a newer version of a system-installed package while giving other apps running on the same server breathing space to update (think: updating a system-installed Django package on a server running four independent apps).

Specifically, the goals I have for this scenario are:

  • It should be easy to use for both development and deployment (using standard tools and locations so developers don’t need to learn the environment),
  • Updating any virtualenv environment should be automatic, but transparent (ie. if the pip requirements.txt changes, the next time I run tests, devserver or deployed server, it’ll automatically ensure the virtualenv is correct),
  • I shouldn’t have to wait unnecessarily for virtualenvs to be created (ie. if I make a change to the requirements to try a new version of a package, and then change it back, I don’t want to re-create the original virtualenv). Similarly, if I revert a deployment to a previous version, the previous virtualenv should still be available.
  •  For deployment, the virtualenv shouldn’t unnecessarily replace system python packages, but allow this as an option (ie. not a –no-site-packages virtualenv).

There are a lot of virtualenv/fabric posts out there for both development and deployment, and using a SHA of the requirements.txt seems an obvious way to go. What I ended up with for my project was this develop and deploy with virtualenv snippet which so far is working quite well (although I’m yet to try a deploy where I override system packages). If the deployed version is using virtualenv exclusively, the requirements.txt file can be shared, but otherwise it would just be a matter of including the requirements.txt for the deploy with the other configuration data (settings.py etc.).

If you can see any reasons why this is not a good idea, or improvements, please let me know!


Filed under: django, python

Read more
Michael

If you’re working on a project that bootstraps a development environment (using virtualenv or similar) it can be painful to bootstrap the environment for every bug/feature branch that you work on. There are lots of ways you can avoid this (local cache etc.), but if you’re using bzr to manage your branches, a single working directory for all your branches can solve the problem for you. Most bzr users will be familiar with this, but people new to bzr might benefit.

The one-time setup is pretty straight forward – create a repository with a local copy of trunk, and a branch of trunk for your new feature/bug:

$ cd dev/
$ bzr init-repo myproject
$ cd myproject
$ bzr branch lp:myproject trunk
$ bzr branch trunk myfeaturex

From now on, you can simply `cd trunk && bzr pull` when you want to update your local trunk [1]. Now we’ll create a light-weight checkout and bootstrap our environment there. I tend to call this directory ‘current_work’ but whatever works for you. Also, most of the projects I’m working on are using fabric for setting up the virtualenv and other tidbits, but replace that with your own bootstrap command:

$ bzr checkout --lightweight myfeaturex current_work
$ cd current_work && fab bootstrap

Assuming everything went well, you now have your development environment setup and are ready to work on myfeaturex. Make changes, commit etc., push to launchpad – it’ll all be for your myfeaturex branch. But what if half-way through, you need to switch and work on an urgent bug? Easy: commit (or shelve) any changes you’ve got for your current branch, then:

$ cd ..
$ bzr branch trunk criticalbugx
$ cd current_work
$ bzr switch ../criticalbugx  [2]

Edit 2011-05-23: See Ricardo’s comment – the above 4 lines can be replaced by his two.

That’s it – your current_work directory now reflects the criticalbugx branch – and is already bootstrapped [3], you can work away, commit push etc., and then switch back to your original branch with:

$ bzr switch ../myfeaturex

If you’re working on large branches, once you’ve got the above workflow, you may want to try next keeping large features reviewable with bzr pipelines.

[1] It’s a good idea to never do any work in your trunk tree… always work in a branch of trunk.
[2] The ../ is not actually required, but when you’ve lots of branches with long names, it means you can tab-complete the branch name.
[3] Of course, if one of your branches changes any virtualenv requirements or similar, you’ll need to re-bootstrap, but that isn’t so often in practise. Or sometimes a new version of a dependency is released and you won’t get it until you rebootstrap.


Filed under: bzr, python

Read more
Michael

Learning a new language is fun…finding new ways of thinking about old problems and simple ways of expressing new ideas.

As a small learning project for Golang, I set out the other day to experiment writing a simple form field validation library in my spare time – as it seems there is not yet anything along the lines of Django’s form API (email thread on go-nuts).

The purpose was to provide an API for creating forms that can validate http.Request.Form data, cleaning the data when it is valid, and collecting errors when it is not.

The initial version provides just CharField, IntegerField and RegexField, allowing form creation like:

    egForm := forms.NewForm(
        forms.NewCharField("description"),
        forms.NewIntegerField("purchase_count"))         

    egForm.SetFormData(request.Form)
    if egForm.IsValid() {
        // process the now populated egForm.CleanedData() and 
        // redirect.
    } else {
        // Use the now populated egForm.Errors map to report
        // the validation errors.
    }

The GoForms package is installable with `goinstall launchpad.net/goforms` or you can browse the goforms code on Launchpad (forms_test.go and fields_test.go have examples of the cleaned data and error). Let me know if you see flaws in the direction, or better ways of doing this in Go.

As a learning project it has been great – I’ve been able to use GoCheck for the tests, use embedded types (for sharing BaseField functionality – similar to composition+delegation without the bookkeeping) and start getting a feel for some of the subtleties of working with interfaces and other types in Go (this felt like all the benefits of z3c interfaces, without any of the overhead). Next I hope to include a small widget library for rendering basic forms/fields.


Filed under: golang, launchpad, testing

Read more