index.html

The index.html page is a static attachement, and the user starts by requesting it with their browser.

It has some small amount of static HTML, part of which creates a div for the javascript to put the data in.

Either inline, or in an included file, there is a small bit of javascript that will initialise the couchapp.

By default this will use the div with the id items, and will attach an evently widget to it.

evently

The evently widget that is attached will then either have an _init event, or a _changes event, either of which will be immediately run by evently.

This event will usually make a couchdb query to get data to transform to HTML and present to the user (see part three for how this works.)

Once that data has been displayed the user any combination of evently widgets or javascript can be used to make further queries and build an app that works however you like.

Previous installments

See part one, part two, and part three.

jQuery events

Our first detour is in to a little bit of background, the excellent javascript libarary that is heavily used in couchapps.

jQuery allows for events on elements in the DOM. There are standard events, such as "click" and "submit", but you are free to define your own.

These events are given a name, and you can then "trigger" them, and bind handlers to act when they are triggered.

By building up events from low level ones such as "click", to more-complex and app-specific ones such as "item purchased", you can break down your code in to smaller chunks, and have different parts of the page react to the same event, such as having the "buy" link disappear from the item that the user just bought, as well as having the total of the shopping cart update.

Events can also have data, or arguments, that travels with them. For instance the "item purchased" event could have the item that was purchased as the data, so that handlers can make use of it when they run.

evently

Now that we know something about jQuery events, we can look at something built on top of them, the "evently" library. This is a layer on top of jQuery that allows you build up your app from pieces that have a specific function, and communicate through events.

An evently "widget" can be bound to an element (or several elements if you want). The widget is a bunch of event handlers which can do anything you like, but have some conveniences built in for fetching data and updating the page based on the result.

When an event is triggered the handler you defined is run. If it is a simple javascript function then that function is run, and can do anything you like.

{click: function() {
        alert("You clicked!");
    }
}

Often though you want to update the element based on the action. evently has built in support for the "mustache" templating language, and if you specify a template in that syntax it will replace the current HTML of the element that it is attached to with the result of rendering that template.

{click:
    {
        mustache: "<div>You clicked!</div>"
    }
}

Which will put "You clicked!" in to the page instead of in an alert. What if you don't want to replace the current content, and just want to append another line? For that use the "render" option.

{click:
    {
        mustache: "<div>You clicked!</div>",
        render: "append"
    }
}

Which would put another "You clicked!" on the page every time you click. As well as "append" there is also "prepend", or really any jQuery method that you want to call.

Simply rendering a static template isn't going to be very useful though, usually you want something dynamic. For that use the "data" option, which can just be an object if you want, but that's still not going to be very dynamic either, so it can be a function that returns an object.

{click:
    {
        data: function(e) {
            return {name, "Bob"};
        },
        mustache: "<div>Hi {{name}}!</div>"
    }
}

The data function gets passed the event object from jQuery (so you can e.g. get the target of the event), and any data for the event too (so it could see what item you just bought).

That's all well and good, but it doesn't help us get data from couchdb in to the page. For that we need the opportunity to make a request to couchdb. We could just fall back to using one function to handle the event, but then we lose the integration with mustache. Therefore there is an "async" key that allows us to make an AJAX request and then use mustache on the result.

{click:
    {
        async: function(callback) {
            /* some code that does an async request, and then calls callback with the result */
        },
        data: function(resp) {
            /* Some code that processes the data from the async function to ready it for the template */
        },
        mustache: "A tempate that will be rendered with the result of the data function"
    }
}

Now, writing an async method to query a couchdb view is so common in couchapps that eventy has special support for it. The query key can either be a json structure that specifies a view and the arguments to it, or a function that returns such a structure based on things such as the query string in the URL.

There are two further functions that you will find helpful from time to time. The first is before that allows you to run some code before the rest of the process starts, and may do something such as trigger another event. The other is its partner after, which can do much the same things as before, but can also do things such as modify the HTML that is output.

Lastly there is another thing that can be done with the HTML that is output, specified with they selectors key. This allows you to perform an action on particular parts of the html. The keys of this structure are jQuery selectors that specify which elements the function will be applied to. For instance you can do something with all the divs in the output, or all the spans with a certain class, or the form with a particular id.

What you can do to those elements is basically unlimited, as you can run arbitrary javascript. However, there is built in support for specifying an evently widget, which will automatically be bound to each element that matches the selector. This nesting is one of the most powerful and useful features of evently, and one you should generally be using often. I will probably talk more about what nested widgets are useful for later.

Special evently events

evently has two special events. The first of these is _init. This event is triggered when the widget is created. This means you can dynamically pre-populate the element, or at least keep the inital state of the element with the rest of your code, rather than putting some in the HTML file and the rest in evently code.

The other special event is tied to couchdb, and is the _changes event, and is triggered whenever the database that the couchapp is in changes. This means that you can have elements on the page that dynamically update whenever the database changes, whether that is through user action, another user doing something, external scripts, or couchdb replication. This makes it very easy to write "live" pages that show updates without refreshes, and is very useful for some applications.

Currently _changes doesn't receive the modified documents, so it is normally just used to make another request to get the updated information, whether that be through async or view. If you wish to get the modified documents in order to update the page directly and reduce requests then you can write some custom code to do this.

Conclusion

As you have seen, evently is just a thin layer on top of jQuery concepts such as events and asynchronous events, with some conveniences for templating and interacting with couchdb.

This combination is well suited to the needs of at least simple and moderately complex couchapps, while still being very powerful, and allowing you to fall back to custom javascript at any point.

You can find more about evently at the couchapp site.

See part one and part two.

jQuery events

Our first detour is in to a little bit of background, the excellent javascript libarary that is heavily used in couchapps.

jQuery allows for events on elements in the DOM. There are standard events, such as "click" and "submit", but you are free to define your own.

These events are given a name, and you can then "trigger" them, and bind handlers to act when they are triggered.

By building up events from low level ones such as "click", to more-complex and app-specific ones such as "item purchased", you can break down your code in to smaller chunks, and have different parts of the page react to the same event, such as having the "buy" link disappear from the item that the user just bought, as well as having the total of the shopping cart update.

Events can also have data, or arguments, that travels with them. For instance the "item purchased" event could have the item that was purchased as the data, so that handlers can make use of it when they run.

evently

Now that we know something about jQuery events, we can look at something built on top of them, the "evently" library. This is a layer on top of jQuery that allows you build up your app from pieces that have a specific function, and communicate through events.

An evently "widget" can be bound to an element (or several elements if you want). The widget is a bunch of event handlers which can do anything you like, but have some conveniences built in for fetching data and updating the page based on the result.

When an event is triggered the handler you defined is run. If it is a simple javascript function then that function is run, and can do anything you like.

::

{click: function() {

alert("You clicked!");

System Message: WARNING/2 (<string>, line 57)

Block quote ends without a blank line; unexpected unindent.

}

System Message: WARNING/2 (<string>, line 58)

Definition list ends without a blank line; unexpected unindent.

}

Often though you want to update the element based on the action. evently has built in support for the "mustache" templating language, and if you specify a template in that syntax it will replace the current HTML of the element that it is attached to with the result of rendering that template.

::

{click:

{

mustache: "<div>You clicked!</div>"

System Message: WARNING/2 (<string>, line 70)

Definition list ends without a blank line; unexpected unindent.

}

System Message: WARNING/2 (<string>, line 71)

Definition list ends without a blank line; unexpected unindent.

}

Which will put "You clicked!" in to the page instead of in an alert. What if you don't want to replace the current content, and just want to append another line? For that use the "render" option.

::

{click:

{

mustache: "<div>You clicked!</div>", render: "append"

System Message: WARNING/2 (<string>, line 82)

Definition list ends without a blank line; unexpected unindent.

}

System Message: WARNING/2 (<string>, line 83)

Definition list ends without a blank line; unexpected unindent.

}

Which would put another "You clicked!" on the page every time you click. As well as "append" there is also "prepend", or really any jQuery method that you want to call.

Simply rendering a static template isn't going to be very useful though, usually you want something dynamic. For that use the "data" option, which can just be an object if you want, but that's still not going to be very dynamic either, so it can be a function that returns an object.

::

{click:

{

data: function(e) {

return {name, "Bob"};

System Message: WARNING/2 (<string>, line 99)

Definition list ends without a blank line; unexpected unindent.

}, mustache: "<div>Hi {{name}}!</div>"

System Message: WARNING/2 (<string>, line 101)

Definition list ends without a blank line; unexpected unindent.

}

System Message: WARNING/2 (<string>, line 102)

Definition list ends without a blank line; unexpected unindent.

}

The data function gets passed the event object from jQuery (so you can e.g. get the target of the event), and any data for the event too (so it could see what item you just bought).

That's all well and good, but it doesn't help us get data from couchdb in to the page. For that we need the opportunity to make a request to couchdb. We could just fall back to using one function to handle the event, but then we lose the integration with mustache. Therefore there is an "async" key that allows us to make an AJAX request and then use mustache on the result.

See part one and part two.

How it works

At it's core couchapp is a rather simple tool. It walks a filesystem tree and assembles the things that it finds there in to a JSON document.

For instance, if it finds a directory at the root of the tree called _attachments it puts the content of each file there in to a list which it puts in to the dict with an "_attachments" key, Which is one of the ways that couchdb accepts document attachments. Therefore if you have an _attachments/index.html file in your tree it will be attached to your design document when the JSON structure is sent to couchdb.

This continues across the tree, so the contents of the "views" directory will become the "views" key of the documents, which is how you do map/reduce queries on the database.

couchapp has various conventions for dealing with files. For instance if it finds a ".js" file it treats it as a javascript snippet which will be encoded in to a string in the resulting document. ".html" files outside of the "_attachments" directory will also be encoded as strings. If it finds a ".json" file then it treats it as literal JSON that will be embebedded.

This way it builds up the JSON structure that a couchapp expects, and will send it to the couchdb of your choice when it is done.

In addition to this functionality the tool can also generate you a skeleton app, and also add new pieces to your app, such as new views.

Getting It

couchapp is a python tool, so you can install it using pip or similar. However, Ubuntu users can install it from a PPA (yay for daily builds with recipes!).

Using It

To use it run

couchapp generate myapp

which will create you a new skeleton in myapp.

cd myapp
ls

You will see for instance the _attachments and views directories, and an _attachments/index.html.

To get your app in to couchdb you can run

couchapp push http://localhost:5984/mydb

and it will tell you the URL to visit to see your new app.

If you want to use desktopcouch you can run

couchapp push desktopcouch://

though I think it has a bug that it prints the wrong URLs when pushing to desktopcouch.

Once you have looked at the HTML generated by your app you should look at the design document that couchapp created. Go to

http://localhost:5984/_utils

or

xdg-open ~/.local/share/desktop-couch/couchdb.html

if you are using desktopcouch.

Click to the mydb database and you will see a document called _design/myapp. Click on this and you will see the content of the design document; you are looking at a couchapp in its raw form.

If you compare what is in that design document with what is in the myapp directory that the tool created you should start to see how it generates it from the filesystem.

Now try making a change on the filesystem, for instance edit _attachments/index.html and put your name somewhere in the body. Then push again, running

couchapp push http://localhost:5984/mydb

and refresh the page in your browser and you should see the change. (Just click on index.html in the design document to get back to viewing your app from there).

I will go in to more detail about the content of the couchapp that was generated for you in another post.

See the previous installment.

Architecture

Firstly something about the pieces that make up a couchapp (or at least those that the tool and documentation recommend,) and the way that they all fit together.

At the core is the couchdb database itself. It is a collection of "documents", each of which can have attachments. Some of these documents are known as "design documents," and they start with a prefix of "_design." Design documents can have "view" functions, and various other special fields that can be used to query or manipulate other documents.

A couchapp is a design document with an attachment, usually called index.html. These attachments are served directly by couchdb and can be accessed at a known URL. You can put anything you like in that html file, and you could just have a static page if you wanted. Usually however though it is is an HTML page that uses javascript in order to display the results of queries on the database. The user will then access the attachment on the design document, and will interact with the resulting page.

In theory you can do anything you like in that page, but it is usual to make use of standard tools in order to query the database and provide information and opportunity for interaction to the user.

The first standard tool is jQuery, with a couple of plugins for working with couchdb and couchapps specifically. These allow for querying views in the database and acting on the results, retrieving and updating documents, and plenty more.

In addition the couchapp tool sets you up with another jQuery plugin called "evently", which is a way to structure interactions with jQuery, and change the page based on various events. I will go in to more detail about how evently works in a later post.

In addition to all the client-side tools for interacting with the database, it is also possible to make use of couchdb features such as shows, lists, update handlers validation functions in order to move some of the processing server-side. This is useful for various reasons, including being more accessible, allowing search engines to index the content, and not having to trust the client not to take malicious actions.

The two approaches can be combined, and you can prototype with the client-side tools, and then move some of the work to the server-side facilities later.

Stay tuned for more on how a simple couchapp generates content based on what is in the db.

Ubuntu Distributed Development

This isn't the catchiest name for a project ever, and has an unfortunate collision with an Debian project, also shortened to "UDD." However, the aim is for this title to become a thing of the past, and this just to be the way things are done.

This effort is firstly about getting Ubuntu to use Bazaar, and a suite of associated tools, to get the packaging work done. There are multiple reasons for this.

First, and most simply, is to give developers the power of version control as they are working on Ubuntu packages. This is useful for both the large things and the small. For instance I sometimes appreciate being able to walk through the history of a package, comparing diffs here and files there when debugging a complex problem. Sometimes though it's just being able to "bzr revert" a file, rather than having to unpack the source again somewhere else, extracting the file and copying it over the top.

There are higher purposes with the work too. The goal is to link the packaging with the upstream code at the version control level, so that one flows in to the other. This has practical uses, such as being able to follow changes as they flow upstream and back down again, or better merging of new upstream versions. I believe it has some other benefits too, such as being able to see the packages more clearly as what they are, a branch of upstream. We won't just talk about them being that, but they truly will be.

Some of you will be thinking "that's all well and good, but <project> uses git," and you are absolutely right. Throughout this work we have had two principles in mind, to work with multiple systems outside of Ubuntu, and to provide a consistent interface within Ubuntu.

Due to the way that Ubuntu works an Ubuntu developer could be working on any package next. I would really like it if the basics of working with that package were the same regardless of what it was. We have a lot of work to do on the packaging level to get there, but this project gets this consistency on the version control level.

We can't get everyone outside of Ubuntu to follow us in this though. We have to work with the system that upstream uses, and also to work with Debian in the middle. This means that we have to design systems that can interface between the two, so we rely a lot on Launchpad's bzr code imports. We also want to interface at the other end as well, at "push" time. This means that if an Ubuntu developer produces a patch that they want to send upstream they can do that without having to reach for a possibly different VCS.

Thanks mainly to the work of Jelmer Vernooij we are doing fairly well at being able to produce patches in the format appropriate for the upstream VCS, but we still have a way to go to close the loop. The difficultly here is more around the hundreds of ways that projects like to have patches submitted, whether it is a mailing list or a bug tracker, or in some other form. At this stage I'd like to provide the building blocks that developers can put together as appropriate for that project.

Daily package builds

Relatedly, but with slightly different aims, I have been working on a project in conjunction with the Launchpad developers to allow people to have daily builds of their projects as packages.

Currently there is too often a gap between using packaged versions of a project, and running the tip of that project daily. I believe that there are lots of people that would like to follow the development of their favourite projects closely, but either don't feel comfortable building from the VCS, or don't want to go through the hassle.

Packages are of course a great way to distribute pre-compiled software, so it was natural to want to provide builds in this format, but I'm not aware of many projects doing that, aside from those which fta provides builds for. Now that Launchpad provides PPAs and code imports, and the previous project provides imports of the packaging of all Debian and Ubuntu packages in to bzr, all the pieces are there in order to allow you to produce packages of a project automatically every day.

This is currently available in beta in Launchpad, so you can go and try it out, though there are a few known problems that we are working on until it will be as pleasant as we want.

This has the potential to do great things for projects if used correctly. It can increase the number of people testing fresh code and giving feedback by orders of magnitude. Also, just building the packages acts as a kind of continuous integration, and can provide early warning of problems that will affect the packaging of the project. Also, they provide an easy way for people to test the latest code if a bug is believed to be fixed.

Obviously there are some dangers associated with automatic builds, but if they are used by people who know what they are doing then it can help to close the loop between users and developers.

There are also many more things that can be done with this feature by people with imagination, so I'm excited to see what directions people will take it in.

Fixing things

Aside from these projects, I was also given some time to work on Ubuntu itself, but without long-term projects to ship. That meant that I was able to fix things that were standing in my way, either in the way of the above projects, or just hampering my use of Ubuntu, or fix important bugs in the release.

In addition I took on smaller projects, such as getting kerneloops enabled by default in Ubuntu. While doing this I realised that the user experience of that tool could be improved a lot for Ubuntu users, as well as allowing us to report the problems caught by the tool as bugs in Launchpad if we wished.

I really enjoyed having this flexibility, as it allowed me to learn about many areas of the Ubuntu system, and beyond, and also played to my strengths of being able to quickly dive in to a new codebase and diagnose problems.

I think that in my own small way, each of these helped to improve Ubuntu releases, and in turn the projects that Ubuntu is built from.

Sponsoring

While I'm sorry to say that other demands have pulled my code review time in to other projects, I did used to spend a lot of time reviewing and sponsoring changes in to Ubuntu.

I highlight this mainly as another chance to emphasise how important I think code review is, especially when it is review of code from people new to the project. It improves code quality, but is also a great opportunity for mentoring, encouraging good habits, and helping new developers join the project. I hope that my efforts in this are had a few of these characteristics and helped increase the number of free software developers. Oh how I wish there were more time to continue doing this.

Linaro

I've now been started working on the Linaro project, specifically in the Infrastructure team, working on tools and Infrastructure for Linaro developers and beyond. I'm not one to be all talk and no action, so I won't talk to much about what I am working on, but I would like to talk about why it is important.

Firstly I think that Linaro is an important project for Free Software, as it has the opportunity to lead to more devices being sold that are built on or entirely free software, some in areas that have historically been home to players that have not been good open source citizens.

Also, I think tools are an important area to work on, not just in Linaro. They pervade the development experience, and can be a huge pain to work with. It's important that we have great tools for developing free software so as not to put people off. Developers, volunteers and paid, aren't going to carry on too long with tools that cause them more problems than they are worth, and not all are going to persist because they value Free Software over their own enjoyment of what they do.

The Problem

For the Ubuntu Distributed Development initative we have a need for a process that imports packages in to bzr on an ongoing basis as they are uploaded to Ubuntu. This is so that we can have a smooth transition rather than a flag day where everyone switches. For those that are familiar with them think Launchpad's code imports but with Debian/Ubuntu packages as the source, rather than a foreign VCS.

This process is required to watch for uploads to Debian and Ubuntu and trigger a run to import that upload to the bzr branches, pushing the result to LP. It should be fast, though we currently have a publication delay in Ubuntu that means we are used to latencies of an hour, so it doesn't have to be greased lightning to get acceptance. It is more important to be reliable, so that the bzr branches can be assumed to be up to date, that is crucial for acceptance.

It should also keep an audit trail of what it thinks is in the branches. As we open up write access to the resulting branches to Ubuntu developers we can not rely on the content of the branches not being tampered with. I don't expect this will ever be a problem, but I wanted to ensure that we could at least detect tampering, even if we couldn't know exactly what had happened by keeping private copies of everything.

The Building Blocks

The first building block of the solution is the import script for a single package. You can run this at any time and it will figure out what is unimported, and do the import of the rest, so you can trigger it as many times as you like without worrying that it will cause problems. Therefore the requirement is only to trigger it at least once when there has been an upload since the last time it was run, which is a nicer requirement than "exactly once per upload" or similar.

However, as it may import to a number of branches (both lucid and karmic-security in the case of a security upload, say), and these must be consistent on Launchpad, only one instance can run at once. There is no way to do atomic operations on sets of branches on Launchpad, therefore we use locks to ensure that only one process is running per-package at any one time. I would like to explore ways to remove this requirement, such as avoiding race conditions by operating on the Launchpad branches in a consistent manner, as this would give more freedom to scale out.

The other part of the system is a driver process. We use separate processes so that any faults in the import script can be caught in the supervisor process, with the errors being logged. The driver process picks a package to import and triggers a run of the script for it. It uses something like the following to do that:

write_failure(package, "died")
try:
    import(package)
except:
    write_failure(packge, stderr)
finally:
    remove_failure(package)

write_failure creates a record that the package failed to import with a reason. This provides a list of problems to work through, and also means that we can avoid trying to import a package if we know it has failed. This ensures that previous failures are dealt with properly without giving them a chance to corrupt things later.

Queuing

I said that the driver picks a package and imports it. To do this it simply queries the database for the highest priority job waiting, dispatching the result, or sleeping if there are no waiting jobs. It can actually dispatch multiple jobs in parallel as it uses processes to do the work.

The queue is filled by a couple of other processes triggered by cron. This is useful as it means that further threads are not required, and there is less code running in the monitor process, and so less chance that bugs will bring it down.

The first process is one that checks for new uploads since the last check and adds a job for them, see below for the details. The second is one that looks at the current list of failures and retries some of them automatically, if the failure looks like it was likely to be transient, such as a timeout error trying to reach Launchpad. It only retries after a timeout of a couple of hours has elapsed, and also if that package hasn't failed in that same way several times in a row (to protect against e.g. the data that job is sending to LP causing it to crash and so give timeout errors.)

It may be better to use an AMQP broker or a job server such as Gearman for this task, rather that just using the database. However, we don't really need any of the more advanced features that these provide, and already have some degree of loose-coupling, so using fewer moving parts seems sensible.

Reacting to new uploads

I find this to be a rather neat solution, thanks to the Launchpad team. We use the API for this, notably a method on IArchive called getPublishedSources(). They key here is the parameter "created_since_date". We keep track of this and pass it to the API calls to get the uploads since the last time we ran, and then act on those. Once we processed them all then we update the stored date and go around again.

This has some nice properties, it is a poll interface, but has some things in common with an event-based one. Key in my eyes is that we don't have to have perfect uptime in order to ensure we never miss events.

However, I am not convinced that we will never get a publication that appears later than one that we have dealt with, but that reports an earlier time. If this happens we would never see it. The times we use always come from LP, so don't require synchronised clocks between the machine where this runs and the LP machines, but it could still happen inside LP. To avoid this I subtract a delta when I send the request, so assuming the skew would not be greater than that delta we won't get hit. This does mean that you repeatedly try and import the same things, but that is just a mild inefficiency.

Synchronisation

There is a synchronisation point when we push to Launchpad. Before and after this critical period we can blow away what we are doing with no issues. During it though we will have an inconsistent state of the world if we did that. Therefore I used a protocol to ensure that we guard this section.

As we know locking ensures that only one process runs at a time, meaning that the only way to race is with "yourself." All the code is written to assume that things can go down at any time as I said, the supervisor catches this and marks the failures, and even guards against itself dying. Therefore when it picks back up and restarts the jobs that it was processing before dying it needs to ensure that it wasn't in the critical section.

To do this we use a three-phase commit on the audit data to accomany the push. When we are doing the import we track the additions to the audit data separately from the committed data. Then if we die before we reach the critical section we can just drop it again, returning to the inital state.

The next phase marks in the database that the critical section has begun. We then start the push back. If we die here we know we were in the critical section and can restart the push. Only once the push has fully completed do we move the new audit data in to place.

The next step cleans up the local branches, dying here means we can just carry on with the cleanup. Finally the mark that we are in the critical section is removed, and we are back to the start state, indicating that the last run was clean, and any subsequent run can proceed.

All of this means that if the processes go down for any reason, they will clean up or continue as they restart as normal.

Dealing with Launchpad API issues

The biggest area of operational headaches I have tends to come from using the Launchpad API. Overall the API is great to have, and generally a pleasure to use, but I find that it isn't as robust as I would like. I have spent quite some time trying to deal with that, and I would like to share some tips from my experience. I'm also keen to help diagnose the issues further if any Launchpad developers would like so that it can be more robust off the bat.

The first tip is: partition the data. Large datasets combined with fluctuating load may mean that you suddenly hit a timeout error. Some calls allow you to partition the data that you request. For instance, getPublishedSources that I spoke about above allows you to specify a distro_series parameter. Doing

distro.main_archive.getPublishedSources()

is far far more likely to timeout than

for s in distro.series:
    distro.main_archive.getPublishedSources(distro_series=s)

in fact, for Ubuntu, the former is guaranteed to timeout, it is a lot of data.

This is more coding, and not the natural way to do it, therefore it would be great if launchpadlib automatically partioned and recombined the data.

The second tip is: expect failure. This one should be obvious, but the API doesn't make it clear, unlike something like python-couchdb. It is a webservice, so you will sometimes get HTTP exceptions, such as when LP goes offline for a rollout. I've implemented randomized exponential backoff to help with this, as I tend to get frequent errors that don't apparently correspond to service issues. I very frequently see 502 return codes, on both edge and production, which I believe means that apache can't reach the appservers in time.

Summary

Overall, I think this architecture is good, given the synchronisation requirements we have for pushing to LP, without those it could be more loosely coupled.

The amount of day-to-day hand-holding required has reduced as I have learnt about the types of issues that are encountered and changed the code to recognise and act on them.