Canonical Voices

What Canonical ISD talks about

Stuart Metcalfe

It’s been a long time in development, but we’re finally drawing close to releasing Paypal support in Ubuntu Pay, the payment service behind Ubuntu Software Centre. Here are a couple of screenshots to whet your appetite…

We’re aiming to launch this new feature before Christmas.

Read more
Stuart Metcalfe

Anyone who uses the OpenID service on Ubuntu SSO (or Launchpad) to log in to sites on the Internet will know that we currently only send your username unless the site is registered with us. This makes logging in with Ubuntu SSO using OpenID less-than-great if the site needs your name or email address, for instance, to create an account for you. Next week we’re rolling out an update which removes this restriction and gives you control over which personal data you want to share.

I’ve personally wanted this feature for a while so it’s great to see it finally land.

Read more
Stuart Metcalfe

Ubuntu developer portal preview

In Ubuntu 10.10 we introduced the ability to purchase software through the Ubuntu Software Centre.  Because of the existing manual process to get new ISVs (independent software vendors) signed up, however, the number of applications available for purchase has grown slower than we’d like.  The solution we’ve selected to this problem is to have a standard developer program for ISVs to sign up to and to provide an online self-service portal where they can manage their applications.  As we’re about to launch the first preview of this portal, we wanted to show the experience a first-time user will have from agreeing to the developer terms to actually having their new software appear in the Software Centre.

For this initial release, access is limited to ISVs we have an existing relationship with.  Once we’ve ironed out the early wrinkles, we’ll be opening the doors to everyone.  If you would like to hear when the service launches publicly, keep an eye on the Canonical blog and

Read more
Anthony Lenton

The great Django command-extensions app gives you an easy way to integrate the Werkzeug debugger into your Django site.

For if you’ve never used this debugger before, imagine the regular Django technical details debugging page, with all same super useful information — environment variables, sql queries, and a full traceback with even the local variables and a bit of code context for each frame.
Now imagine you could have a pdb prompt running within each frame of the traceback, actually running interactively via ajax within the frames that you see.  That’s the kind of raw power Werkzeug gives you.

A big disclaimer though: this is not for running on production sites.  You’ll be giving anybody that happens to stumble upon a failure page an interactive (unrestricted) python terminal running with the web server’s permissions on your box.

So, this is just what out the box django-command-extension’s runserver_plus gives you.  With a bit of tweaking, here are a couple of other neat things you can do:

Make your server multithreaded

There are times when you need a multithreaded server in your development environment.  I’ve seen arguments about this being a smell for bad design, so here ar two reasonable scenarios I’ve come across:

  • Developing a web API, you might have a view that provides a test consumer for the API, not intended to be available on production.  The test consumer view will make a (blocking) call out to the API that’s provided by the same server, hanging everything if you don’t have multithreadedness.
  • Or, in larger systems where you have more than one application server, you might have a view or two that do a bit of communication between servers (to see if everything’s ok, to provide a neat server status screen), that you’ll need to test calling out to the same server during development.

For whatever reason it may be, if you need to enable threading, you can just set threaded=True when you call Werkzeug’s run_simple function:

Debug your whole wsgi app instead of just Django

If you’re running Django as part of a larger wsgi stack (a non-Django web api, or some authentication wsgi middleware), you probably would love to be able to run the whole stack in your development server, and even have those same great debugging features available for everything, not only Django.
You can do this by modifying the wsgi app that Werkzeug debugs:

It’s something with raw power that keeps getting better the more you look…

Read more
Danny Tamez

Tsung Quick Start Part 2

In part 1 of this article ( we did sort of a hello world for a Tsung load test.  This time we’re going to continue by getting into more of the details of how Tsung works and how to use it for real world tests.

Using the Recorder

More likely than not, the application you want to test consists of lots of urls with lots of query string values or form post values.  Typing this in by hand is not practical and would take way too long.  What Tsung provides is a a proxy recorder that you can set up to record all of the http calls you are making to and receiving from you application.  The end result is a Tsung “session” in an xml file.  You can then tweak the file however you want to better suit your tests.

Here’s how to set it up.
1) Configure your browser to use a proxy server at http://localhost:8090.  For firefox you can follow the instructions at window – Advanced panel?as=u#advanced_network .
2) Start the proxy recorder

$ tsung-recorder start

3) Point your browser to your application and go through one or more scenarios that you would like to group together as a teset scenario.
4) When you’re ready to stop recording simply stop the recorder.

$tsung-recorder stop

You now have a Tsung session xml file.  You can now change things like thinktimes or change values you typed in to be dynamically generated.  In order to use the file in your main tsung.xml file all you need to is include it as an xml entity and then reference it inside the sessions tag.

<?xml version="1.0"?>
<!DOCTYPE tsung SYSTEM "/usr/share/tsung/tsung-1.0.dtd" [
<!ENTITY new_accounts SYSTEM "./new_accounts.xml">
<!ENTITY log_out SYSTEM "./log_out.xml">
<!ENTITY log_in SYSTEM "./log_in.xml">
<!ENTITY auth_logged_in SYSTEM "./auth_already_logged_in.xml">
<!ENTITY auth_logged_out SYSTEM "./auth_not_logged_in.xml">
<!ENTITY api_register SYSTEM "./api_register.xml">
<!ENTITY api_server SYSTEM "./api_server.xml">

<tsung loglevel=”debug” dumptraffic=”true” version=”1.0″>

<session name="web" probability="70" type='ts_http'>

Creating Load

Load is configured in Tsung inside the ‘load’ tag. This tag has an optional ‘duration’ attribute that can specify a maximum time for the test to run regardless of whether or not all requests have been processed. Inside of the load tag you specify one or more ‘arrivalphase’ tags each with a specified duration.  Finally, within each arrival phase you specify via the ‘users’ tag the rate at which users are generated for that phase.

<arrivalphase phase="1" duration="6" unit="minute">
<users interarrival="1.0" unit="second"/>
<arrivalphase phase="2" duration="5" unit="minute">
<users interarrival="0.8" unit="second"/>

Some caveats:
Keep in mind that the number of concurrent users will depend on how long it takes for each user’s session to be processed.  If sessions are long then there will be overlap between arrival phases. Likewise, request made per second is not a direct correlation of the user arrival rate as each sessioin  may have thinktimes between each request that it makes. Additionally, as sessions usually have more than one request, each user created will overlap subsequent requests with new users’ requests.

One of the reasons that you would choose Tsung over other load testing tools is that it will generate much higher load than other tools.  Being built on Erlang, one server can typically generate tens of thousands of users.  However, you can also generate even more load if necessary by running a cluster of servers.

Using Clusters

It is easy to use Tsung with one client or several clients and just as easy to use it with one or more servers.

<client host="&client_host_1;" use_controller_vm="true" maxusers="2000" cpu="2" weight="2"/>
<client host="&client_host_2;" use_controller_vm="true" maxusers="2000" weight="1"/>
<client host="&client_host_3;" use_controller_vm="true" maxusers="2000" weight="1"/>
<server host="&provider_host_1;" port="80" type="tcp"></server>
<server host="&provider_host_2;" port="80" type="tcp"></server>

In the above example Tsung will send twice as many requests to client 1 than it sends to client 2 or 3.  Another way of looking at is that 1/2 of the requests will come from client 1, 1/4 from client 2 and 1/4 from client 3.  The weight will be evenly distributed between the two servers via simple round robin.

Using Dynamic Variables

Often you will have situations where you can’t hardocode parameters into your session files.  For instance, sometimes you need to use a value in one request that was generated by the previous request.  An example of this is if your application uses Cross Site Request Forgery tokens.

<thinktime min="2" max="10" random="true"/>
<transaction name="display_log_in">
<dyn_variable name="csrfmiddlewaretoken"/>
<http url='&provider_url;' version='1.1' method='GET'/>
<thinktime min="7" max="30" random="true"/>
<transaction name="log_in">
<request subst="true">
<http url='&provider_url;/+login' version='1.1'
content_type='application/x-www-form-urlencoded' method='POST'/>
<http url='&provider_url;' version='1.1' method='GET'/>

In the example above a form is displayed in the “display_log_in” transaction.  A hidden csrf token is in that pages login form.  We are capturing that value via the ‘dyn_variable’ tag in the first requeset.  We need that value in the second requeset and use  it in the ‘content’ attribute as ‘%%_csrfmiddlewaretoken’. When using dynamic variables, be sure to use ‘subst=”true”‘ in the request tag or the variables will be interepreted as literals.

Sometimes the value you need is on the page but is not a form field.  You can use a regex to match it and store it in a variable.

<dyn_variable name="janrain_nonce" regexp='janrain_nonce=\([^\"]*\)'/>
<http url='&consumer_url;/verify?openid_identifier=&provider_url;/'
version='1.1' method='GET'/>

It’s also easy to share variables across requests as they can be defined for a session.

<session name="web" probability="70" type='ts_http'>
<setdynvars sourcetype="random_string" length="8">
<var name="password"/>
<setdynvars sourcetype="random_string" length="5">
<var name="first"/>
<setdynvars sourcetype="random_string" length="5">
<var name="last"/>
<setdynvars sourcetype="random_string" length="5">
<var name="domain"/>
<setdynvars sourcetype="random_string" length="5">
<var name="email"/>
<setdynvars sourcetype="random_string" length="16">
<var name="secret"/>

If you need something more powerful you can call custom Erlang code defined in an external module,

<setdynvars sourcetype="eval"
code="fun({Pid, DynVars}) -> sso:auth_header(DynVars) end.">
<var name="authorization_header"/>

or evaluate Erlang code inline,

<var name="assoc_handle"/>

or by looking up a value from an external file.

<option name="file_server" id="accounts" value="../accounts.csv"/>
<session name="all" probability="100" type='ts_http'>
<setdynvars sourcetype="file" fileid="accounts" delimiter="," order="iter">
<var name="email" />
<var name="domain" />
<var name="password" />

Looping and Branching

Tsung allows you to use simple branching and looping in your Tsung files. For example you can bracket a request inside an if tag so that it only is called under certain conditions,

<if var="is_super_user" eq="1">
<request> <http url="/delete_account"></http> </request>

or loop a certain request several times for the same user.

<for from="1" to="5" incr="1" var="product_id">
<request> <http url="/product?id=%%_product_id%%"></http> </request>

There is still much more that Tsung can do and if you’re interested in using it there is a great community of people around it that can be of assistatnce.  I hope these two articles will be of some help to those of you starting with Tsung and that it becomes a great tool in your toolbox.  Good luck!

Read more
Danny Tamez

What is Tsung and what is load testing?

One of the tools we use on the ISD team is Tsung.  We use it to load test our web based applications in our staging environment.  In this article we’re going to go through all the steps necessary to setup and run a simple as possible load test.  Later we’ll look at how to configure Tsung and your environment for more complex scenarios.

Load testing is the process of examining a system (such as an application or a device) under higher than normal load in order to see how it performs under those conditions.  It is a very important part of testing any application where load amounts are variable and unpredictable. Load testing is a huge topic by itself but in brief here a few things that load testing can do for you:

  • Benchmark the performance of the application under load so that it can be compared to previous or future releases.
  • Identify trouble spots in the application that break only when under high load.
  • Help managers more accurately anticipate and plan for resources such as hardware, employees, bandwidth etc.
  • Test whether the application is going to perform within its specifications under projected usage.

Tsung is a distributed, multi protocol load testing framework that can be used to stress web and database servers.  Tsung has many things in common with other load testing suites but what distinguishes it from the others is its ability to produce very high load with limited resources.  Tsung is written in Erlang, (a language built for high concurrency and fault tolerance) but uses simple XML files for its configuration of load testing scenarios.

Let’s Try it Out

The best way to get familiar with Tsung is just to try it out.  It doesn’t take long at all to get a working Tsung load test up and running.


My examples were all run on Kubuntu 10.10 so you may have to adapt tweak these steps a little for other distributions.

The first thing you’ll need to do is install erlang-nox on all of the servers that will be tested or used as clients.

 $sudo apt-get install erlang-nox

You should now be able to run


from your terminal and be in an Erlang shell. Type ‘Control-C’ and then ‘a’ to get out of the Erlang shell.

Next you’ll need need to install Tsung on your client machine.  Tsung has not yet been packaged for Ubuntu but a Debian package is available.

 $sudo dpkg -i tsung_1.3.3-1_all.deb

The following are needed for generating reports from the collected data:

 $sudo apt-get install gnuplot perl libtemplate-perl python-matplotlib

You should now be able to run

$tsung -v

You should then see ‘Tsung version 1.3.3’ displayed.

In order for Tsung to communicate with the other servers in the cluster and obtain statistics from them, password-less ssh must be enabled between the client machine(s) and the server machine(s).

 $ssh-copy-id -i ~/.ssh/ username@host

If for the purposes of experimenting you’re just running both client and server on one machine the above is not necessary.


You will recall that Tsung uses XML files for its configuration.  Here is an example of pretty much the smallest Tsung configuration file:

<?xml version=”1.0″?>
<!DOCTYPE tsung SYSTEM “/usr/share/tsung/tsung-1.0.dtd”[]>
<tsung loglevel=”info”>
<!– Minimal tsung configuration file –>
<client host=”localhost”/>
<server host=”localhost” port=”8000″ type=”tcp”/>
<arrivalphase phase=”1″ duration=”1″ unit=”minute”>
<users interarrival=”1″ unit=”second”/>
<session name=”foo” probability=”100″ type=”ts_http”>
<http url=’/’ version=’1.1′ method=’GET’/>

Save the above file and give it a name, for example minimal.XML.

Basically before it can run Tsung has to know 4 things: the IP address of the machine on which the client is running, the IP address of the machine on which the server is running, how many users to create, and what each user is to do.  In the above example both the client and the server are running on localhost, a new user will created each second for one minute, and each user will run one get request of the home page of the tested site.

Before we can run our simple load test we’ll need to run our server. I am running a Django application that’s listening on port 8000 so you’ll need to change your server line to match your environment.  Once your server is running you’re ready to start.

 $sudo tsung -r erl -f minimal.xml start

Note: Normally you do not need to run Tsung via sudo, but the very first time that it is run it needs to create files that require root access.

If Tsung ran without any problems you should have seen displayed the path to the directory where the log files for this load test can be found. CD to that directory and generate your report like this:


You can now view your report from a browser:

 $firefox report.html &

Congratulations!  You’ve just setup and run your first Tsung load test. Next time we’ll look at more complex scenarios and delve deeper into Tsung’s capabilities.

Read more
Ricardo Kirkner

What is configglue?

configglue is a library that glues together Python’s optparse.OptionParser and
ConfigParser.ConfigParser, so that you don’t have to repeat yourself when you
want to export the same options to a configuration file and a commandline

The main features of configglue are:

  • ini-style configuration files
  • schema-based configuration
  • commandline integration
  • configuration validation

Why would I want to use configglue?

Some of the benefits of using configglue are that it allows you to:

  • separate configuration declaration (which options are available) from
    definition (what value does each option take)
  • validate configuration files (there are no required options missing, prevent
    typos in option names, assert each option value is of the correct type)
  • use standards-compatible configuration files (standard ini-files)
  • use standard types out of the box (integer, string, bool, tuple, list, dict)
  • create your own custom types beyond what’s provided in the library
  • easily support commandline integration
  • override options locally by using several configuration files (useful for
    separating configuration files for different environments)

configglue and django-configglue are already available in Ubuntu 10.10 (Maverick), so they can be installed via apt-get. configglue should already be installed if you have the desktop edition, as it’s being used by Ubuntu One’s client.

Who else is using configglue?

  • Ubuntu Pay
  • Ubuntu Software Center
  • Ubuntu Single Sign On
  • Ubuntu One

Got curious?

You can find a quickstart guide for configglue on and you can get its code at

As an additional bonus, there is another project called django-configglue
which allows you to use all the benefits of configglue on your Django
projects. You can find a quickstart guide for django-configglue on and you can get its code at

Read more
Michael Foord

After the release of Ubuntu Maverick Meerkat we have a brief time of housekeeping in the ISD team, where we work on clearing our technical debt and implementing improvements to our development processes. One of these improvements has been getting continuous integration setup for some of our projects. Continuous integration means not just having an automated test suite, but having an automated system for running the tests (continuously)

We have settled on Hudson as our CI server. Despite being a Java based tool it is a popular choice in the Python world, mainly because Hudson is both easy to install / configure and provides an attractive and powerful web based interface out-of-the-box.

You can use Hudson to report test run pass or fail, and view the console output from a test run, with virtually no work at all. Simply provide a shell command for running your tests and off you go. Hudson works best when your test run generates an XML description of the test run in a poorly specified (but widely implemented) format called JUnit-XML. This is the same JUnit that was the original inspiration for the Python unittest testing library.

With JUnit XML describing your test run you can use the Hudson UI to view individual test failures and generate pretty graphs for how the time taken by tests changes over successive builds.

Our projects are based on Django, which in turn uses the standard unittest test runner for running tests. Python has a wealth of different choices for coaxing JUnit-XML out of a unittest test run. As we’re deploying on Ubuntu Lucid servers we needed a solution easy to integrate with the Lucid distribution of Django (approximately version 1.1.2).

After trying several alternatives we settled on the pyjunitxml package, which just happens to be the creation of Robert Collins, a fellow Canonical developer.

For a suite (unittest terminology for a collection) of tests, getting a junit-xml report from pyjunitxml is gloriously easy. Here’s the code:

    import junitxml
    with open('xmlresults.xml', 'w') as report:
        result = junitxml.JUnitXmlResult(report)

If you’re familiar with unittest code you may be surprised that there is no test runner involved. The unittest TextTestRunner class is useful for generating console output of a test run, but as this isn’t needed for a run under continuous integration we only need a test suite and the test result object from junitxml.

To integrate this into a standard django test run we had to copy the run_tests function from the django.test.simple module and add a parameter to use this code when running under Hudson.

Unsurprisingly our projects are managed on Launchpad and use Bazaar for version control. Although not enabled by default Hudson ships with a plugin for Bazaar integration. Here’s a guide for setting up Hudson for a Launchpad / Bazaar based project. It assumes you have a script “” for running your test suite with the junitxml code active:

First install hudson. For debian and Ubuntu this link gives the details:

(You will also need a version of Java and the jre installed which I don’t think that guide covers.)

Once installed Hudson will start on boot. Immediately after install you may need to start it manually:

sudo /etc/init.d/hudson restart

You can view the locally running Hudson from this URL:


You need to enable the bzr plugin in Hudson. You can do this through the Hudson plugin manager:


Find it in the ‘available’ tab of the plugin manager. You will need to restart Hudson after installing the plugin.

Next you need to give the Hudson user access to your repository. You can this step if your repository is public and can be fetched by an anonymous user.

  • Switch to the Hudson user: sudo su – hudson
  • Tell bzr your launchpad user: bzr lp-login your-username
  • Generate ssh keys without a passphrase: ssh-keygen -t rsa
  • Add the public key to your user on launchpad.

Next create a new Hudson job, with whatever name you want.

  • Select Build a free-style software project
  • Source code management: Bazaar
  • Repository URL: lp:~your-username/project-name/branch
  • Add build step -> Execute Shell:
  • Post-build action: Check Publish JUnit test result report
  • Test report XMLs: xmlresults.xml

Finally: Build Now

You can watch the build in progress via the console output.

There is no console output during the test run itself, the results go to the junit xml file used by Hudson to report the results. The console output is still useful as any build errors or unhandled exceptions will show up there.

At the moment Hudson is happily running scheduled builds on my local machine. The next step is to run it on our private cloud and decide whether or not to use a commit hook to trigger the builds. I have a personal preference for scheduled builds (as close to back-to-back as possible). Multiple runs per revision gives you the opportunity to shake out fragilities in your tests and collect better data (more data == better data) for performance tests.

Read more
Ricardo Kirkner

We are pleased to announce that Ubuntu Pay, the new payment service that allows you to buy commercial software (by means of the Ubuntu Software Centre) or subscriptions to services like Ubuntu One (in the near future), is ready to start accepting translations from the community.

Currently, we support a restricted set of languages, namely

– Chinese
– English
– German
– Polish
– Spanish

However, we would like to be able to provide a native language interface for everyone! So, if you’d like to see this service translated to your language, you can help us by

a) fixing translation mistakes on currently supported languages
b) adding translations for your language, even if it’s not supported right now

We’ll try to adopt new languages as they grow a community behind them that can help keep the translations up to date.

To start contributing your translations, just go to the translations overview page.

Read more
?ukasz Czy?ykowski

What is tox?

tox is a tool for testing python libraries (or more generally Python projects) against multiple Python implementations. Before tox, this would have been quite involved and any project wanting to check had to create its own solution. With tox it’s just a matter of creating a simple .ini file and running it as often as you need.

It uses virtualenv to properly manage and separate environments.

It’s compatible with all Python versions starting from 2.4, currently up to n 3.2. Support for Jython and PyPy is also included.

How to use it?

First thing is to create tox.ini file in your project’s directory, next to the Format is quite simple. Here’s example:

envlist = py26,py27

deps = nose
commands = nosetests

Where envlist is a list of all Python implementations you want the tests (py26, py27 are built-in, one can also create custom environments) to run in. deps is a list of all dependencies your package requires, other than packages specified in file. Finally commands is a list of all commands which tox will run in that test environment. Here it’s only running nose, but you can also have commands for building documentation or anything else that you want to ensure that it works.

With this file in package directory running tox is just a matter of:

$ tox

It will create each environment, install required packages (using pip), run all the commands and finally report success or failure. It saves all output to log files and caches all downloaded files, so no need worry about that.

Real life example

To test tox on a real project I’ve chosen configglue, our configuration library. It is always a good idea to make sure it runs properly on various Python versions. So, there will be no surprises in the future.

First step was to download the source code ($ bzr branch lp:configglue). Next in the project’s directory. I’ve created very simple tox.ini file displayed below:

envlist = py25,py26,jython

deps = nose
commands = nosetests []


As you can see, one addition is testing against Jython.

Another small caveat with configglue is that one file in the source code requires small addition to be compatible with Python 2.5 (from __future__ import with_statement). That doesn’t ensure that all tests will pass, it just prevents syntax error from being raised.

As you can see you can customize each test environment separately by using testenv:pyver section name.

Running tox with configglue reveals that it works perfectly on Python 2.5 and 2.6 but one test fails on Jython (offending line _open = __builtins__['open']).

Final Notes

Additionally tox has support for working with Hudson and should be fairly straightforward to integrate with other CI solutions.

Because this project is really young (released on June 2010) there’s not so many real life examples. One of them is from mock library; it’s tox.ini is fairly complex, as it includes building documentation using Sphinx.

You can find an even more complicated example in the kombu source code.

tox.ini in the tox source code has an example of integration with Hudson.

Of course, tox site contains a list of examples.

Read more
Szilveszter Farkas

In this blog entry I would like to describe our deployment strategies we use at the different stages of our development process. The stages are the following:

  1. local development
  2. QA
  3. staging (+ QA)
  4. production (+ QA)

For local development everyone is welcome to use his preferred way, but most of us bet on Virtualenv. Especially given the fact that we maintain a large number of different projects, it makes our lives a lot easier to have a separate environment for each of those. We also have to make sure that we align with the production environment, which is in some cases still Python 2.5-based, but we’re currently in transition to 2.6.

If a feature or bugfix is ready to be QA’d, we deploy the application to an Amazon EC2 instance. Our team mate, ?ukasz Czy?ykowski, wrote a collection of extensions to Fabric, that provides a few useful functions (e.g. using private PPAs very easily). With a few dozens of lines of simple Python code, we can deploy the whole application to a running EC2 instance. We also use EC2 to QA all the features and bugfixes targeted at a release together before deploying to staging, so that if there is an issue, we can re-deploy very quickly (during the next two stages, QA is mainly about testing regressions).

The staging and production environments are identical from the deployment process perspective. We simply create a binary Debian package from our application: Launchpad’s PPA feature makes the build process a breeze. The main reason we decided to go with Debian packages is that we can also specify system level dependencies, not only Python packages (and of course there’s some dogfooding involved since the company supports Ubuntu). This also requires that all of the team members have packaging skills, so we had several training sessions, and a two-day online sprint where we packaged lazr.restful and all of its dependencies which were not available in Ubuntu 8.04 (around 30 packages, half of them backports, half of them new packages – thanks to our hard-working team members, these are available for Ubuntu 10.04 as well).

For configuration we don’t use Django’s built-in settings mechanism, but a custom solution that will be open sourced in the near future (one more reason to keep an eye on our blog). It consists of two components: schemaconfig is responsible for parsing the config files (which are INI-style, but have some extra features, like layering, typing, and support for data structures like lists and dictionaries – basically we looked around for solutions, and stole a little bit from everywhere to put together one that fits us most), and there’s django-settings which is a glue between schemaconfig and Django’s settings (so in the end we still use django.conf.settings). One of the biggest problems we had with our previous setup that it was very prone to human error, and that caused us unexpected deployment issues between staging and production. This is solved by the layering and the non-Python style of the config files, so they are easily manageable by both us and IS (our operations team).

Watch this blog for more about schemaconfig and other exciting projects and articles!

Read more