Canonical Voices

What James Westby talks about

Posts tagged with 'tech'

At UDS last week there was another "Testing in Ubuntu" session. During the event I gave a brief presentation on monitoring and testability. The thesis was that there are a lot of parallels between monitoring and testing, so many that it's worth thinking of monitoring as a type of testing at times. Due to that great monitoring requires a testable system, as well as thinking about monitoring right at the start to build a monitorable system as well as a testable one.

You can watch a video of the talk here. (Thanks to the video team for recording it and getting it online quickly.)

I have two main questions. Firstly, what are the conventional names for the "passive" and "active" monitoring that I describe? Seecondly, do you agree with me about monitoring?

Read more

At UDS last week there was another "Testing in Ubuntu" session. During the event I gave a brief presentation on monitoring and testability. The thesis was that there are a lot of parallels between monitoring and testing, so many that it's worth thinking of monitoring as a type of testing at times. Due to that great monitoring requires a testable system, as well as thinking about monitoring right at the start to build a monitorable system as well as a testable one.

You can watch a video of the talk here. (Thanks to the video team for recording it and getting it online quickly.)

I have two main questions. Firstly, what are the conventional names for the "passive" and "active" monitoring that I describe? Seecondly, do you agree with me about monitoring?

Read more

Today is the first day of the Linaro Connect in Cambridge. Linaro has gathered to spend a week talking, coding and having fun.

The Infrastructure team is spending most of the week coding, on a few select topics, chosen to make good use of the time that we have together.

In order to help us focus on our goals for the week I've put together a hard copy version of status.linaro.org.

/images/connect-progress-start.jpg

We'll be updating it during the week as we make progress. I'll report back on how it looks at the end of the week.

Read more

Today is the first day of the Linaro Connect in Cambridge. Linaro has gathered to spend a week talking, coding and having fun.

The Infrastructure team is spending most of the week coding, on a few select topics, chosen to make good use of the time that we have together.

In order to help us focus on our goals for the week I've put together a hard copy version of status.linaro.org.

/images/connect-progress-start.jpg

We'll be updating it during the week as we make progress. I'll report back on how it looks at the end of the week.

Read more

If you are an application developer and you want to distribute your new application for a linux distribution, then you currently have several hurdles in your path. Beyond picking which one to start with, you either have a learn a packaging format well enough that you can do the work yourself, or find someone that can do it for you.

At the early stages though neither of these options is particularly compelling. You don't want to learn a packaging format, as there is lots of code to write, and that's what you want to focus on. Finding someone to do the work for you would be great, but there are far more applications than skilled packagers, and convincing someone to help you with something larval is tough: there are going to be a lot of updates, with plenty of churn, to stay on top of, and it may be too early for them to tell if the application will be any good.

This is where pkgme comes in. This is a tool that can take care of the packaging for you, so that you can focus on writing the code, and skilled packagers can focus on packages that need high-quality packaging as they will have lots of users.

This isn't a new idea, and there are plenty of tools out there to generate the packaging for e.g. a Python application. I don't think it is a particularly good use of developer time to produce tools like that for every language/project type out there.

Instead, a few of us created pkgme. This is a tool in two parts. The first part knows about packaging, and how to create the necessary files to build a working package, but it doesn't know anything about your application. This knowledge is delegated to a backend, which doesn't need to understand packaging, and just needs to be able to tell pkgme certain facts about the application.

pkgme is now at a stage where we would like to work with people to develop backends for whatever application type you would like (Python/ Ruby On Rails/GNOME/KDE/CMake/Autotools/Vala etc.) You don't have to be an expert on packaging, or indeed on the project type you want to work on. All it takes is writing a few scripts (in whatever language makes sense), which can introspect an application and report things such as the name, version, dependencies, etc.

If this sounds like something that you would like to do then please take a look at the documentation, write the scripts, and then submit your backend for inclusion in pkgme.

You can also contact the developers, see the nascent website at pkgme.net, or visit the Launchpad page. (We are also very interested in help with the website and documentation if that is where you skills or interests lie.)

Read more

If you are an application developer and you want to distribute your new application for a linux distribution, then you currently have several hurdles in your path. Beyond picking which one to start with, you either have a learn a packaging format well enough that you can do the work yourself, or find someone that can do it for you.

At the early stages though neither of these options is particularly compelling. You don't want to learn a packaging format, as there is lots of code to write, and that's what you want to focus on. Finding someone to do the work for you would be great, but there are far more applications than skilled packagers, and convincing someone to help you with something larval is tough: there are going to be a lot of updates, with plenty of churn, to stay on top of, and it may be too early for them to tell if the application will be any good.

This is where pkgme comes in. This is a tool that can take care of the packaging for you, so that you can focus on writing the code, and skilled packagers can focus on packages that need high-quality packaging as they will have lots of users.

This isn't a new idea, and there are plenty of tools out there to generate the packaging for e.g. a Python application. I don't think it is a particularly good use of developer time to produce tools like that for every language/project type out there.

Instead, a few of us created pkgme. This is a tool in two parts. The first part knows about packaging, and how to create the necessary files to build a working package, but it doesn't know anything about your application. This knowledge is delegated to a backend, which doesn't need to understand packaging, and just needs to be able to tell pkgme certain facts about the application.

pkgme is now at a stage where we would like to work with people to develop backends for whatever application type you would like (Python/ Ruby On Rails/GNOME/KDE/CMake/Autotools/Vala etc.) You don't have to be an expert on packaging, or indeed on the project type you want to work on. All it takes is writing a few scripts (in whatever language makes sense), which can introspect an application and report things such as the name, version, dependencies, etc.

If this sounds like something that you would like to do then please take a look at the documentation, write the scripts, and then submit your backend for inclusion in pkgme.

You can also contact the developers, see the nascent website at pkgme.net, or visit the Launchpad page. (We are also very interested in help with the website and documentation if that is where you skills or interests lie.)

Read more

This was the confusing part when I first ran couchapp to create a new app, I couldn't really see where the "entry point" of the app was. In the hope that it might help someone else I'm going to present a quick overview of the default setup.

index.html

The index.html page is a static attachement, and the user starts by requesting it with their browser.

It has some small amount of static HTML, part of which creates a div for the javascript to put the data in.

Either inline, or in an included file, there is a small bit of javascript that will initialise the couchapp.

By default this will use the div with the id items, and will attach an evently widget to it.

evently

The evently widget that is attached will then either have an _init event, or a _changes event, either of which will be immediately run by evently.

This event will usually make a couchdb query to get data to transform to HTML and present to the user (see part three for how this works.)

Once that data has been displayed the user any combination of evently widgets or javascript can be used to make further queries and build an app that works however you like.

Previous installments

See part one, part two, and part three.

Read more

This was the confusing part when I first ran couchapp to create a new app, I couldn't really see where the "entry point" of the app was. In the hope that it might help someone else I'm going to present a quick overview of the default setup.

index.html

The index.html page is a static attachement, and the user starts by requesting it with their browser.

It has some small amount of static HTML, part of which creates a div for the javascript to put the data in.

Either inline, or in an included file, there is a small bit of javascript that will initialise the couchapp.

By default this will use the div with the id items, and will attach an evently widget to it.

evently

The evently widget that is attached will then either have an _init event, or a _changes event, either of which will be immediately run by evently.

This event will usually make a couchdb query to get data to transform to HTML and present to the user (see part three for how this works.)

Once that data has been displayed the user any combination of evently widgets or javascript can be used to make further queries and build an app that works however you like.

Previous installments

See part one, part two, and part three.

Read more

Introducing soupmatchers

jml just announced testtools 0.9.8 and in it mentioned the soupmatchers project that I started. Given that I haven't talked about it here before, I wanted to do a post to introduce it, and explain some of the rationale behind it.

soupmatchers is a library for unit testing HTML, allowing you to assert that certain things are present or not within an HTML string. Asserting this based on substring matching is going to be too fragile to be usable, and so soupmatchers works on a parsed representation of the HTML. It uses the wonderful BeautifulSoup library for parsing the HTML, and allows you to assert the presence or not of tags based on the attributes that you care about.

self.assertThat(some_html,
                HTMLContains(Tag('testtools link', 'a',
                attrs={'href': 'https://launchpad.net/testtools'})))

You can see more examples in the README.

Basing this on the testtools matchers frameworks allows you to do this in a semi-declarative way. I think there is a lot of potential here to improve your unit tests. For instance you can start to build a suite of matchers tailored to talking about the HTML that your application outputs. You can have matchers that match areas of the page, and then talk about other elements relative to them ("This link is placed within the sidebar"). One thing that particularly interests me is to create a class hierarchy that allows you test particular things across your application. For instance, you could have an ExternalLink class that asserts that a particular class is set on all of your external links. Assuming that you use this at the appropriate places in your tests then you will know that the style that is applied to class will be on all external links. Should you wish to change the way that external links are represented in the HTML you can change the one class and your tests should tell you all the places that the code has to be updated.

Please go ahead and try the library and let me know how it could be improved.

Read more

Introducing soupmatchers

jml just announced testtools 0.9.8 and in it mentioned the soupmatchers project that I started. Given that I haven't talked about it here before, I wanted to do a post to introduce it, and explain some of the rationale behind it.

soupmatchers is a library for unit testing HTML, allowing you to assert that certain things are present or not within an HTML string. Asserting this based on substring matching is going to be too fragile to be usable, and so soupmatchers works on a parsed representation of the HTML. It uses the wonderful BeautifulSoup library for parsing the HTML, and allows you to assert the presence or not of tags based on the attributes that you care about.

self.assertThat(some_html,
                HTMLContains(Tag('testtools link', 'a',
                attrs={'href': 'https://launchpad.net/testtools'})))

You can see more examples in the README.

Basing this on the testtools matchers frameworks allows you to do this in a semi-declarative way. I think there is a lot of potential here to improve your unit tests. For instance you can start to build a suite of matchers tailored to talking about the HTML that your application outputs. You can have matchers that match areas of the page, and then talk about other elements relative to them ("This link is placed within the sidebar"). One thing that particularly interests me is to create a class hierarchy that allows you test particular things across your application. For instance, you could have an ExternalLink class that asserts that a particular class is set on all of your external links. Assuming that you use this at the appropriate places in your tests then you will know that the style that is applied to class will be on all external links. Should you wish to change the way that external links are represented in the HTML you can change the one class and your tests should tell you all the places that the code has to be updated.

Please go ahead and try the library and let me know how it could be improved.

Read more

[ Apologies to those that saw this half-finished when I published rather than saving a draft ]

This is the part that it took me a long time to understand: how the different parts of the default couchapp collaborate to present data to the user.

In this post I'm just going to deal with client-side couchapps using the default technologies. As explained in the previous post you can use any combination of HTML and javascript in a couchapp, and you can also do some of the work server-side in couchdb. However, I'm going to explain what the couchapp tool gives you when you create a new project, as that is where you are likely to be starting, and once you understand that you can choose where to deviate from that model.

jQuery events

Our first detour is in to a little bit of background, the excellent javascript libarary that is heavily used in couchapps.

jQuery allows for events on elements in the DOM. There are standard events, such as "click" and "submit", but you are free to define your own.

These events are given a name, and you can then "trigger" them, and bind handlers to act when they are triggered.

By building up events from low level ones such as "click", to more-complex and app-specific ones such as "item purchased", you can break down your code in to smaller chunks, and have different parts of the page react to the same event, such as having the "buy" link disappear from the item that the user just bought, as well as having the total of the shopping cart update.

Events can also have data, or arguments, that travels with them. For instance the "item purchased" event could have the item that was purchased as the data, so that handlers can make use of it when they run.

evently

Now that we know something about jQuery events, we can look at something built on top of them, the "evently" library. This is a layer on top of jQuery that allows you build up your app from pieces that have a specific function, and communicate through events.

An evently "widget" can be bound to an element (or several elements if you want). The widget is a bunch of event handlers which can do anything you like, but have some conveniences built in for fetching data and updating the page based on the result.

When an event is triggered the handler you defined is run. If it is a simple javascript function then that function is run, and can do anything you like.

{click: function() {
        alert("You clicked!");
    }
}

Often though you want to update the element based on the action. evently has built in support for the "mustache" templating language, and if you specify a template in that syntax it will replace the current HTML of the element that it is attached to with the result of rendering that template.

{click:
    {
        mustache: "<div>You clicked!</div>"
    }
}

Which will put "You clicked!" in to the page instead of in an alert. What if you don't want to replace the current content, and just want to append another line? For that use the "render" option.

{click:
    {
        mustache: "<div>You clicked!</div>",
        render: "append"
    }
}

Which would put another "You clicked!" on the page every time you click. As well as "append" there is also "prepend", or really any jQuery method that you want to call.

Simply rendering a static template isn't going to be very useful though, usually you want something dynamic. For that use the "data" option, which can just be an object if you want, but that's still not going to be very dynamic either, so it can be a function that returns an object.

{click:
    {
        data: function(e) {
            return {name, "Bob"};
        },
        mustache: "<div>Hi {{name}}!</div>"
    }
}

The data function gets passed the event object from jQuery (so you can e.g. get the target of the event), and any data for the event too (so it could see what item you just bought).

That's all well and good, but it doesn't help us get data from couchdb in to the page. For that we need the opportunity to make a request to couchdb. We could just fall back to using one function to handle the event, but then we lose the integration with mustache. Therefore there is an "async" key that allows us to make an AJAX request and then use mustache on the result.

{click:
    {
        async: function(callback) {
            /* some code that does an async request, and then calls callback with the result */
        },
        data: function(resp) {
            /* Some code that processes the data from the async function to ready it for the template */
        },
        mustache: "A tempate that will be rendered with the result of the data function"
    }
}

Now, writing an async method to query a couchdb view is so common in couchapps that eventy has special support for it. The query key can either be a json structure that specifies a view and the arguments to it, or a function that returns such a structure based on things such as the query string in the URL.

There are two further functions that you will find helpful from time to time. The first is before that allows you to run some code before the rest of the process starts, and may do something such as trigger another event. The other is its partner after, which can do much the same things as before, but can also do things such as modify the HTML that is output.

Lastly there is another thing that can be done with the HTML that is output, specified with they selectors key. This allows you to perform an action on particular parts of the html. The keys of this structure are jQuery selectors that specify which elements the function will be applied to. For instance you can do something with all the divs in the output, or all the spans with a certain class, or the form with a particular id.

What you can do to those elements is basically unlimited, as you can run arbitrary javascript. However, there is built in support for specifying an evently widget, which will automatically be bound to each element that matches the selector. This nesting is one of the most powerful and useful features of evently, and one you should generally be using often. I will probably talk more about what nested widgets are useful for later.

Special evently events

evently has two special events. The first of these is _init. This event is triggered when the widget is created. This means you can dynamically pre-populate the element, or at least keep the inital state of the element with the rest of your code, rather than putting some in the HTML file and the rest in evently code.

The other special event is tied to couchdb, and is the _changes event, and is triggered whenever the database that the couchapp is in changes. This means that you can have elements on the page that dynamically update whenever the database changes, whether that is through user action, another user doing something, external scripts, or couchdb replication. This makes it very easy to write "live" pages that show updates without refreshes, and is very useful for some applications.

Currently _changes doesn't receive the modified documents, so it is normally just used to make another request to get the updated information, whether that be through async or view. If you wish to get the modified documents in order to update the page directly and reduce requests then you can write some custom code to do this.

Conclusion

As you have seen, evently is just a thin layer on top of jQuery concepts such as events and asynchronous events, with some conveniences for templating and interacting with couchdb.

This combination is well suited to the needs of at least simple and moderately complex couchapps, while still being very powerful, and allowing you to fall back to custom javascript at any point.

You can find more about evently at the couchapp site.

See part one and part two.

Read more

[ Apologies to those that saw this half-finished when I published rather than saving a draft ]

This is the part that it took me a long time to understand: how the different parts of the default couchapp collaborate to present data to the user.

In this post I'm just going to deal with client-side couchapps using the default technologies. As explained in the previous post you can use any combination of HTML and javascript in a couchapp, and you can also do some of the work server-side in couchdb. However, I'm going to explain what the couchapp tool gives you when you create a new project, as that is where you are likely to be starting, and once you understand that you can choose where to deviate from that model.

jQuery events

Our first detour is in to a little bit of background, the excellent javascript libarary that is heavily used in couchapps.

jQuery allows for events on elements in the DOM. There are standard events, such as "click" and "submit", but you are free to define your own.

These events are given a name, and you can then "trigger" them, and bind handlers to act when they are triggered.

By building up events from low level ones such as "click", to more-complex and app-specific ones such as "item purchased", you can break down your code in to smaller chunks, and have different parts of the page react to the same event, such as having the "buy" link disappear from the item that the user just bought, as well as having the total of the shopping cart update.

Events can also have data, or arguments, that travels with them. For instance the "item purchased" event could have the item that was purchased as the data, so that handlers can make use of it when they run.

evently

Now that we know something about jQuery events, we can look at something built on top of them, the "evently" library. This is a layer on top of jQuery that allows you build up your app from pieces that have a specific function, and communicate through events.

An evently "widget" can be bound to an element (or several elements if you want). The widget is a bunch of event handlers which can do anything you like, but have some conveniences built in for fetching data and updating the page based on the result.

When an event is triggered the handler you defined is run. If it is a simple javascript function then that function is run, and can do anything you like.

{click: function() {
        alert("You clicked!");
    }
}

Often though you want to update the element based on the action. evently has built in support for the "mustache" templating language, and if you specify a template in that syntax it will replace the current HTML of the element that it is attached to with the result of rendering that template.

{click:
    {
        mustache: "<div>You clicked!</div>"
    }
}

Which will put "You clicked!" in to the page instead of in an alert. What if you don't want to replace the current content, and just want to append another line? For that use the "render" option.

{click:
    {
        mustache: "<div>You clicked!</div>",
        render: "append"
    }
}

Which would put another "You clicked!" on the page every time you click. As well as "append" there is also "prepend", or really any jQuery method that you want to call.

Simply rendering a static template isn't going to be very useful though, usually you want something dynamic. For that use the "data" option, which can just be an object if you want, but that's still not going to be very dynamic either, so it can be a function that returns an object.

{click:
    {
        data: function(e) {
            return {name, "Bob"};
        },
        mustache: "<div>Hi {{name}}!</div>"
    }
}

The data function gets passed the event object from jQuery (so you can e.g. get the target of the event), and any data for the event too (so it could see what item you just bought).

That's all well and good, but it doesn't help us get data from couchdb in to the page. For that we need the opportunity to make a request to couchdb. We could just fall back to using one function to handle the event, but then we lose the integration with mustache. Therefore there is an "async" key that allows us to make an AJAX request and then use mustache on the result.

{click:
    {
        async: function(callback) {
            /* some code that does an async request, and then calls callback with the result */
        },
        data: function(resp) {
            /* Some code that processes the data from the async function to ready it for the template */
        },
        mustache: "A tempate that will be rendered with the result of the data function"
    }
}

Now, writing an async method to query a couchdb view is so common in couchapps that eventy has special support for it. The query key can either be a json structure that specifies a view and the arguments to it, or a function that returns such a structure based on things such as the query string in the URL.

There are two further functions that you will find helpful from time to time. The first is before that allows you to run some code before the rest of the process starts, and may do something such as trigger another event. The other is its partner after, which can do much the same things as before, but can also do things such as modify the HTML that is output.

Lastly there is another thing that can be done with the HTML that is output, specified with they selectors key. This allows you to perform an action on particular parts of the html. The keys of this structure are jQuery selectors that specify which elements the function will be applied to. For instance you can do something with all the divs in the output, or all the spans with a certain class, or the form with a particular id.

What you can do to those elements is basically unlimited, as you can run arbitrary javascript. However, there is built in support for specifying an evently widget, which will automatically be bound to each element that matches the selector. This nesting is one of the most powerful and useful features of evently, and one you should generally be using often. I will probably talk more about what nested widgets are useful for later.

Special evently events

evently has two special events. The first of these is _init. This event is triggered when the widget is created. This means you can dynamically pre-populate the element, or at least keep the inital state of the element with the rest of your code, rather than putting some in the HTML file and the rest in evently code.

The other special event is tied to couchdb, and is the _changes event, and is triggered whenever the database that the couchapp is in changes. This means that you can have elements on the page that dynamically update whenever the database changes, whether that is through user action, another user doing something, external scripts, or couchdb replication. This makes it very easy to write "live" pages that show updates without refreshes, and is very useful for some applications.

Currently _changes doesn't receive the modified documents, so it is normally just used to make another request to get the updated information, whether that be through async or view. If you wish to get the modified documents in order to update the page directly and reduce requests then you can write some custom code to do this.

Conclusion

As you have seen, evently is just a thin layer on top of jQuery concepts such as events and asynchronous events, with some conveniences for templating and interacting with couchdb.

This combination is well suited to the needs of at least simple and moderately complex couchapps, while still being very powerful, and allowing you to fall back to custom javascript at any point.

You can find more about evently at the couchapp site.

See part one and part two.

Read more

This is the part that it took me a long time to understand: how the different parts of the default couchapp collaborate to present data to the user.

In this post I'm just going to deal with client-side couchapps using the default technologies. As explained in the previous post you can use any combination of HTML and javascript in a couchapp, and you can also do some of the work server-side in couchdb. However, I'm going to explain what the couchapp tool gives you when you create a new project, as that is where you are likely to be starting, and once you understand that you can choose where to deviate from that model.

jQuery events

Our first detour is in to a little bit of background, the excellent javascript libarary that is heavily used in couchapps.

jQuery allows for events on elements in the DOM. There are standard events, such as "click" and "submit", but you are free to define your own.

These events are given a name, and you can then "trigger" them, and bind handlers to act when they are triggered.

By building up events from low level ones such as "click", to more-complex and app-specific ones such as "item purchased", you can break down your code in to smaller chunks, and have different parts of the page react to the same event, such as having the "buy" link disappear from the item that the user just bought, as well as having the total of the shopping cart update.

Events can also have data, or arguments, that travels with them. For instance the "item purchased" event could have the item that was purchased as the data, so that handlers can make use of it when they run.

evently

Now that we know something about jQuery events, we can look at something built on top of them, the "evently" library. This is a layer on top of jQuery that allows you build up your app from pieces that have a specific function, and communicate through events.

An evently "widget" can be bound to an element (or several elements if you want). The widget is a bunch of event handlers which can do anything you like, but have some conveniences built in for fetching data and updating the page based on the result.

When an event is triggered the handler you defined is run. If it is a simple javascript function then that function is run, and can do anything you like.

::
{click: function() {
alert("You clicked!");

System Message: WARNING/2 (<string>, line 57)

Block quote ends without a blank line; unexpected unindent.

}

System Message: WARNING/2 (<string>, line 58)

Definition list ends without a blank line; unexpected unindent.

}

Often though you want to update the element based on the action. evently has built in support for the "mustache" templating language, and if you specify a template in that syntax it will replace the current HTML of the element that it is attached to with the result of rendering that template.

::
{click:
{
mustache: "<div>You clicked!</div>"

System Message: WARNING/2 (<string>, line 70)

Definition list ends without a blank line; unexpected unindent.

}

System Message: WARNING/2 (<string>, line 71)

Definition list ends without a blank line; unexpected unindent.

}

Which will put "You clicked!" in to the page instead of in an alert. What if you don't want to replace the current content, and just want to append another line? For that use the "render" option.

::
{click:
{
mustache: "<div>You clicked!</div>", render: "append"

System Message: WARNING/2 (<string>, line 82)

Definition list ends without a blank line; unexpected unindent.

}

System Message: WARNING/2 (<string>, line 83)

Definition list ends without a blank line; unexpected unindent.

}

Which would put another "You clicked!" on the page every time you click. As well as "append" there is also "prepend", or really any jQuery method that you want to call.

Simply rendering a static template isn't going to be very useful though, usually you want something dynamic. For that use the "data" option, which can just be an object if you want, but that's still not going to be very dynamic either, so it can be a function that returns an object.

::
{click:
{
data: function(e) {
return {name, "Bob"};

System Message: WARNING/2 (<string>, line 99)

Definition list ends without a blank line; unexpected unindent.

}, mustache: "<div>Hi {{name}}!</div>"

System Message: WARNING/2 (<string>, line 101)

Definition list ends without a blank line; unexpected unindent.

}

System Message: WARNING/2 (<string>, line 102)

Definition list ends without a blank line; unexpected unindent.

}

The data function gets passed the event object from jQuery (so you can e.g. get the target of the event), and any data for the event too (so it could see what item you just bought).

That's all well and good, but it doesn't help us get data from couchdb in to the page. For that we need the opportunity to make a request to couchdb. We could just fall back to using one function to handle the event, but then we lose the integration with mustache. Therefore there is an "async" key that allows us to make an AJAX request and then use mustache on the result.

See part one and part two.

Read more

Today I would like to talk about the couchapp tool. This is something that you can use when working on couchapps, and provides a way to quickly iterate your design.

However, rather confusingly, the couchapp tool isn't actually required for couchapps. If you get a design document with HTML attachments in to your database then you have a couchapp.

Why would you want to use such a tool then? Firstly because it will generate a skeleton couchapp for you, so that you don't have to remember how to organise it, and if it is your first couchapp it's good to start from something working.

More importantly though, the couchapp tool is useful as you develop as it allows you to edit the parts of your app in their native format. This means that if you are writing a HTML snippet you can just put some HTML in a file, rather than having to write it as part of a deeply nested JSON structure and deal with the errors that you would make if you did that. Also it means that you can use things like syntax highlighting in your preferred text editor without any special magic.

How it works

At it's core couchapp is a rather simple tool. It walks a filesystem tree and assembles the things that it finds there in to a JSON document.

For instance, if it finds a directory at the root of the tree called _attachments it puts the content of each file there in to a list which it puts in to the dict with an "_attachments" key, Which is one of the ways that couchdb accepts document attachments. Therefore if you have an _attachments/index.html file in your tree it will be attached to your design document when the JSON structure is sent to couchdb.

This continues across the tree, so the contents of the "views" directory will become the "views" key of the documents, which is how you do map/reduce queries on the database.

couchapp has various conventions for dealing with files. For instance if it finds a ".js" file it treats it as a javascript snippet which will be encoded in to a string in the resulting document. ".html" files outside of the "_attachments" directory will also be encoded as strings. If it finds a ".json" file then it treats it as literal JSON that will be embebedded.

This way it builds up the JSON structure that a couchapp expects, and will send it to the couchdb of your choice when it is done.

In addition to this functionality the tool can also generate you a skeleton app, and also add new pieces to your app, such as new views.

Getting It

couchapp is a python tool, so you can install it using pip or similar. However, Ubuntu users can install it from a PPA (yay for daily builds with recipes!).

Using It

To use it run

couchapp generate myapp

which will create you a new skeleton in myapp.

cd myapp
ls

You will see for instance the _attachments and views directories, and an _attachments/index.html.

To get your app in to couchdb you can run

couchapp push http://localhost:5984/mydb

and it will tell you the URL to visit to see your new app.

If you want to use desktopcouch you can run

couchapp push desktopcouch://

though I think it has a bug that it prints the wrong URLs when pushing to desktopcouch.

Once you have looked at the HTML generated by your app you should look at the design document that couchapp created. Go to

http://localhost:5984/_utils

or

xdg-open ~/.local/share/desktop-couch/couchdb.html

if you are using desktopcouch.

Click to the mydb database and you will see a document called _design/myapp. Click on this and you will see the content of the design document; you are looking at a couchapp in its raw form.

If you compare what is in that design document with what is in the myapp directory that the tool created you should start to see how it generates it from the filesystem.

Now try making a change on the filesystem, for instance edit _attachments/index.html and put your name somewhere in the body. Then push again, running

couchapp push http://localhost:5984/mydb

and refresh the page in your browser and you should see the change. (Just click on index.html in the design document to get back to viewing your app from there).

I will go in to more detail about the content of the couchapp that was generated for you in another post.

See the previous installment.

Read more

Today I would like to talk about the couchapp tool. This is something that you can use when working on couchapps, and provides a way to quickly iterate your design.

However, rather confusingly, the couchapp tool isn't actually required for couchapps. If you get a design document with HTML attachments in to your database then you have a couchapp.

Why would you want to use such a tool then? Firstly because it will generate a skeleton couchapp for you, so that you don't have to remember how to organise it, and if it is your first couchapp it's good to start from something working.

More importantly though, the couchapp tool is useful as you develop as it allows you to edit the parts of your app in their native format. This means that if you are writing a HTML snippet you can just put some HTML in a file, rather than having to write it as part of a deeply nested JSON structure and deal with the errors that you would make if you did that. Also it means that you can use things like syntax highlighting in your preferred text editor without any special magic.

How it works

At it's core couchapp is a rather simple tool. It walks a filesystem tree and assembles the things that it finds there in to a JSON document.

For instance, if it finds a directory at the root of the tree called _attachments it puts the content of each file there in to a list which it puts in to the dict with an "_attachments" key, Which is one of the ways that couchdb accepts document attachments. Therefore if you have an _attachments/index.html file in your tree it will be attached to your design document when the JSON structure is sent to couchdb.

This continues across the tree, so the contents of the "views" directory will become the "views" key of the documents, which is how you do map/reduce queries on the database.

couchapp has various conventions for dealing with files. For instance if it finds a ".js" file it treats it as a javascript snippet which will be encoded in to a string in the resulting document. ".html" files outside of the "_attachments" directory will also be encoded as strings. If it finds a ".json" file then it treats it as literal JSON that will be embebedded.

This way it builds up the JSON structure that a couchapp expects, and will send it to the couchdb of your choice when it is done.

In addition to this functionality the tool can also generate you a skeleton app, and also add new pieces to your app, such as new views.

Getting It

couchapp is a python tool, so you can install it using pip or similar. However, Ubuntu users can install it from a PPA (yay for daily builds with recipes!).

Using It

To use it run

couchapp generate myapp

which will create you a new skeleton in myapp.

cd myapp
ls

You will see for instance the _attachments and views directories, and an _attachments/index.html.

To get your app in to couchdb you can run

couchapp push http://localhost:5984/mydb

and it will tell you the URL to visit to see your new app.

If you want to use desktopcouch you can run

couchapp push desktopcouch://

though I think it has a bug that it prints the wrong URLs when pushing to desktopcouch.

Once you have looked at the HTML generated by your app you should look at the design document that couchapp created. Go to

http://localhost:5984/_utils

or

xdg-open ~/.local/share/desktop-couch/couchdb.html

if you are using desktopcouch.

Click to the mydb database and you will see a document called _design/myapp. Click on this and you will see the content of the design document; you are looking at a couchapp in its raw form.

If you compare what is in that design document with what is in the myapp directory that the tool created you should start to see how it generates it from the filesystem.

Now try making a change on the filesystem, for instance edit _attachments/index.html and put your name somewhere in the body. Then push again, running

couchapp push http://localhost:5984/mydb

and refresh the page in your browser and you should see the change. (Just click on index.html in the design document to get back to viewing your app from there).

I will go in to more detail about the content of the couchapp that was generated for you in another post.

See the previous installment.

Read more

Couchapps are a particular way of using couchdb that allow you to serve web applications directly from the database. These applications generate HTML and javascript to present data from couchdb to the user, and then update the database and the UI based on their actions.

Of course there are plenty of frameworks out there that do this sort of thing, and more and more of them are adding couchdb support. What makes couchapps particularly interesting are two things. Firstly, the ease with which they can be developed and deployed. As they are served directly from couchdb they require little infrastructure, and the couchapp tool allows for a rapid iteration. In addition, the conveniences that are provided mean that simple things can be done very quickly with little code.

The other thing that makes couchapps attractive is that they live inside the database. This means that the code lives alongside the data, and will travel with it as it is replicated. This means that you can easily have an app that you have fast, local access to on your desktop, while at the same time replicating to a server so that you can access the same data from your phone while you are out. Again, this doesn't require couchapps, and they won't be suitable for all needs, but they are certainly an interesting idea.

You can read more about couchapps at http://couchapp.org.

Intrigued by couchapps I set out to play with them over a weekend. Unfortunately the documentation is rather lacking currently, so I wouldn't recommend experimenting yourself if you are not happy digging around for answers, and sometimes not finding them outside the code. In order to go a little way to rectifying this, I intend to write a few posts about the things I wish I had known when I started out. I found everything to be a little strange at first, and it wasn't even clear where the entry point of a couchapp was for instance. Hopefully these posts will be found using google by others who are struggling in a similar way.

Architecture

Firstly something about the pieces that make up a couchapp (or at least those that the tool and documentation recommend,) and the way that they all fit together.

At the core is the couchdb database itself. It is a collection of "documents", each of which can have attachments. Some of these documents are known as "design documents," and they start with a prefix of "_design." Design documents can have "view" functions, and various other special fields that can be used to query or manipulate other documents.

A couchapp is a design document with an attachment, usually called index.html. These attachments are served directly by couchdb and can be accessed at a known URL. You can put anything you like in that html file, and you could just have a static page if you wanted. Usually however though it is is an HTML page that uses javascript in order to display the results of queries on the database. The user will then access the attachment on the design document, and will interact with the resulting page.

In theory you can do anything you like in that page, but it is usual to make use of standard tools in order to query the database and provide information and opportunity for interaction to the user.

The first standard tool is jQuery, with a couple of plugins for working with couchdb and couchapps specifically. These allow for querying views in the database and acting on the results, retrieving and updating documents, and plenty more.

In addition the couchapp tool sets you up with another jQuery plugin called "evently", which is a way to structure interactions with jQuery, and change the page based on various events. I will go in to more detail about how evently works in a later post.

In addition to all the client-side tools for interacting with the database, it is also possible to make use of couchdb features such as shows, lists, update handlers validation functions in order to move some of the processing server-side. This is useful for various reasons, including being more accessible, allowing search engines to index the content, and not having to trust the client not to take malicious actions.

The two approaches can be combined, and you can prototype with the client-side tools, and then move some of the work to the server-side facilities later.

Stay tuned for more on how a simple couchapp generates content based on what is in the db.

Read more

Couchapps are a particular way of using couchdb that allow you to serve web applications directly from the database. These applications generate HTML and javascript to present data from couchdb to the user, and then update the database and the UI based on their actions.

Of course there are plenty of frameworks out there that do this sort of thing, and more and more of them are adding couchdb support. What makes couchapps particularly interesting are two things. Firstly, the ease with which they can be developed and deployed. As they are served directly from couchdb they require little infrastructure, and the couchapp tool allows for a rapid iteration. In addition, the conveniences that are provided mean that simple things can be done very quickly with little code.

The other thing that makes couchapps attractive is that they live inside the database. This means that the code lives alongside the data, and will travel with it as it is replicated. This means that you can easily have an app that you have fast, local access to on your desktop, while at the same time replicating to a server so that you can access the same data from your phone while you are out. Again, this doesn't require couchapps, and they won't be suitable for all needs, but they are certainly an interesting idea.

You can read more about couchapps at http://couchapp.org.

Intrigued by couchapps I set out to play with them over a weekend. Unfortunately the documentation is rather lacking currently, so I wouldn't recommend experimenting yourself if you are not happy digging around for answers, and sometimes not finding them outside the code. In order to go a little way to rectifying this, I intend to write a few posts about the things I wish I had known when I started out. I found everything to be a little strange at first, and it wasn't even clear where the entry point of a couchapp was for instance. Hopefully these posts will be found using google by others who are struggling in a similar way.

Architecture

Firstly something about the pieces that make up a couchapp (or at least those that the tool and documentation recommend,) and the way that they all fit together.

At the core is the couchdb database itself. It is a collection of "documents", each of which can have attachments. Some of these documents are known as "design documents," and they start with a prefix of "_design." Design documents can have "view" functions, and various other special fields that can be used to query or manipulate other documents.

A couchapp is a design document with an attachment, usually called index.html. These attachments are served directly by couchdb and can be accessed at a known URL. You can put anything you like in that html file, and you could just have a static page if you wanted. Usually however though it is is an HTML page that uses javascript in order to display the results of queries on the database. The user will then access the attachment on the design document, and will interact with the resulting page.

In theory you can do anything you like in that page, but it is usual to make use of standard tools in order to query the database and provide information and opportunity for interaction to the user.

The first standard tool is jQuery, with a couple of plugins for working with couchdb and couchapps specifically. These allow for querying views in the database and acting on the results, retrieving and updating documents, and plenty more.

In addition the couchapp tool sets you up with another jQuery plugin called "evently", which is a way to structure interactions with jQuery, and change the page based on various events. I will go in to more detail about how evently works in a later post.

In addition to all the client-side tools for interacting with the database, it is also possible to make use of couchdb features such as shows, lists, update handlers validation functions in order to move some of the processing server-side. This is useful for various reasons, including being more accessible, allowing search engines to index the content, and not having to trust the client not to take malicious actions.

The two approaches can be combined, and you can prototype with the client-side tools, and then move some of the work to the server-side facilities later.

Stay tuned for more on how a simple couchapp generates content based on what is in the db.

Read more

The examples for Django testing point you towards hardcoding a username and password for a user to impersonate in tests, and the API of the test client encourages this too.

However, Django has a nice pluggable authentication system that means you can easily use something such as OpenID instead of passwords.

Putting the passwords in your tests ties you to having the password support enabled, and while you could do this for just the tests, it's completely out of the scope of most tests (I'm not talking about any tests for the actual login process here.)

When I saw this while reviewing code recently I worked with Zygmunt to write a Client subclass that didn't have this restriction. With this subclass you can just choose a User object, and have that client login as that user, without them having to have a password at all. Doing this decoples your tests from the implementation of the authentication system, and makes them target the code you want to test more precisely.

Here's the code:

from django.conf import settings
from django.contrib.auth import login
from django.http import HttpRequest
from django.test.client import Client


class TestClient(Client):

    def login_user(self, user):
        """
        Login as specified user, does not depend on auth backend (hopefully)

        This is based on Client.login() with a small hack that does not
        require the call to authenticate()
        """
        if not 'django.contrib.sessions' in settings.INSTALLED_APPS:
            raise AssertionError("Unable to login without django.contrib.sessions in INSTALLED_APPS")
        user.backend = "%s.%s" % ("django.contrib.auth.backends",
                                  "ModelBackend")
        engine = import_module(settings.SESSION_ENGINE)

        # Create a fake request to store login details.
        request = HttpRequest()
        if self.session:
            request.session = self.session
        else:
            request.session = engine.SessionStore()
        login(request, user)

        # Set the cookie to represent the session.
        session_cookie = settings.SESSION_COOKIE_NAME
        self.cookies[session_cookie] = request.session.session_key
        cookie_data = {
            'max-age': None,
            'path': '/',
            'domain': settings.SESSION_COOKIE_DOMAIN,
            'secure': settings.SESSION_COOKIE_SECURE or None,
            'expires': None,
        }
        self.cookies[session_cookie].update(cookie_data)

        # Save the session values.
        request.session.save()

Then you can use it in your tests like this:

from django.contrib.auth.models import User


client = TestClient()
user = User(username="eve")
user.save()
client.login_user(user)

Then any requests you make with that client will be authenticated as the user that was created.

Ticket submitted with Django to have this available for everyone in future.

Read more

The examples for Django testing point you towards hardcoding a username and password for a user to impersonate in tests, and the API of the test client encourages this too.

However, Django has a nice pluggable authentication system that means you can easily use something such as OpenID instead of passwords.

Putting the passwords in your tests ties you to having the password support enabled, and while you could do this for just the tests, it's completely out of the scope of most tests (I'm not talking about any tests for the actual login process here.)

When I saw this while reviewing code recently I worked with Zygmunt to write a Client subclass that didn't have this restriction. With this subclass you can just choose a User object, and have that client login as that user, without them having to have a password at all. Doing this decoples your tests from the implementation of the authentication system, and makes them target the code you want to test more precisely.

Here's the code:

from django.conf import settings
from django.contrib.auth import login
from django.http import HttpRequest
from django.test.client import Client


class TestClient(Client):

    def login_user(self, user):
        """
        Login as specified user, does not depend on auth backend (hopefully)

        This is based on Client.login() with a small hack that does not
        require the call to authenticate()
        """
        if not 'django.contrib.sessions' in settings.INSTALLED_APPS:
            raise AssertionError("Unable to login without django.contrib.sessions in INSTALLED_APPS")
        user.backend = "%s.%s" % ("django.contrib.auth.backends",
                                  "ModelBackend")
        engine = import_module(settings.SESSION_ENGINE)

        # Create a fake request to store login details.
        request = HttpRequest()
        if self.session:
            request.session = self.session
        else:
            request.session = engine.SessionStore()
        login(request, user)

        # Set the cookie to represent the session.
        session_cookie = settings.SESSION_COOKIE_NAME
        self.cookies[session_cookie] = request.session.session_key
        cookie_data = {
            'max-age': None,
            'path': '/',
            'domain': settings.SESSION_COOKIE_DOMAIN,
            'secure': settings.SESSION_COOKIE_SECURE or None,
            'expires': None,
        }
        self.cookies[session_cookie].update(cookie_data)

        # Save the session values.
        request.session.save()

Then you can use it in your tests like this:

from django.contrib.auth.models import User


client = TestClient()
user = User(username="eve")
user.save()
client.login_user(user)

Then any requests you make with that client will be authenticated as the user that was created.

Ticket submitted with Django to have this available for everyone in future.

Read more

Normally when you write some code using launchpadlib you end up with Launchpad showing your users something like this:

/images/lplib-before.png

This isn't great, how is the user supposed to know which option to click? What do you do if they don't choose the option you want?

Instead it's possible to limit the choices that the user has to make to only those that your application can use, plus the option to deny all access, by changing the way you create your Launchpad object.

from launchpadlib.launchpad import Launchpad

lp = Launchpad.get_token_and_login("testing", allow_access_levels=["WRITE_PUBLIC"])

This will present your users with something like this:

/images/lplib-after.png

which is easier to understand. There could be further improvements, but they would happen on the Launchpad side.

This approach works for both Launchpad.get_token_and_login and Launchpad.login_with.

The values that you can pass here aren't documented, and should probably be constants in launchpadlib, rather than hardcoded in every application, but for now you can use:

  • READ_PUBLIC
  • READ_PRIVATE
  • WRITE_PUBLIC
  • WRITE_PRIVATE

Read more