Canonical Voices

Posts tagged with 'development'

Jussi Pakkanen

There have been several posts in this blog about compile speed. However most have been about theory. This time it’s all about measurements.

I took the source code of Scribus, which is a largeish C++ application and looked at how much faster I could make it compile. There are three different configurations to test. The first one is building with default settings out of the box. The second one is about changes that can be done without changing any source code, meaning building with the Ninja backend instead of Make and using Gold instead of ld. The third configuration adds precompiled headers to the second configuration.

The measurements turned out to have lots of variance, which I could not really nail down. However it seemed to affect all configurations in the same way at the same time so the results should be comparable. All tests were run on a 4 core laptop with 4 GB of ram. Make was run with ‘-j 6′ as that is the default value of Ninja.

Default:    11-12 minutes
Ninja+Gold: ~9 minutes
PCH:        7 minutes

We can see that a bit of work the compile time can be cut almost in half. Enabling PCH does not require changing any existing source files (though you’ll get slightly better performance if you do). All in all it takes less than 100 lines of CMake code to enable precompiled headers, and half of that is duplicating some functionality that CMake should be exposing already. For further info, see this bug.

Is it upstream? Can I try it? Will it work on my project?

The patch is not upstreamed, because it is not yet clean enough. However you can check out most of it in this merge request to Unity. In Unity’s case the speedup was roughly 40%, though only one library build time was measured. The total build time impact is probably less.

Note that if you can’t just grab the code and expect magic speedups. You have to select which headers to precompile and so on.

Finally, for a well tuned code base, precompiled headers should only give around 10-20% speed boost. If you get more, it probably means that you have an #include maze in your header files. You should probably get that fixed sooner rather than later.

Read more
pitti

I found it surprisingly hard to determine in tearDown() whether or not the test that currently ran succeeded or not. I am writing some tests for gnome-settings-daemon and want to show the log output of the daemon if a test failed.

I now cobbled together the following hack, but I wonder if there’s a more elegant way? The interwebs don’t seem to have a good solution for this either.

    def tearDown(self):
        [...]
        # collect log, run() shows it on failures
        with open(self.daemon_log.name) as f:
            self.log_output = f.read()

    def run(self, result=None):
        '''Show log output on failed tests'''

        if result:
            orig_err_fail = result.errors + result.failures
        super().run(result)
        if result and result.errors + result.failures > orig_err_fail:
            print('\n----- daemon log -----\n%s\n------\n' % self.log_output)

Read more
pitti

I was working on writing tests for gnome-settings-daemon a week or so ago, and finally got blocked on being unable to set up upower/ConsoleKit/etc. the way I need them. Also, doing so needs root privileges, I don’t want my test suite to actually suspend my machine, and using the real service is generally not suitable for test suites that are supposed to run during “make check”, in jhbuild, and the like — these do not have the polkit privileges to do all that, and may not even have a system D-Bus running in the first place.

So I wrote a little test_upower.py helper, then realized that I need another one for systemd/ConsoleKit (for the “system idle” property), also looked at the mock polkit in udisks and finally sat down for two days to generalize this and do this properly.

The result is python-dbusmock, I just released the first tarball. With this you can easily create mock objects on D-Bus from any programming language with a D-Bus binding, or even from the shell.

The mock objects look like the real API (or at least the parts that you actually need), but they do not actually do anything (or only some action that you specify yourself). You can configure their state, behaviour and responses as you like in your test, without making any assumptions about the real system status.

When using a local system/session bus, you can do unit or integration testing without needing root privileges or disturbing a running system. The Python API offers some convenience functions like “start_session_bus()“ and “start_system_bus()“ for this, in a “DBusTestCase“ class (subclass of the standard “unittest.TestCase“).

Surprisingly I found very little precedence here. There is a Perl module, but that’s not particuarly helpful for test suites in C/Vala/Python. And there is Phil’s excellent Bendy Bus, but this has a different goal: If you want to thoroughly test a particular D-BUS service, such as ensuring that it does the right thing, doesn’t crash on bad input, etc., then Bendy Bus is for you (and python-dbusmock isn’t). However, it is too much overhead and rather inconvenient if you want to test a client-side program and just need a few system services around it which you want to set up in different states for each test.

You can use python-dbusmock with any programming language, as you can run the mocker as a normal program. The actual setup of the mock (adding objects, methods, properties, etc.) all happen via D-Bus methods on the “org.freedesktop.DBus.Mock“ interface. You just don’t have the convenience D-Bus launch API.

The simplest possible example is to create a mock upower with a single Suspend() method, which you can set up like this from Python:

import dbus
import dbusmock

class TestMyProgram(dbusmock.DBusTestCase):
[...]
    def setUp(self):
        self.p_mock = self.spawn_server('org.freedesktop.UPower',
                                        '/org/freedesktop/UPower',
                                        'org.freedesktop.UPower',
                                        system_bus=True,
                                        stdout=subprocess.PIPE)

        # Get a proxy for the UPower object's Mock interface
        self.dbus_upower_mock = dbus.Interface(self.dbus_con.get_object(
            'org.freedesktop.UPower', '/org/freedesktop/UPower'),
            'org.freedesktop.DBus.Mock')

        self.dbus_upower_mock.AddMethod('', 'Suspend', '', '', '')

[...]

    def test_suspend_on_idle(self):
        # run your program in a way that should trigger one suspend call

        # now check the log that we got one Suspend() call
        self.assertRegex(self.p_mock.stdout.readline(), b'^[0-9.]+ Suspend$')

This doesn’t depend on Python, you can just as well run the mocker like this:

python3 -m dbusmock org.freedesktop.UPower /org/freedesktop/UPower org.freedesktop.UPower

and then set up the mocks through D-Bus like

gdbus call --system -d org.freedesktop.UPower -o /org/freedesktop/UPower \
      -m org.freedesktop.DBus.Mock.AddMethod '' Suspend '' '' ''

If you use it with Python, you get access to the dbusmock.DBusTestCase class which provides some convenience functions to set up and tear down local private session and system buses. If you use it from another language, you have to call dbus-launch yourself.

Please see the README for some more details, pointers to documentation and examples.

Update: You can now install this via pip from PyPI or from the daily builds PPA.

Update 2: Adjusted blog entry for version 0.0.3 API, to avoid spreading now false information too far.

Read more
Jussi Pakkanen

A relatively large portion of software development time is not spent on writing, running, debugging or even designing code, but waiting for it to finish compiling. This is usually seen as necessary evil and accepted as an unfortunate fact of life. This is a shame, because spending some time optimizing the build system can yield quite dramatic productivity gains.

Suppose a build system takes some thirty seconds to run for even trivial changes. This means that even in theory you can do at most two changes a minute. In practice the rate is a lot lower. If the build step takes only a few seconds, trying out new code becomes a lot faster. It is easier to stay in the zone when you don’t have to pause every so often to wait for your tools to finish doing their thing.

Making fundamental changes in the code often triggers a complete rebuild. If this takes an hour or more (there are code bases that take 10+ hours to build), people try to avoid fundamental changes as much as possible. This causes loss of flexibility. It becomes very tempting to just do a band-aid tweak rather than thoroughly fix the issue at hand. If the entire rebuild could be done in five to ten minutes, this issue would become moot.

In order to make things fast, we first have to understand what is happening when C/C++ software is compiled. The steps are roughly as follows:

  1. Configuration
  2. Build tool startup
  3. Dependency checking
  4. Compilation
  5. Linking

We will now look at each step in more detail focusing on how they can be made faster.

Configuration

This is the first step when starting to build. Usually means running a configure script or CMake, Gyp, SCons or some other tool. This can take anything from one second to several minutes for very large Autotools-based configure scripts.

This step happens relatively rarely. It only needs to be run when changing configurations or changing the build configuration. Short of changing build systems, there is not much to be done to make this step faster.

Build tool startup

This is what happens when you run make or click on the build icon on an IDE (which is usually an alias for make). The build tool binary starts and reads its configuration files as well as the build configuration, which are usually the same thing.

Depending on build complexity and size, this can take anywhere from a fraction of a second to several seconds. By itself this would not be so bad. Unfortunately most make-based build systems cause make to be invocated tens to hundreds of times for every single build. Usually this is caused by recursive use of make (which is bad).

It should be noted that the reason Make is so slow is not an implementation bug. The syntax of Makefiles has some quirks that make a really fast implementation all but impossible. This problem is even more noticeable when combined with the next step.

Dependency checking

Once the build tool has read its configuration, it has to determine what files have changed and which ones need to be recompiled. The configuration files contain a directed acyclic graph describing the build dependencies. This graph is usually built during the configure step. Suppose we have a file called SomeClass.cc which contains this line of code:

#include "OtherClass.hh"

This means that whenever OtherClass.hh changes, the build system needs to rebuild SomeClass.cc. Usually this is done by comparing the timestamp of SomeClass.o against OtherClass.hh. If the object file is older than the source file or any header it includes, the source file is rebuilt.

Build tool startup time and the dependency scanner are run on every single build. Their combined runtime determines the lower bound on the edit-compile-debug cycle. For small projects this time is usually a few seconds or so. This is tolerable.

The problem is that Make scales terribly to large projects. As an example, running Make on the codebase of the Clang compiler with no changes takes over half a minute, even if everything is in cache. The sad truth is that in practice large projects can not be built fast with Make. They will be slow and there’s nothing that can be done about it.

There are alternatives to Make. The fastest of them is Ninja, which was built by Google engineers for Chromium. When run on the same Clang code as above it finishes in one second. The difference is even bigger when building Chromium. This is a massive boost in productivity, it’s one of those things that make the difference between tolerable and pleasant.

If you are using CMake or Gyp to build, just switch to their Ninja backends. You don’t have to change anything in the build files themselves, just enjoy the speed boost. Ninja is not packaged on most distributions, though, so you might have to install it yourself.

If you are using Autotools, you are forever married to Make. This is because the syntax of autotools is defined in terms of Make. There is no way to separate the two without a backwards compatibility breaking complete rewrite. What this means in practice is that Autotool build systems are slow by design, and can never be made fast.

Compilation

At this point we finally invoke the compiler. Cutting some corners, here are the approximate steps taken.

  1. Merging includes
  2. Parsing the code
  3. Code generation/optimization

Let’s look at these one at a time. The explanations given below are not 100% accurate descriptions of what happens inside the compiler. They have been simplified to emphasize the facets important to this discussion. For a more thorough description, have a look at any compiler textbook.

The first step joins all source code in use into one clump. What happens is that whenever the compiler finds an include statement like #include “somefile.h”, it finds that particular source file and replaces the #include with the full contents of that file. If that file contained other #includes, they are inserted recursively. The end result is one big self-contained source file.

The next step is parsing. This means analyzing the source file, splitting it into tokens and building an abstract syntax tree. This step translates the human understandable source code into a computer understandable unambiguous format. It is what allows the compiler to understand what the user wants the code to do.

Code generation takes the syntax tree and transforms it into machine code sequences called object code. This code is almost ready to run on a CPU.

Each one of these steps can be slow. Let’s look at ways to make them faster.

Faster #includes

Including by itself is not slow, slowness comes from the cascade effect. Including even one other file causes everything included in it to be included as well. In the worst case every single source file depends on every header file. This means that touching any header file causes the recompilation of every source file whether they use that particular header’s contents or not.

Cutting down on interdependencies is straightforward. Only #include those headers that you actually use. In addition, header files must not include any other header files if at all possible. The main tool for this is called forward declaration. Basically what it means is that instead of having a header file that looks like this:

#include "SomeClass.hh"

class MyClass {
  SomeClass s;
};

You have this:

class SomeClass;

class MyClass {
  SomeClass *s;
}

Because the definition of SomeClass is not know, you have to use pointers or references to it in the header.

Remember that #including MyClass.hh would have caused SomeClass.hh and all its #includes to be added to the original source file. Now they aren’t, so the compiler’s work has been reduced. We also don’t have to recompile the users of MyClass if SomeClass changes. Cutting the dependency chain like this everywhere in the code base can have a major effect in build time, especially when combined with the next step. For a more detailed analysis including measurements and code, see here.

Faster parsing

The most popular C++ libraries, STL and Boost, are implemented as header only libraries. That is, they don’t have a dynamically linkable library but rather the code is generated anew into every binary file that uses them. Compared to most C++ code, STL and Boost are complex. Really, really complex. In fact they are most likely the hardest pieces of code a C++ compiler has to compile. Boost is often used as a stress test on C++ compilers, because it is so difficult to compile.

It is not an exaggeration to say that for most C++ code using STL, parsing the STL headers is up to 10 times slower than parsing all the rest. This leads to massively slow build times because of class headers like this:

#include <vector>

class SomeClass {
private:
  vector<int> numbers;

public:
  ...
};

As we learned in the previous chapter, this means that every single file that includes this header must parse STL’s vector definition, which is an internal implementation detail of SomeClass and even if they would not use vector themselves. Add some other class include that uses a map, one for unordered_map, a few Boost includes and what do you end up with? A code base where compiling any file requires parsing all of STL and possibly Boost. This is a factor of 3-10 slowdown on compile times.

Getting around this is relatively simple, though takes a bit of work. It is known as the pImpl idiom. One way of achieving it is this:

---header---

struct someClassPrivate;

class SomeClass {
private:
  someClassPrivate *p;
};

---- implementation ---
#include <vector>
struct someClassPrivate {
  vector<int> numbers;
};

SomeClass::SomeClass() {
  p = new someClassPrivate;
}

SomeClass::~SomeClass() {
  delete p;
}

Now the dependency chain is cut and users of SomeClass don’t have to parse vector. As an added bonus the vector can be changed to a map or anything else without needing to recompile files that use SomeClass.

 Faster code generation

Code generation is mostly an implementation detail of the compiler, and there’s not much that can be done about it. There are a few ways to make it faster, though.

Optimizing code is slow. In every day development all optimizations should be disabled. Most build systems do this by default, but Autotools builds optimized binaries by default. In addition to being slow, this makes debugging a massive pain, because most of the time trying to print the value of some variable just prints out “value optimised out”.

Making Autotools build non-optimised binaries is relatively straightforward. You just have to run configure like this: ./configure CFLAGS=’O0 -g’ CXXFLAGS=’-O0 -g’. Unfortunately many people mangle their autotools cflags in config files so the above command might not work. In this case the only fix is to inspect all autotools config files and fix them yourself.

The other trick is about reducing the amount of generated code. If two different source files use vector<int>, the compiler has to generate the complete vector code in both of them. During linking (discussed in the next chapter) one of them is just discarded. There is a way to tell the compiler not to generate the code in the other file using a technique that was introduced in C++0x called extern templates. They are used like this.

file A:

#include <vector>
template class std::vector<int>;

void func() {
  std::vector<int> numbers;
}

file B:

#include <vector>
extern template class std::vector<int>;

void func2() {
  std::vector<int> idList;
}

This instructs the compiler not to generate vector code when compiling file B. The linker makes it use the code generated in file A.

Build speedup tools

CCache is an application that stores compiled object code into a global cache. If the same code is compiled again with the same compiler flags, it grabs the object file from the cache rather than running the compiler. If you have to recompile the same code multiple times, CCache may offer noticeable speedups.

A tool often mentioned alongside CCache is DistCC, which increases parallelism by spreading the build to many different machines. If you have a monster machine it may be worth it. On regular laptop/desktop machines the speed gains are minor (it might even be slower).

Precompiled headers

Precompiled headers is a feature of some C++ compilers that basically serializes the in-memory representation of parsed code into a binary file. This can then be read back directly to memory instead of reparsing the header file when used again. This is a feature that can provide massive speedups.

Out of all the speedup tricks listed in this post, this has by far the biggest payoff. It turns the massively slow STL includes into, effectively, no-ops.

So why is it not used anywhere?

Mostly it comes down to poor toolchain support. Precompiled headers are fickle beasts. For example with GCC they only work between two different compilation units if the compiler switches are exactly the same. Most people don’t know that precompiled headers exist, and those that do don’t want to deal with getting all the details right.

CMake does not have direct support for them. There are a few modules floating around the Internet, but I have not tested them myself. Autotools is extremely precompiled header hostile, because its syntax allows for wacky and dynamic alterations of compiler flags.

Faster Linking

When the compiler compiles a file and comes to a function call that is somewhere outside the current file, such as in the standard library or some other source file, it effectively writes a placeholder saying “at this point jump to function X”. The linker takes all these different compiled files and connects the jump points to their actual locations. When linking is done, the binary is ready to use.

Linking is surprisingly slow. It can easily take minutes on relatively large applications. As an extreme case, linking the Chromium browser on ARM takes 3 gigs of RAM and takes 18 hours.

Yes, hours.

The main reason for this is that the standard GNU linker is quite slow. Fortunately there is a new, faster linker called Gold. It is not the default linker yet, but hopefully it will be soon. In the mean time you can install and use it manually.

A different way of making linking faster is to simply cut down on these symbols using a technique called symbol visibility. The gist of it is that you hide all non-public symbols from the list of exported symbols. This means less work and memory use for the linker, which makes it faster.

Conclusions

Contrary to popular belief, compiling C++ is not actually all that slow. The STL is slow and most build tools used to compile C++ are slow. However there are faster tools and ways to mitigate the slow parts of the language.

Using them takes a bit of elbow grease, but the benefits are undeniable. Faster build times lead to happier developers, more agility and, eventually, better code.

Read more
Jussi Pakkanen

Say you start work on a new code base. Would you, as a user, rather have 90% or 10% of its API functions commented with Doxygen or something similar?

Using my psychic powers I suspect that you chose 90%.

It seems like the obvious thing. Lots of companies even have a mandate that all API functions (or >90% of them) must be documented. Not having comments is just bad. This seems like a perfectly obvious no-brainer issue.

But is it really?

Unfortunately there are some problems with this assumption. The main one being that the comments will be written by human beings. What they probably end up being is something like this.

/*
 * Takes a foobar and frobnicates it.
 *
 * @param f the foobar to be frobnicated.
 * @param strength how strongly to frobnicate.
 * @return the frobnicated result.
 */
int frobnicate_foobar(Foobar f, int strength);

This something I like to call documentation by word order shuffle. Now we can ask the truly relevant question: what additional information does this kind of a comment provide?

The answer is, of course, absolutely nothing. It is only noise. No, actually it is even worse: it is noise that has a very large probability of being wrong. When some coder changes the function, it is very easy to forget to update the comments.

On the other hand, if only 10% of the functions are documented, most functions don’t have any comments, but the ones that do probably have something like this:

/** 
 * The Foobar argument must not have been initialized in a different
 * thread because that can lead to race conditions.
 */ 
int frobnicate_foobar(Foobar f, int strength)

This is the kind of comment that is actually useful. Naturally it would be better to check for the specified condition inside the function but sometimes you can’t. Having it in a comment is the right thing to do in these cases. Not having tons of junk documentation makes these kinds of remarks stand out. This means, paradoxically, that having less comments leads to better documentation and user experience.

As a rough estimate, 95% of functions in any code base should be so simple and specific that their signature is all you need to use them. If they are not, API design has failed: back to the drawing board.

Read more
Jussi Pakkanen

Developing with the newest of the new packages is always a bit tricky. Every now and then they break in interesting ways. Sometimes they corrupt the system so much that downgrading becomes impossible. Extreme circumstances may corrupt the system’s package database and so on. Traditionally fixing this has meant reinstalling the entire system, which is unpleasant and time consuming. Fortunately there is now a better way: snapshotting the system with btrfs.

The following guide assumes that you are running btrfs as your root file system. Newest quantal can boot off of btrfs root, but there may be issues, so please read the documentation in the wiki.

The basic concept in snapshotting your system is called a subvolume. It is kind of like a subpartition inside the main btrfs partition. By default Ubuntu’s installer creates a btrfs root partition with two subvolumes called @ and @home. The first one of these is mounted as root and the latter as the home directory.

Suppose you are going to do something really risky, and want to preserve your system. First you mount the raw btrfs partition somewhere:

sudo mkdir /mnt/root
sudo mount /dev/sda1 /mnt/root
cd /mnt/root

Here /dev/sda1 is your root partition. You can mount it like this even though the subvolumes are already mounted. If you do an ls, you see two subdirectories, @ and @home. Snapshotting is simple:

sudo btrfs subvolume snapshot @ @snapshot-XXXX

This takes maybe on second and when the command returns the system is secured. You are now free to trash your system in whatever way you want, though you might want to unmount /mnt/root so you don’t accidentally destroy your snapshots.

Restoring the snapshot is just as simple. Mount /mnt/root again and do:

sudo mv @ @broken
sudo subvolume snapshot @snapshot-XXXX @

If you are sure you don’t need @snapshot-XXX any more, you can just rename it @. You can do this even if you are booted in the system, i.e. are using @ as your current system root fs.

Reboot your machine and your system has been restored to the state it was when running the snapshot command. As an added bonus your home directory does not rollback, but retains all changes made during the trashing, which is what you want most of the time. If you want to rollback home as well, just snapshot it at the same time as the root directory.

You can get rid of useless and broken snapshots with this command:

sudo btrfs subvolume delete @useless-snapshot

You can’t remove subvolumes with rm -r, even if run with sudo.

Read more
Jussi Pakkanen

The conventional wisdom in build systems is that GNU Autotools is the one true established standard and other ones are used only rarely.

But is this really true?

I created a script that downloads all original source packages from Ubuntu’s main pool. If there were multple versions of the same project, only the newest was chosen. Then I created a second script that goes through those packages and checks what build system they actually use. Here’s the breakdown:

CMake:           348     9%
Autofoo:        1618    45%
SCons:            10     0%
Ant:             149     4%
Maven:            41     1%
Distutil:        313     8%
Waf:               8     0%
Perl:            341     9%
Make(ish):       351     9%
Customconf:       45     1%
Unknown:         361    10%

Here Make(ish) means packages that don’t have any other build system, but do have a makefile. This usually indicates building via custom makefiles. Correspondingly customconf is for projects that don’t have any other build system, but have a configure file. This is usually a handwritten shell or Python file.

This data is skewed by the fact that the pool is just a jumble of packages. It would be interesting to run this analysis separately for precise, oneiric etc to see the progression over time. For truly interesting results you would run it against the whole of Debian.

The relative popularity of CMake and Autotools is roughly 20/80. This shows that Autotools is not the sole dominant player for C/C++ it once was. It’s still far and away the most popular, though.

The unknown set contains stuff such as Rake builds. I simply did not have time to add them all. It also has a lot of fonts, which makes sense, since you don’t really build those in the traditional sense.

The scripts can be downloaded here. A word of warning: to run the analysis you need to download 13 GB of source. Don’t do it just for the heck of it. The parser script does not download anything, it just produces a list of urls. Download the packages with wget -i.

Some orig packages are compressed with xz, which Python’s tarfile module can’t handle. You have to repack them yourself prior to running the analysis script.

Read more
pitti

I just released PyGObject 3.3.92, for GNOME 3.5.92.

There is nothing too exciting in this release; a couple of small bug fixes and a lot of new test cases. See the detailled list of changes below.

Thanks to all contributors!

Changes:

  • release-news: Generate HTML changelog (Martin Pitt)
  • [API add] Add ObjectInfo.get_abstract method (Simon Feltman) (#675581)
  • Add deprecation warning when setting gpointers to anything other than int. (Simon Feltman) (#683599)
  • test_properties: Test accessing a property from a superclass (Martin Pitt) (#684058)
  • test_properties.py: Consistent test names (Martin Pitt)
  • test_everything: Ensure TestSignals callback does get called (Martin Pitt)
  • argument: Fix 64bit integer convertion from GValue (Nicolas Dufresne) (#683596)
  • Add Simon Feltman as a project maintainer (Martin Pitt)
  • test_signals.py: Drop global type variables (Martin Pitt)
  • test_signals.py: Consistent test names (Martin Pitt)
  • Add test cases for GValue signal arguments (Martin Pitt) (#683775)
  • Add test for GValue signal return values (Martin Pitt) (#683596)
  • Improve setting pointer fields/arguments to NULL using None (Simon Feltman) (#683150)
  • Test gint64 C signal arguments and return values (Martin Pitt)
  • Test in/out int64 GValue method arguments. (Martin Pitt) (#683596)
  • Bump g-i dependency to 1.33.10 (Martin Pitt)
  • Fix -uninstalled.pc.in file (Thibault Saunier) (#683379)

Read more
Daniel Holbach

I’m quite happy with the progress the Packaging Guide is making. We managed to fix a bunch of bugs this cycle and most importantly we got it into Ubuntu and made it translatable. We only opened translations a couple of weeks ago, but some language teams have been hard at work:

  1. pt_BR.po (18%)
  2. ja.po (14%)
  3. ru.po (9%)
  4. es.po (5%)
  5. id.po (4%)
    de.po (4%)
  6. nl.po (1%)
  7. sv.po (0%)
    fr.po (0%)
    lv.po (0%)
    zh_TW.po (0%)
    hu.po (0%)
    ca.po (0%)

At UDS we decided that for translations which came to a percentage of completion of >= 70% we would build separate packages for those languages. Up until to that percentage we will only keep the translations in Launchpad.

This means there is still some way to go for all of us, but this is a great great step already. Thanks a lot for your hard work on this!

There are obviously many more bugs to fix and we’d love your help.

Bitesize bugs:

Make it prettier:

One bug we’d love to see some help with is #1043232 Packaging Guide FTBFS – it looks like the build fails due to Japanese translations. Right now all translations are disabled, which serves as a workaround for now.

Thanks again to everyone who helped out with the Packaging Guide. Your help has got many many contributors on their way. Keep up the good work!

Read more
Daniel Holbach

We need some feedback. Can you please leave a comment with the information

  • you wish you had had heard when you got involved with Ubuntu development
  • you want to share with new starters in Ubuntu development
  • you learnt and found invaluable

As you can imagine, your feedback is going to make the experience for new contributors even better. Thanks a lot in advance.

Read more
pitti

I just released PyGObject 3.3.91, for GNOME 3.5.91.

The big new feature in this release (thanks to the release team for granting an exception) is Simon Feltman’s new Signal helper class, which makes defining custom signals a whole lot simpler and more obvious. In the past, you had to do

 class C(GObject.GObject):
    __gsignals__ = {
        'my_signal': (GObject.SIGNAL_RUN_FIRST, GObject.TYPE_NONE,
                      (GObject.TYPE_INT,))
    }

    def do_my_signal(self, arg):
        print("my_signal called with %i" % arg)

whereas now this looks like

class C(GObject.GObject):
    @GObject.Signal(arg_types=(int,))
    def my_signal(self, arg):
        print("my_signal called with %i" % arg)

or even more elegantly when using Python 3 and its new type annotations:

class C(GObject.GObject):
    @GObject.Signal
    def my_signal(self, arg:int):
        print("my_signal called with %i" % arg)

Check out the updated example and docstring for other ways how to use it.

Overrides can now be in a directory different from the one that pygobject installs itself into. These overrides need to put this into their __init__.py at the top:

from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

and put themselves somewhere into the default PYTHONPATH. This should make it a lot easier for library packages to ship their own overrides for Python.

This new version also comes with a couple of new overrides and bug fixes. See the detailled list of changes below.

Thanks to all contributors!

  • Fix exception test case for Python 2 (Martin Pitt)
  • Bump g-i dependency to >= 1.3.9 (Martin Pitt)
  • Show proper exception when trying to allocate a disguised struct (Martin Pitt) (#639972)
  • Support marshalling GParamSpec signal arguments (Mark Nauwelaerts) (#683099)
  • Add test for a signal that returns a GParamSpec (Martin Pitt) (#683265)
  • [API add] Add Signal class for adding and connecting custom signals. (Simon Feltman) (#434924)
  • Fix pygtkcompat’s Gtk.TreeView.insert_column_with_attributes() (Martin Pitt)
  • Add override for Gtk.TreeView.insert_column_with_attributes() (Marta Maria Casetti) (#679415)
  • .gitignore: Add missing built files (Martin Pitt)
  • Ship tests/gi in tarball (Martin Pitt)
  • Split test_overrides.py (Martin Pitt) (#683188)
  • _pygi_argument_to_object(): Clean up array unmarshalling (Martin Pitt)
  • Fix memory leak in _pygi_argument_to_object() (Alban Browaeys) (#682979)
  • Fix setting pointer fields/arguments to NULL using None. (Simon Feltman) (#683150)
  • Fix for python 2.6, officially drop support for < 2.6 (Martin Pitt) (#682422)
  • Allow overrides in other directories than gi itself (Thibault Saunier) (#680913)
  • Clean up sys.path handling in tests (Simon Feltman) (#680913)
  • Fix dynamic creation of enum and flag gi types for Python 3.3 (Simon Feltman) (#682323)
  • [API add] Override g_menu_item_set_attribute (Paolo Borelli) (#682436)

Read more
Anthony Dillon

I wondered if I could make an easily updateable, prototype site in order to test a number of different IA’s using an XML file to represent the sitemap. This post is about what I did and how to get some sample code if you want to use or extend it for yourself.

The XML

The goal is to be able to simply edit, delete or add a section in the XML and refresh the site. There you have it. The navigation menu has changed.

Each <page> child of <sitemap> is top level navigation. You add children to that by placing <page> nodes inside each other.

<page title="Contact" url="contact-me">
  • title – The name of the menu item and the title of the page
  • url – The permalink for that page, this is added to the parents name if a child

Below is an example navigation to build a personal website:

<sitemap>
   <page title="Work" url="work">
       <page title="Develop" url="develop">
           <page title="Web development" url="web-development"></page>
           <page title="Apps" url="apps"></page>
       </page>
       <page title="Design" url="design">
           <page title="Web design" url="web-design"></page>
           <page title="Magazines" url="magazines"></page>
           <page title="Cartoons" url="cartoons"></page>
       </page>
       <page title="Videos" url="videos">
           <page title="Animation" url="animation"></page>
           <page title="Live footage" url="live-footage"></page>
           <page title="Showreel" url="showreel"></page>
       </page>
   </page>
   <page title="About Me" url="about-me">
       <page title="Life" url="life">
           <page title="Photos" url="photos"></page>
           <page title="Videos" url="videos"></page>
           <page title="Inspiration" url="inspiration"></page>
       </page>
           <page title="Socal" url="social">
           <page title="Friends" url="friends"></page>
           <page title="Family" url="family"></page>
           <page title="Things I've done" url="done"></page>
           <page title="Bucket list" url="bucket-list"></page>
       </page>
   </page>
   <page title="Contact" url="contact-me"></page>
</sitemap>

The system has only been developed to navigate three levels deep.

Adverts

I also wanted to have little ads in the corner of some of the mega menu dropdowns. To do this, there is an promo.xml in the XML folder. Simply wrap a title, img, p and link, if you would like, in the url of the top level link. This will result in an advert style banner on the right side of the menu. You can restyle this banner with CSS.

Here is the example of promo.xml:

<adverts>
    <work>
        <title>Latest website</title>
        <img>assets/images/200x150-b8b8b8-fff.jpg</img>
        <link url="#work/develop/web-development">See more</link>
    </work>
    <about-me>
        <title>My picture</title>
        <img>assets/images//200x150-b8b8b8-fff.jpg</img>
        <p>This is a little blarb all about what I do and where I am.</p>
    </about-me>
</adverts>

This is a little blarb all about what I do and where I am.

Hash bang

Using hash bangs, the site does not reload the page when a menu item is clicked. The JavaScript takes the updated hash value and updates the page values to mimic a new page. There are even three different page contents that are changed to reinforce to the user the fact we have changed page.

Demo

Check out the demo: http://www.anthonydillon.com/navigation-prototyping

The Result

The results were good. We were able to use this code to test three potential IAs and the mega menu itself with ten users. The XML allowed the me to easily tweak the menus between tests and it looked real enough that the users weren’t distracted by the lack of content behind the pages.

The Code

Feel free to use the code for quick navigation prototyping at: https://launchpad.net/navigation-prototyping

If you would like contribute to this project all your help is appreciated!

Read more
Daniel Holbach

I talked many times about getting involved with developing Ubuntu and how it can seem daunting and that there’s much to learn. When I talked to contributors who had reached the critical point where they understood what they can do, who they can talk to and how the processes roughly work, most of them said that three things helped them to get to the point:

Code review

Today I want to talk about code reviews. It’s probably the most straight-forward way to learn by osmosis: you easily pick up conventions, distinctions which are made and which processes to follow.

Everybody has to go through code reviews, no matter which team they are in, which company they work for or when they joined the project. Up until a point they get their developer application approved and get upload rights.

This is the reason why code reviews in Ubuntu are so important and why we should constantly strive for timely replies and decisions on review requests.

Sponsoring Queue

The sponsoring queue is reviewed by developers with upload rights. Sometimes it’s very easy to approve a request and upload the package, sometimes it takes a bit longer, especially when you have a comment ping-pong between the reviewer and the patch author.

We came up with a number of points in our documentation which should help keeping the queue manageable:

For Bugs fixing small details, you could do the following:

  1. Ask the contributor to forward the patch upstream.
  2. Open an empty upstream bug task.
  3. Mark the Ubuntu task as ‘Fix Committed’.
  4. Unsubscribe ubuntu-sponsors, or mark the merge proposal status as “Work in Progress”. (Be sure to tell the contributor to reverse the process.)

This will get the review item off the list for the time being and once we can import the code from upstream, it will get fully closed.

We also get requests which are not suitable for the current release period. In this case you could:

  1. Let the contributor know that the patch is not suitable for the current release period.
  2. Unsubscribe ubuntu-sponsors, or mark the merge proposal status as “Work in Progress”. (Be sure to tell the contributor to reverse the process.)
  3. Subscribe yourself to the bug report.
  4. Milestone the bug to ‘later’.
  5. Visit https://bugs.launchpad.net/people/+me/+bugs/?field.milestone%3Alist=196 once the new release opens and upload the fix.

This are just some points which help to keep things on the queue relevant.

Patch Pilots

From the Bazaar team we borrowed the scheme of “patch pilots”. Here’s how they explain how it works: “The word pilot is in the sense of a maritime pilot: we help patches come through congested waters safely in to harbour. The main thing to watch is the bzr active reviews page in Launchpad. When you’re piloting, put some concentrated effort into helping people have a good and satisfying experience contributing to Bazaar. Just how you do that is up to you.

Instead of trying to review each and every bit in the queue – sometimes there are packages you know less about and where you can’t make a decision for example – you try to help nudge the patch along. You help to talk to upstream about it, try to find somebody who can make a decision, etc.

Canonical engineers with upload rights who work on Ubuntu are expected to spend an hour per week on the Ubuntu sponsoring queue, so everybody’s on the hook for having a piloting shift 4h every four weeks. This usually works much better, as you have an extended period of time where you do nothing else. Current patch pilots can be seen in the #ubuntu-devel channel topic.

Up until now I mostly noticed Canonical engineers who did piloted. If you have upload rights and are interested, let me know and I can add you to a preliminary schedule, so you get a reminder mail and you can try it out and see if you like it.

Please all help making this work even better. :-)

Read more
pitti

I just released PyGObject 3.3.90, for GNOME 3.5.90.

This is now working correctly on big-endian 64 bit machines such as powerpc64, and fixes marshalling for GParamSpec attributes and return values, as well as a few small bug fixes.

Thanks to all contributors!

Complete list of changes:

  • Implement marshalling for GParamSpec (Mathieu Duponchelle) (#681565)
  • Fix erronous import statements for Python 3.3 (Simon Feltman) (#682051)
  • Do not fail tests if pyflakes or pep8 are not installed (Martin Pitt)
  • Fix PEP-8 whitespace checking and issues in the code (Martin Pitt)
  • Fix unmarshalling of gssize (David Malcolm) (#680693)
  • Fix various endianess errors (David Malcolm) (#680692)
  • Gtk overrides: Add TreeModelSort.__init__(self, model) (Simon Feltman) (#681477)
  • Convert Gtk.CellRendererState in the pygi-convert script (Manuel Quiñones) (#681596)

Read more
Anthony Dillon

We realise that changing Operating System (OS) is a big thing for anyone thinking of testing something out. That becomes a huge barrier for people trying out Ubuntu for the first time and seeing if they like it. As a member of the web team I decided to take on the challenge as a cool way to testing out some HTML5 and jQuery. The purpose of this blog is to talk about some of the challenges and thought processes I went through during the build.

Getting started

I started by breaking down the Ubuntu interface into DOM elements, similar to developing a web page from a design. So I took on the background first, each part of the build brought different challenges. In the case of the background, I needed it to stretch full width and height of the viewport.

The launcher

Once I had that nailed, I moved onto the launcher on the left of the screen. At the time of building (11.10) the menu would slide out of view when a window was fullscreen. By creating a DIV with a list of icons and styled with CSS to replicate the menu in the OS. Now I needed to add the interactive slide to the menu, which I decided to apply with jQuery because I wanted the animation to work on older browsers and not just browsers with CSS3. When a window is fullscreened I would trigger the menu system to slide the menu out of the viewport. Then binds a mouseover event to slide the menu back into view when the user needs it again.

Precise launcher

The global menu bar

Precise global menu bar

All that was left to make the web page look like the static Ubuntu interface was to add the global menu bar at the top for settings and window controls. This was again a simple DIV with a window control DIV floated to the left and a list of settings icons floated to the right.

Building applications

Now the interface looked like the initial Ubuntu screen. I turned to focus on applying the functionality to the menu icons, beginning with the folder icon. Created a Folder class and a File class, Folder class below:


function Folder($name, $location){
  if ($name==undefined) { $name='Untitled Folder';}
  if ($location==undefined) { $location='/Home';}
  var _name = $name;
  var _size = '6.2 GB';
  var _location = $location;
  var _in_bin = false;this.name = function (){ return _name; };
  this.size = function (){ return _size; };
  this.location = function (){ return _location; };
  this.in_bin = function (){ return _in_bin; };
  this.rename = function($name){
    _name = $name;
  }

  this.bin = function($in_bin){
    _in_bin = $in_bin;
  }

  this.move = function($location){
    _location = $location;
  }

  this.drawIcon = function($id, $type){
    return  "<div class='file-folder "+$type+"' data-type='folder' data-id='"+$id+"'><p></p><span>"+_name+"</span></div>";
  }

  this.type = function(){
    return 'folder';
  }
}

Both of these classes are used by the file system class that created them and stored them in an array to access and modify them later.
Now that we have a file system, the fun part comes when you connect it all up and see if it comes to life. This is how the connections happened:


Menu window workflow


When the folder icon on the menu is clicked it tells Base which triggers the open function in the folder system. I built the Folder and File classes with the aim to give the folder system full functionality, like rename, drag and drop to move, etc. But that was descoped to finish the tour in time for the release of 11.10.

Seeing this work I felt the buzz to jump to the next application, then the next, until all the applications I set out to develop were finished and working.

Have a go for yourself: Ubuntu Online Tour

Conclusion

One of the questions I was asked is `how long did it take you?’ In total it took about 4 months, from what started as a quick test to being given the time to bring the development to fruition. It was crucial that all elements of the demo looked and felt exactly like the OS as this will be most people first try of Ubuntu so we didn’t want it to be different if they install Ubuntu.

I plan to keep the tour up-to-date with the latest version of Ubuntu every six months. With every update I try to add features and application which I haven’t managed yet. A few features that come to mind would be to progressive enhance the movie player to use HTML5 video if available rather than Flash. I would also like to see the music player (Rhythmbox) be developed.

If you have HTML, CSS and jQuery skills and time to spare. Please feel free to contribute. The code can be forked from: Ubuntu Online Tour Launchpad Page.

Read more
pitti

I started to collect some easy PyGObject bugs which are appropriate for the PyGObject hackfest at GUADEC on July 30th. These are bugs which do not need a lot of previous knowlege and are excellent starters for new contributors, such as adding overrides, fixing build system issues, etc.

I also created an initial idea pool/agenda/coordination page, where participants can add or signup for things to work on.

Feel free to add your own topics! I’m really looking forward to GUADEC and the hackfest, see you there!

GUADEC 2012

Read more
pitti

I released PyGObject 3.3.4. This is mostly a bug fix only release to fix existing API. Highlights are that lists of GVariants and other corner cases are now working correctly when being passed from C to Python, and that calling help() on a GI module now does something sensible.

Thanks to all contributors!

Complete list of changes:

  • pygi-convert.sh: Drop bogus filter_new() conversion (Martin Pitt) (#679999)
  • Fix help() for GI modules (Martin Pitt) (#679804)
  • Skip gi.CallbackInfo objects from a module’s dir() (Martin Pitt) (#679804)
  • Fix __path__ module attribute (Martin Pitt)
  • pygi-convert.sh: Fix some child ? getChild() false positives (Joe R. Nassimian) (#680004)
  • Fix array handling for interfaces, properties, and signals (Mikkel Kamstrup Erlandsen) (#667244)
  • Add conversion of the Gdk.PropMode constants to pygi-convert.sh script (Manuel Quiñones) (#679775)
  • Add the same rules for pack_start to convert pack_end (Manuel Quiñones) (#679760)
  • Add error-checking for the case where _arg_cache_new() fails (Dave Malcolm) (#678914)
  • Add conversion of the Gdk.NotifyType constants to pygi-convert.sh script (Manuel Quiñones) (#679754)
  • Fix PyObject_Repr and PyObject_Str reference leaks (Simon Feltman) (#675857)
  • [API add] Gtk overrides: Add TreePath.__len__() (Martin Pitt) (#679199)
  • GLib.Variant: Fix repr(), add proper str() (Martin Pitt) (#679336)
  • m4/python.m4: Update Python version list (Martin Pitt)
  • Remove “label” property from Gtk.MenuItem if it is not set (Micah Carrick) (#670575)

Read more
Laura czajkowski

Location: Flexible. If home based, reliable broadband connectivity required.

Role Summary

Do you want to be one of the engineers building the infrastructure at the heart of the cloud revolution?

At Canonical we’re developing technologies that are key to the transition to the cloud, with Ubuntu as the number one cloud operating system. We are looking for a fun, talented software engineer whose ingenuity, self-motivation and engineering skill have contributed to a shining track record of successful projects.

Alongside four or five other engineers, you’ll be part of an agile engineering squad, in Canonical’s Launchpad team, working in either a new development or maintenance role on a different cloud-related project every six to nine months. Your work will touch projects such as OpenStack, MAAS, AWSome and the Launchpad SaaS developer tools platform.

To succeed you’ll need to share our love of hard work and our passion for free software, Ubuntu and the cloud.

Your energy and enthusiasm will be key to delivering the project, and to making the squad fun to be a part of.

Key Skills and Accountabilities

  • Develop new features in existing web or cloud applications or even start new ones from scratch.
  • Participate in the maintenance of the portfolio of applications maintained by the Launchpad team (a group of six development squads).
  • Collaborate within a small team of four to five engineers to design and deliver agreed features on an established schedule.
  • Ensure high quality results from across the team by participating in established team practices such as code review and testing.
  • Maintain readable developer-oriented documentation.
  • Coordinate regularly with the rest of the Launchpad team.

Required Skills and Experience

  • You have extensive experience in development of web applications using a major object or oriented application framework
  • You are proficient with the technologies powering the web such as Python, HTTP, HTML, CSS and JavaScript
  • You live and breathe open source technology.  You know the industry, understand the community and share the ideals.  You know your OpenStack from your intel, your ARM from your aaS and your Bugzilla from your Git
  • You are well experienced with at least one web application framework, such as Rails, Django, Zope/Plone, Pyramid, Turbogears, Web Objects, etc
  • You are well experienced with at least one JavaScript library/framework such as YUI 3/2, jQuery, Dojo, MooTools, or Prototype
  • You love easy to use software and pay particular attention to making your applications a joy to use
  • You have created stellar user interfaces using JavaScript, HTML and CSS
  • You’re skilled in object-oriented programming in the Python language
  • How people solve complex problems in software fascinates you.  You also know that reliable and maintainable code are essential to long-term success.  You’re familiar with writing about what needs to be done, as well as test-driven development and other “agile methods
  • You have strong spoken English communication skills, and can communicate clearly in writing, including email and IRC environments.
  • You have a good sense of humour and enjoy building a fun working environment with your colleagues.
  • You are willing to travel internationally, for periods of one or two weeks and occasionally longer, for conferences, developer-oriented meetings and sprints

Desired Skills and Experience

  • You are familiar with interaction design and have contributed to the user interface of a leading web application.
  • You have built and managed a community around an open source project
  • You have contributed code to an open source project
  • You understand the basics of one or more of the following:
    • laaS platforms such as OpenStack, AWS, Eucalyptus
    • Ubuntu Server, particularly in cloud contexts
    • ARM server
    • Services Oriented Architecture
    • Message-passing systems
    • Distributed version control systems
    • A form of Linux packaging, such as .deb or .rpm
  • You are familiar with Agile/Lean development practices
  • You enjoy exploring new languages like Go, Haskell or Clojure
  • You have system programming experience in C
  • You worked as part of a distributed software engineering team and can demonstrate the self-motivation and discipline required in such an environment

Apply online, or talk to us in #launchpad-dev if you want to see what we do!

Read more
pitti

I just received confirmation that my request for a PyGObject hackfest has been approved by the GUADEC organizers.

If you are developing GObject-introspection based Python applications and have some problems with PyGObject, this is the time and place to get to know each other, getting bugs fixed, learn about pygobject’s innards, or update libraries to become introspectable. I will prepare a list of easy things to look into if you are interested in learning about and getting involved in PyGObject’s development.

See you on July 30th in A Coruña!

GUADEC Badge

Read more
pitti

I released PyGObject 3.3.3.

The most notable changes are that you can now access methods (and other identifiers) which are Python keywords, PyGObject automatically escapes them now by appending a ‘_’. For example, you can now call myGdkWindow.raise_() or GLib.Thread.yield_() instead of having to resort to the previous workaround getattr(myGdkWindow, 'raise')().

This version also restores the deprecated get_data() and set_data() methods. They were never really meant to be used from Python programs, they can potentially mess up your program and cause crashes, and do not give you anything that regular Python object properties would not already provide in a much safer way (i. e. just write my_obj.foo = 'bar' instead of my_obj.set_data('foo', 'bar')). Apparently some software projects are using them, so they will now raise a deprecation warning and be removed for the GNOME 3.8 cycle instead.

Thanks to all contributors!

Complete list of changes:

  • Remove obsolete release-tag make target (Martin Pitt)
  • Do not do any python calls when GObjects are destroyed after the python interpreter has been finalized (Simon Schampijer) (#678046)
  • Do not change constructor-only “type” Window property (Martin Pitt) (#678510)
  • Escape identifiers which are Python keywords (Martin Pitt) (#676746)
  • Fix code for PEP-8 violations detected by the latest pep8 checker. (Martin Pitt)
  • Fix crash in GLib.find_program_in_path() (Martin Pitt) (#678119)
  • Revert “Do not bind gobject_get_data() and gobject_set_data()” (Martin Pitt) (#641944)
  • GVariant: Raise proper TypeError on invalid tuple input (David Keijser) (#678317)

Update:Just released 3.3.3.1 to fix a regresssion from the keyword escaping patch. It also escaped enum and flags names, but as they are translated to upper case they are never keywords.

Read more