Canonical Voices

Posts tagged with 'development'


With python-dbusmock you can provide mocks for arbitrary D-BUS services for your test suites or if you want to reproduce a bug.

However, when writing actual tests for gnome-settings-daemon etc. I noticed that it is rather cumbersome to always have to set up the “skeleton” of common services such as UPower. python-dbusmock 0.2 now introduces the concept of “templates” which provide those skeletons for common standard services so that your code only needs to set up the particular properties and specific D-BUS objects that you need. These templates can be parameterized for common customizations, and they can provide additional convenience methods on the org.freedesktop.DBus.Mock interface to provide more abstract functionality like “add a battery”.

So if you want to pretend you have one AC and a half-charged battery, you can now simply do

  def setUp(self):
     (self.p_mock, self.obj_upower) = self.spawn_server_template('upower', {})

  def test_ac_bat(self):
     self.obj_upower.AddAC('mock_AC', 'Mock AC')
     self.obj_upower.AddChargingBattery('mock_BAT', 'Mock Battery', 50.0, 1200)

Or, if your code is not in Python, use the CLI/D-BUS interface, like in shell:

  # start a fake system bus
  eval `dbus-launch`

  # start mock upower on the fake bus
  python3 -m dbusmock --template upower &

  # add devices
  gdbus call --system -d org.freedesktop.UPower -o /org/freedesktop/UPower \
      -m org.freedesktop.DBus.Mock.AddAC mock_ac 'Mock AC'
  gdbus call --system -d org.freedesktop.UPower -o /org/freedesktop/UPower \
      -m org.freedesktop.DBus.Mock.AddChargingBattery mock_bat 'Mock Bat' 50.0 1200

In both cases upower --dump or gnome-power-statistics will show you the expected devices (of course you need to run that within the environment of the fake $DBUS_SYSTEM_BUS_ADDRESS, or run the mock on the real system bus as root).

Iftikhar Ahmad contributed a template for NetworkManager, which allows you to easily set up ethernet and wifi devices and wifi access points. See pydoc3 dbusmock.templates.networkmanager for details and the test cases for how this looks like in practice.

I just released python-dbusmock 0.2.1 and uploaded the new version to Debian experimental. I will sync it into Ubuntu Raring in a few hours.

Read more

I just released PyGObject 3.4.2, a bug fix release for GNOME 3.6.2.

Thanks to all contributors!

  • Fix marshalling of GByteArrays (Martin Pitt)
  • Fix marshalling of ssize_t to smaller ints (Martin Pitt)
  • Fix crash with GLib.child_watch_add (Daniel Narvaez) (#688067)
  • Fix various bugs in GLib.IOChannel (Martin Pitt)
  • Work around wrong 64 bit constants in GLib Gir (Martin Pitt) (#685022)
  • Fix OverflowError in source_remove() (Martin Pitt) (#684526)
  • Fix Signal decorator to not use base class gsignals dict (Simon Feltman) (#686496)

Read more

For writing tests for GVFS (current tests, proposed improvements) I want to run Samba as normal user, so that we can test gvfs’ smb backend without root privileges and thus can run them safely and conveniently in a “make check” environment for developers and in JHBuild for continuous integration testing. Before these tests could only run under gvfs-testbed, which needs root.

Unlike other servers such as ssh or ftp, this turned out surprisingly non-obvious and hard, so I want to document it in this blog post for posterity’s benefit.

Running the server

Running smbd itself is mainly an exercise of figuring out all the options that you need to set; Alex Larsson and I had some fun figuring out all the quirks and hiccups that happen between Ubuntu’s and Fedora’s packaging and 3.6 vs. 4.0, but finally arrived at something working.

First, you need to create an empty directory where smbd can put all its databases and state files in. For tests you would use mkdtemp(), but for easier reading I just assume mkdir /tmp/samba here.

The main knowledge is in the Samba configuration file, let’s call it /tmp/smb.conf:

workgroup = TESTGROUP
interfaces = lo
smb ports = 1445
log level = 2
map to guest = Bad User
passdb backend = smbpasswd
smb passwd file = /tmp/smbpasswd
lock directory = /tmp/samba
state directory = /tmp/samba
cache directory = /tmp/samba
pid directory = /tmp/samba
private dir = /tmp/samba
ncalrpc dir = /tmp/samba

path = /tmp/public
guest ok = yes

path = /tmp/private
read only = no

For running this as a normal user you need to set a port > 1024, so this uses 1445 to resemble the original (privileged) port 445. The map to guest line makes anonymous logins work on Fedora/Samba 4.0 (I’m not sure whether it’s a distribution or a version issue). Don’t ask about “dir” vs. “directory”, that’s an inconsistency in Samba; with above names it works on both 3.6 and 4.0.

We use the old “smbpasswd” backend as shipping large tdb files is usually too inconvenient and brittle for test suites. I created an smbpasswd file by running smbpasswd on a “real” Samba installation, and then using pdbedit to convert it to a smbpasswd file:

sudo smbpasswd -a martin
sudo pdbedit -i tdbsam:/var/lib/samba/passdb.tdb -e smbpasswd:/tmp/smbpasswd

The result for password “foo” is


which you are welcome to copy&paste (you can replace “myuser” with any valid user name, of course).

This also defines two shares, one public, one authenticated. You need to create the directories and populate them a bit:

mkdir /tmp/public /tmp/private
echo hello > /tmp/public/hello.txt
echo secret > /tmp/private/myfile.txt

Now you can run the server with

smbd -iFS -s /tmp/smb.conf

The main problem with this approach is that smbd exits (“Server exit (failed to receive smb request)”) after a client terminates, so you need to write your tests in a way to only run one connection/request per test, or to start smbd in a loop.

Running the client

If you merely use the smbclient command line tool, this is rather simple: It has a -p option for specifying the port:

$ smbclient -p 1445 //localhost/private
Enter martin's password: [enter "foo" here]
Domain=[TESTGROUP] OS=[Unix] Server=[Samba 3.6.6]
smb: \> dir
  .                                   D        0  Wed Oct 17 08:28:23 2012
  ..                                  D        0  Wed Oct 17 08:31:24 2012
  myfile.txt                                   7  Wed Oct 17 08:28:23 2012

In the case of gvfs it wasn’t so simple, however. Surprisingly, libsmbclient does not have an API to set the port, it always assumes 445. smbclient itself uses some internal “libcli” API which does have a way to change the port, but it’s not exposed through libsmbclient. However, Alex and I found some mailing list posts ([1], [2]) that mention $LIBSMB_PROG, and it’s also mentioned in smbclient’s manpage. It doesn’t quite work as advertised in the second ML post (you can’t set it to smbd, smbd apparently doesn’t speak the socket protocol over stdin/stdout), and it’s not being used anywhere in the current Samba sources, but what does work is to use good old netcat:

export LIBSMB_PROG="nc localhost 1445"

with that, you can use smbclient or any program using libsmbclient to talk to our test smb server running as user.

Read more

I just released PyGObject 3.4.1, in time for the GNOME 3.6.1 release on Wednesday.

This version provides a nice set of bug fixes. no API changes.

Thanks to all contributors!

Complete list of changes:

  • Skip Regress tests with –disable-cairo (Martin Pitt) (#685094)
  • _pygi_marshal_from_py_uint64: Re-fix check of negative values (Martin Pitt) (#685000)
  • Fix leak with python callables as closure argument. (Simon Feltman) (#685598)
  • Gio overrides: Handle setting GSettings enum keys (Martin Pitt) (#685947)
  • tests: Check reading GSettings enums in Gio overrides (Martin Pitt)
  • Fix unsigned values in GArray/GList/GSList/GHash (Martin Pitt) (#685860)
  • build: Fix srcdir != builddir (Colin Walters)
  • _pygi_marshal_from_py_uint64(): Use correct data type in py2.7 check (Alban Browaeys) (#685000)
  • Install an .egg-info file (Johan Dahlin) (#680138)
  • PyGProps_getattro(): Fix GObjectClass leak (Johan Dahlin) (#685218)
  • pygobject.c: Don’t leak GObjectClass reference (Olivier Crête) (#684062)
  • Fix memory leak in _pygi_argument_to_array() (Alban Browaeys) (#685082)
  • Fix error messages for out of range numbers (Martin Pitt) (#684314)
  • Kill dbus-daemon after running tests (Martin Pitt) (#685009)
  • GVariant overrides: Support empty tuple arrays (Martin Pitt) (#684928)
  • TestGVariant: Split creation test case into several smaller ones (Martin Pitt)
  • Fix unused variables and results (Martin Pitt)
  • tests: Fix wrong return type in test_int64_callback() (Martin Pitt) (#684700)
  • Fix GValue marshalling of long and unsigned long (Giovanni Campagna) (#684331)
  • Clean up deprecation message for assigning gpointers to objects. (Simon Feltman) (#683599)
  • pygi-property: Lookup property in base classes of non-introspected types (Olivier Crête) (#684058)

Read more

I found it surprisingly hard to determine in tearDown() whether or not the test that currently ran succeeded or not. I am writing some tests for gnome-settings-daemon and want to show the log output of the daemon if a test failed.

I now cobbled together the following hack, but I wonder if there’s a more elegant way? The interwebs don’t seem to have a good solution for this either.

    def tearDown(self):
        # collect log, run() shows it on failures
        with open( as f:
            self.log_output =

    def run(self, result=None):
        '''Show log output on failed tests'''

        if result:
            orig_err_fail = result.errors + result.failures
        if result and result.errors + result.failures > orig_err_fail:
            print('\n----- daemon log -----\n%s\n------\n' % self.log_output)

Read more

I was working on writing tests for gnome-settings-daemon a week or so ago, and finally got blocked on being unable to set up upower/ConsoleKit/etc. the way I need them. Also, doing so needs root privileges, I don’t want my test suite to actually suspend my machine, and using the real service is generally not suitable for test suites that are supposed to run during “make check”, in jhbuild, and the like — these do not have the polkit privileges to do all that, and may not even have a system D-Bus running in the first place.

So I wrote a little helper, then realized that I need another one for systemd/ConsoleKit (for the “system idle” property), also looked at the mock polkit in udisks and finally sat down for two days to generalize this and do this properly.

The result is python-dbusmock, I just released the first tarball. With this you can easily create mock objects on D-Bus from any programming language with a D-Bus binding, or even from the shell.

The mock objects look like the real API (or at least the parts that you actually need), but they do not actually do anything (or only some action that you specify yourself). You can configure their state, behaviour and responses as you like in your test, without making any assumptions about the real system status.

When using a local system/session bus, you can do unit or integration testing without needing root privileges or disturbing a running system. The Python API offers some convenience functions like “start_session_bus()“ and “start_system_bus()“ for this, in a “DBusTestCase“ class (subclass of the standard “unittest.TestCase“).

Surprisingly I found very little precedence here. There is a Perl module, but that’s not particuarly helpful for test suites in C/Vala/Python. And there is Phil’s excellent Bendy Bus, but this has a different goal: If you want to thoroughly test a particular D-BUS service, such as ensuring that it does the right thing, doesn’t crash on bad input, etc., then Bendy Bus is for you (and python-dbusmock isn’t). However, it is too much overhead and rather inconvenient if you want to test a client-side program and just need a few system services around it which you want to set up in different states for each test.

You can use python-dbusmock with any programming language, as you can run the mocker as a normal program. The actual setup of the mock (adding objects, methods, properties, etc.) all happen via D-Bus methods on the “org.freedesktop.DBus.Mock“ interface. You just don’t have the convenience D-Bus launch API.

The simplest possible example is to create a mock upower with a single Suspend() method, which you can set up like this from Python:

import dbus
import dbusmock

class TestMyProgram(dbusmock.DBusTestCase):
    def setUp(self):
        self.p_mock = self.spawn_server('org.freedesktop.UPower',

        # Get a proxy for the UPower object's Mock interface
        self.dbus_upower_mock = dbus.Interface(self.dbus_con.get_object(
            'org.freedesktop.UPower', '/org/freedesktop/UPower'),

        self.dbus_upower_mock.AddMethod('', 'Suspend', '', '', '')


    def test_suspend_on_idle(self):
        # run your program in a way that should trigger one suspend call

        # now check the log that we got one Suspend() call
        self.assertRegex(self.p_mock.stdout.readline(), b'^[0-9.]+ Suspend$')

This doesn’t depend on Python, you can just as well run the mocker like this:

python3 -m dbusmock org.freedesktop.UPower /org/freedesktop/UPower org.freedesktop.UPower

and then set up the mocks through D-Bus like

gdbus call --system -d org.freedesktop.UPower -o /org/freedesktop/UPower \
      -m org.freedesktop.DBus.Mock.AddMethod '' Suspend '' '' ''

If you use it with Python, you get access to the dbusmock.DBusTestCase class which provides some convenience functions to set up and tear down local private session and system buses. If you use it from another language, you have to call dbus-launch yourself.

Please see the README for some more details, pointers to documentation and examples.

Update: You can now install this via pip from PyPI or from the daily builds PPA.

Update 2: Adjusted blog entry for version 0.0.3 API, to avoid spreading now false information too far.

Read more

I just released PyGObject 3.3.92, for GNOME 3.5.92.

There is nothing too exciting in this release; a couple of small bug fixes and a lot of new test cases. See the detailled list of changes below.

Thanks to all contributors!


  • release-news: Generate HTML changelog (Martin Pitt)
  • [API add] Add ObjectInfo.get_abstract method (Simon Feltman) (#675581)
  • Add deprecation warning when setting gpointers to anything other than int. (Simon Feltman) (#683599)
  • test_properties: Test accessing a property from a superclass (Martin Pitt) (#684058)
  • Consistent test names (Martin Pitt)
  • test_everything: Ensure TestSignals callback does get called (Martin Pitt)
  • argument: Fix 64bit integer convertion from GValue (Nicolas Dufresne) (#683596)
  • Add Simon Feltman as a project maintainer (Martin Pitt)
  • Drop global type variables (Martin Pitt)
  • Consistent test names (Martin Pitt)
  • Add test cases for GValue signal arguments (Martin Pitt) (#683775)
  • Add test for GValue signal return values (Martin Pitt) (#683596)
  • Improve setting pointer fields/arguments to NULL using None (Simon Feltman) (#683150)
  • Test gint64 C signal arguments and return values (Martin Pitt)
  • Test in/out int64 GValue method arguments. (Martin Pitt) (#683596)
  • Bump g-i dependency to 1.33.10 (Martin Pitt)
  • Fix file (Thibault Saunier) (#683379)

Read more

I just released PyGObject 3.3.91, for GNOME 3.5.91.

The big new feature in this release (thanks to the release team for granting an exception) is Simon Feltman’s new Signal helper class, which makes defining custom signals a whole lot simpler and more obvious. In the past, you had to do

 class C(GObject.GObject):
    __gsignals__ = {
        'my_signal': (GObject.SIGNAL_RUN_FIRST, GObject.TYPE_NONE,

    def do_my_signal(self, arg):
        print("my_signal called with %i" % arg)

whereas now this looks like

class C(GObject.GObject):
    def my_signal(self, arg):
        print("my_signal called with %i" % arg)

or even more elegantly when using Python 3 and its new type annotations:

class C(GObject.GObject):
    def my_signal(self, arg:int):
        print("my_signal called with %i" % arg)

Check out the updated example and docstring for other ways how to use it.

Overrides can now be in a directory different from the one that pygobject installs itself into. These overrides need to put this into their at the top:

from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

and put themselves somewhere into the default PYTHONPATH. This should make it a lot easier for library packages to ship their own overrides for Python.

This new version also comes with a couple of new overrides and bug fixes. See the detailled list of changes below.

Thanks to all contributors!

  • Fix exception test case for Python 2 (Martin Pitt)
  • Bump g-i dependency to >= 1.3.9 (Martin Pitt)
  • Show proper exception when trying to allocate a disguised struct (Martin Pitt) (#639972)
  • Support marshalling GParamSpec signal arguments (Mark Nauwelaerts) (#683099)
  • Add test for a signal that returns a GParamSpec (Martin Pitt) (#683265)
  • [API add] Add Signal class for adding and connecting custom signals. (Simon Feltman) (#434924)
  • Fix pygtkcompat’s Gtk.TreeView.insert_column_with_attributes() (Martin Pitt)
  • Add override for Gtk.TreeView.insert_column_with_attributes() (Marta Maria Casetti) (#679415)
  • .gitignore: Add missing built files (Martin Pitt)
  • Ship tests/gi in tarball (Martin Pitt)
  • Split (Martin Pitt) (#683188)
  • _pygi_argument_to_object(): Clean up array unmarshalling (Martin Pitt)
  • Fix memory leak in _pygi_argument_to_object() (Alban Browaeys) (#682979)
  • Fix setting pointer fields/arguments to NULL using None. (Simon Feltman) (#683150)
  • Fix for python 2.6, officially drop support for < 2.6 (Martin Pitt) (#682422)
  • Allow overrides in other directories than gi itself (Thibault Saunier) (#680913)
  • Clean up sys.path handling in tests (Simon Feltman) (#680913)
  • Fix dynamic creation of enum and flag gi types for Python 3.3 (Simon Feltman) (#682323)
  • [API add] Override g_menu_item_set_attribute (Paolo Borelli) (#682436)

Read more

I just released PyGObject 3.3.90, for GNOME 3.5.90.

This is now working correctly on big-endian 64 bit machines such as powerpc64, and fixes marshalling for GParamSpec attributes and return values, as well as a few small bug fixes.

Thanks to all contributors!

Complete list of changes:

  • Implement marshalling for GParamSpec (Mathieu Duponchelle) (#681565)
  • Fix erronous import statements for Python 3.3 (Simon Feltman) (#682051)
  • Do not fail tests if pyflakes or pep8 are not installed (Martin Pitt)
  • Fix PEP-8 whitespace checking and issues in the code (Martin Pitt)
  • Fix unmarshalling of gssize (David Malcolm) (#680693)
  • Fix various endianess errors (David Malcolm) (#680692)
  • Gtk overrides: Add TreeModelSort.__init__(self, model) (Simon Feltman) (#681477)
  • Convert Gtk.CellRendererState in the pygi-convert script (Manuel Quiñones) (#681596)

Read more

I started to collect some easy PyGObject bugs which are appropriate for the PyGObject hackfest at GUADEC on July 30th. These are bugs which do not need a lot of previous knowlege and are excellent starters for new contributors, such as adding overrides, fixing build system issues, etc.

I also created an initial idea pool/agenda/coordination page, where participants can add or signup for things to work on.

Feel free to add your own topics! I’m really looking forward to GUADEC and the hackfest, see you there!


Read more

I released PyGObject 3.3.4. This is mostly a bug fix only release to fix existing API. Highlights are that lists of GVariants and other corner cases are now working correctly when being passed from C to Python, and that calling help() on a GI module now does something sensible.

Thanks to all contributors!

Complete list of changes:

  • Drop bogus filter_new() conversion (Martin Pitt) (#679999)
  • Fix help() for GI modules (Martin Pitt) (#679804)
  • Skip gi.CallbackInfo objects from a module’s dir() (Martin Pitt) (#679804)
  • Fix __path__ module attribute (Martin Pitt)
  • Fix some child ? getChild() false positives (Joe R. Nassimian) (#680004)
  • Fix array handling for interfaces, properties, and signals (Mikkel Kamstrup Erlandsen) (#667244)
  • Add conversion of the Gdk.PropMode constants to script (Manuel Quiñones) (#679775)
  • Add the same rules for pack_start to convert pack_end (Manuel Quiñones) (#679760)
  • Add error-checking for the case where _arg_cache_new() fails (Dave Malcolm) (#678914)
  • Add conversion of the Gdk.NotifyType constants to script (Manuel Quiñones) (#679754)
  • Fix PyObject_Repr and PyObject_Str reference leaks (Simon Feltman) (#675857)
  • [API add] Gtk overrides: Add TreePath.__len__() (Martin Pitt) (#679199)
  • GLib.Variant: Fix repr(), add proper str() (Martin Pitt) (#679336)
  • m4/python.m4: Update Python version list (Martin Pitt)
  • Remove “label” property from Gtk.MenuItem if it is not set (Micah Carrick) (#670575)

Read more

I just received confirmation that my request for a PyGObject hackfest has been approved by the GUADEC organizers.

If you are developing GObject-introspection based Python applications and have some problems with PyGObject, this is the time and place to get to know each other, getting bugs fixed, learn about pygobject’s innards, or update libraries to become introspectable. I will prepare a list of easy things to look into if you are interested in learning about and getting involved in PyGObject’s development.

See you on July 30th in A Coruña!


Read more

I released PyGObject 3.3.3.

The most notable changes are that you can now access methods (and other identifiers) which are Python keywords, PyGObject automatically escapes them now by appending a ‘_’. For example, you can now call myGdkWindow.raise_() or GLib.Thread.yield_() instead of having to resort to the previous workaround getattr(myGdkWindow, 'raise')().

This version also restores the deprecated get_data() and set_data() methods. They were never really meant to be used from Python programs, they can potentially mess up your program and cause crashes, and do not give you anything that regular Python object properties would not already provide in a much safer way (i. e. just write = 'bar' instead of my_obj.set_data('foo', 'bar')). Apparently some software projects are using them, so they will now raise a deprecation warning and be removed for the GNOME 3.8 cycle instead.

Thanks to all contributors!

Complete list of changes:

  • Remove obsolete release-tag make target (Martin Pitt)
  • Do not do any python calls when GObjects are destroyed after the python interpreter has been finalized (Simon Schampijer) (#678046)
  • Do not change constructor-only “type” Window property (Martin Pitt) (#678510)
  • Escape identifiers which are Python keywords (Martin Pitt) (#676746)
  • Fix code for PEP-8 violations detected by the latest pep8 checker. (Martin Pitt)
  • Fix crash in GLib.find_program_in_path() (Martin Pitt) (#678119)
  • Revert “Do not bind gobject_get_data() and gobject_set_data()” (Martin Pitt) (#641944)
  • GVariant: Raise proper TypeError on invalid tuple input (David Keijser) (#678317)

Update:Just released to fix a regresssion from the keyword escaping patch. It also escaped enum and flags names, but as they are translated to upper case they are never keywords.

Read more

I just released PyGObject 3.3.2, (almost) in time for tomorrow’s GNOME 3.5.2 release. No API breaks or new features this time, just lots of bug fixes and some minor API completions. My personal favorite is making closure calls work with GVariant arguments, which I finally figured out after over half a year; this finally unblocks making GDBus fully introspectable with not too much additional work, only that in the meantime dbus-python was ported to Python 3 so that the need for it is actually a lot smaller now.

Thanks to all contributors!

Complete list of changes:

  • foreign: Register cairo.Path and cairo.FontOptions foreign structs (Bastian Winkler) (#677388)
  • Check types in GBoxed assignments (Marien Zwart) (#676603)
  • [API add] Gtk overrides: Add TreeModelRow.get_previous() (Bastian Winkler) (#677389)
  • [API add] Add missing GObject.TYPE_VARIANT (Bastian Winkler) (#677387)
  • Fix boxed type equality (Jasper St. Pierre) (#677249)
  • Fix TestProperties.testBoxed test (Jose Rostagno) (#676644)
  • Fix handling of by-reference structs as out parameters (Carlos Garnacho) (#653151)
  • tests: Add more vfunc checks for GIMarshallingTestsObject (Martin Pitt)
  • Test caller-allocated GValue out parameter (Martin Pitt) (#653151)
  • GObject.bind_property: Support transform functions (Bastian Winkler) (#676169)
  • Fix lookup of vfuncs in parent classes (Carlos Garnacho) (#672864)
  • tests/ Fix whitespace (Martin Pitt)
  • gi: Support zero-terminated arrays with length arguments (Jasper St. Pierre) (#677124)
  • [API add] Add GObject.bind_property method (Simon Feltman) (#675582)
  • pygtkcompat: Correctly set flags (Jose Rostagno) (#675911)
  • Gtk overrides: Implement __delitem__ on TreeModel (Jose Rostagno) (#675892)
  • Gdk Color override should support red/green/blue_float properties (Simon Feltman) (#675579)
  • Support marshalling of GVariants for closures (Martin Pitt) (#656554)
  • _pygi_argument_from_object(): Check for compatible data type (Martin Pitt)
  • pygtkcompat: Fix color conversion (Martin Pitt)
  • test_gi: Check setting properties in constructor (Martin Pitt)
  • Support getting and setting GStrv properties (Martin Pitt)
  • Support defining GStrv properties from Python (Martin Pitt)
  • Add GObject.TYPE_STRV constant (Martin Pitt)
  • Unref GVariants when destroying the wrapper (Martin Pitt) (#675472)
  • Fix TestArrayGVariant test cases (Martin Pitt)
  • pygtkcompat: Add gdk.pixbuf_get_formats compat code (Jose Rostagno) (#675489)
  • pygtkcompat: Add some more compat functions (Jose Rostagno) (#675489)
  • Fix tests for Python 3 (Martin Pitt)
  • Fix building with –disable-cairo (Martin Pitt)
  • tests: Fix deprecated assertions (Martin Pitt)
  • Run tests with MALLOC_PERTURB_ (Martin Pitt)

Read more

As I wrote two weeks ago, I consider the QA related changes in Ubuntu 12.04 a great success. But while we will continue and even extend our efforts there, this is not where the ball stops: it’s great to have the feedback cycle between “break it” and “notice the bug” reduced from potentially a few months to one day in many cases, but wouldn’t it be cool to reduce that to a few minutes, and also put the machinery right at where stuff really happens — into the upstream trunks? If for every commit to PyGObject, GTK, NetworkManager, udisks, D-BUS, telepathy, gvfs, etc. we’d immediately build and test all reverse dependencies and the committer would be told about regressions?

I have had the desire to work on automated tests in Linux Plumbing and GNOME for quite a while now. Also, after 8 years of doing distribution work of packaging and processes (tech lead, release engineering/management, stable release updates, etc.) I wanted to shift my focus towards technology development. So I’ve been looking for a new role for some time now.

It seems that time is finally there: At the recent UDS, Mark announced that we will extend our QA efforts to upstream. I am very happy that in two weeks I can now move into a role to make this happen: Developing technology to make testing easier, work with our key upstreams to set up test suites and reporting, and I also can do some general development in areas that are near and dear to my heart, like udev/systemd, udisks, pygobject, etc. This work will be following the upstream conventions for infrastructure and development policies. In particular, it is not bound by Canonical copyright license agreements.

I have a bunch of random ideas what to work on, such as:

  • Making it possible/easier to write tests with fake hardware. E. g. in the upower integration tests that I wrote a while ago there is some code to create a fake sysfs tree which should really go into udev itself, available from C and introspection and be greatly extended. Also, it’s currently not possible to simulate a uevent that way, that’s something I’d like to add. Right now you can only set up /sys, start your daemon, and check the state after the coldplugging phase.
  • Interview some GNOME developers what kind of bugs/regressions/code they have most trouble with and what/how they would like to test. Then write a test suite with a few working and one non-working case (bugzilla should help with finding these), discuss the structure with the maintainer again, find some ways to make the tests radically simpler by adding/enhancing the API available from gudev/glib/gtk, etc. E. g. in the tests for apport-gtk I noticed that while it’s possible to do automatic testing of GUI applications it is still way harder than it should and needs to be. I guess that’s the main reason why there are hardly any GUI tests in GNOME?
  • I’ve heard from several people that it would be nice to be able to generate some mock wifi/ethernet/modem adapters to be able to automatically test NetworkManager and the like. As network devices are handled specially in Linux, not in the usual /dev and sysfs manner, they are not easy to fake. It probably needs a kernel module similar to scsi_debug, which fakes enough of the properties and behaviour of particular nmetwork card devices to be useful for testing. One could certainly provide a pipe or a regular bridge device at the other end to actually talk to the application through the fake device. (NB this is just an idea, I haven’t looked into details at all yet).
  • For some GUI tests it would be much easier if there was a very simple way of providing mocks/stubs for D-BUS services like udisks or NetworkManager than having to set up the actual daemons, coerce them into some corner-case behaviour, and needing root privileges for the test due to that. There seems to be some prior art in Ruby, but I’d really like to see this in D-BUS itself (perhaps a new library there?), and/or having this in GDBus where it would even be useful for Python or JavaScript tests through gobject-introspection.
  • There are nice tools like the Clang static code analyzer out there. I’d like to play with those and see how we can use it without generating a lot of false positives.
  • Robustify jhbuild to be able to keep up with building everything largely unattended. Right now you need to blow away and rebuild your tree way too often, due to brittle autotools or undeclared dependencies, and if we want to run this automatically in Jenkins it needs to be able to recover by itself. It should be able to keep up with the daily development, automatically starting build/test jobs for all reverse dependencies for a module that just has changed (and for basic libraries like GLib or GTK that’s going to be a lot), and perhaps send out mail notifications when a commit breaks something else. This also needs some discussion first, about how/where to do the notifications, etc.

Other ideas will emerge, and I hope lots of you have their own ideas what we can do. So please talk to me! We’ll also look for a second person to work in that area, so that we have some capacity and also the possibility to bounce ideas and code reviews between each other.

Read more

Part of our efforts to reduce power consumption in Ubuntu is to provide an easy tool to hunt down which programs and devices are to blame for inordinate power consumption. powertop’s interactive mode is pretty good for this if you are sitting in a train and want to tweak some knobs to max out battery life, but we need something more reproducible and noninteractive for developers who want to file proper bug reports.

So I wrote a little script power-usage-report which calls fatrace for measuring file access activity from programs, and powertop-1.13 to measure process and device wakeups, clean up and sort their ouput, and generate a report which is appropriate to attach to bug reports, send around, put into Jenkins for measuring daily progress, etc. It is now part of fatrace version 0.4, so today’s Precise upgrades will have it.

The output has several sections for disk access (which prevent the disk from spinning down), wakeups (causing CPU power usage), and device activity. Disk/wakeups are sorted in descending order by process:

$ sudo power-usage-report
Measurement will begin in 5 seconds. Please make sure that the
computer is idle, i. e. do not press keys, start or operate programs, and that
programs are not busy with active tasks other than the one you want to examine.
Starting measurement for 60 seconds...
Measurement complete. Generating report...
======= unity-panel-ser: 5 file access events ======
/usr/share/zoneinfo/UTC: 1 reads
/usr/share/zoneinfo/posix/Europe/Berlin: 1 reads
/etc/localtime: 3 reads

======= gnome-settings-: 1 file access events ======
/etc/fstab: 1 reads

======= telepathy-gabbl: 1 file access events ======
/home/martin/.cache/wocky/caps/caps-cache.db: 1 reads

====== Wakeups ======
  30,9% ( 52,0)   compiz
  16,3% ( 27,4)   [iwlwifi] 
  12,5% ( 21,0)   [i915] 
   3,7% (  6,3)   [ahci] 
   2,3% (  3,9)   swapper/3
   1,2% (  2,0)   gvfs-afc-volume

====== Devices ======
An audio device is active 100,0% of the time:
hwC0D0 Conexant CX20585 

Recent USB suspend statistics
Active  Device name
100,0%	USB device 1- : USB Mouse (A4Tech)
100,0%	/sys/bus/usb/devices/1-
100,0%	USB device 1-1.5.4 : Kinesis Keyboard Hub (PI Engineering)
  0,0%	USB device 1-1.5.2 : USB2.0 Hub Controller (NEC Corporation)


You can redirect output to a file, of course. The top header (“Starting measurement..” etc.) will go to stderr and thus not be part of the redirected output.

Read more

Part of our efforts to reduce power consumption is to identify processes which keep waking up the disk even when the computer is idle. This already resulted in a few bug reports (and some fixes, too), but we only really just began with this.

Unfortunately there is no really good tool to trace file access events system-wide. powertop claims to, but its output is both very incomplete, and also wrong (e. g. it claims that read accesses are writes). strace gives you everything you do and don’t want to know about what’s going on, but is per-process, and attaching strace to all running and new processes is cumbersome. blktrace is system-wide, but operates at a way too low level for this task: its output has nothing to do any more with files or even inodes, just raw block numbers which are impossible to convert back to an inode and file path.

So I created a little tool called fatrace (“file access trace”, not “fat race” :-) ) which uses fanotify, a couple of /proc lookups and some glue to provide this. By default it monitors the whole system, i. e. all mounts (except the virtual ones like /proc, tmpfs, etc.), but you can also tell it to just consider the mount of the current directory. You can write the log into a file (stdout by default), and run it for a specified number of seconds. Optional time stamps and PID filters are also provided.

$ sudo fatrace
rsyslogd(967): W /var/log/auth.log
notify-osd(2264): O /usr/share/pixmaps/weechat.xpm
compiz(2001): R device 8:2 inode 658203

It shows the process name and pid, the event type (Rread, Write, Open, or Close), and the path. Sometimes its’ not possible to determine a path (usually because it’s a temporary file which already got deleted, and I suspect mmaps as well), in that case it shows the device and inode number; such programs then need closer inspection with strace.

If you run this in gnome-terminal, there is an annoying feedback loop, as gnome-terminal causes a disk access with each output line, which then causes another output line, ad infinitum. To fix this, you can either redirect output to a file (-o /tmp/trace) or ignore the PID of gnome-terminal (-p `pidof gnome-terminal`).

So to investigate which programs are keeping your disk spinning, run something like

  $ sudo fatrace -o /tmp/trace -s 60

and then do nothing until it finishes.

My next task will be to write an integration program which calls fatrace and powertop, and creates a nice little report out of that raw data, sorted by number of accesses and process name, and all that. But it might already help some folks as it is right now.

The code lives in bzr branch lp:fatrace (web view), you can just run make and sudo ./fatrace. I also uploaded a package to Ubuntu Precise, but it still needs to go through the NEW queue. I also made a 0.1 release, so you can just grab the release tarball if you prefer. Have a look at the manpage and --help, it should be pretty self-explanatory.

Read more

PackageKit has a “WhatProvides” API for mapping distribution independent concepts to particular package names. For example, you could ask “which packages provide a decoder for AC3 audio files?

$ pkcon what-provides  "gstreamer0.10(decoder-audio/ac3)"
Installed   	gstreamer0.10-plugins-good-	GStreamer plugins from the "good" set
Available  	gstreamer0.10-plugins-ugly-0.10.18-3ubuntu4.amd64	GStreamer plugins from the "ugly" set

This is the kind of question your video player would ask the system if it encounters a video it cannot play. In reality they of course use the D-BUS or the library API, but it’s easier to demonstrate with the PackageKit command line client.

PackageKit provides a fair number of those concepts; I recently added LANGUAGE_SUPPORT for packages which provide dictionaries, spell checkers, and other language support for a given language or locale code.

However, PackageKit’s apt backend does not actually implement a lot of these (only CODEC and MODALIAS), and aptdaemons’s PackageKit compatibility API does not implement any. That might be because their upstreams do not know enough how to do the mapping for a particular distro/backend, because doing so involves distro specific code which should not go into upstreams, or simply because of the usual chicken-egg problem of app developers rather doing their own thing instead of using generic APIs.

So this got discussed between Sebastian Heinlein and me, and voila, there it is: it is now very easy to provide Python plugins for “what-provides” to implement any of the existing types. For example, language-selector now ships a plugin which implements LANGUAGE_SUPPORT, so you can ask “which packages do I need for Chinese in China” (i. e. simplified Chinese)?

$ pkcon what-provides "locale(zh_CN)"
Available   	firefox-locale-zh-hans-10.0+build1-0ubuntu1.all	Simplified Chinese language pack for Firefox
Available   	ibus-sunpinyin-2.0.3-2.amd64            	sunpinyin engine for ibus
Available   	language-pack-gnome-zh-hans-1:12.04+20120130.all	GNOME translation updates for language Simplified Chinese
Available   	ttf-arphic-ukai-0.2.20080216.1-1.all    	"AR PL UKai" Chinese Unicode TrueType font collection Kaiti style

Rodrigo Moya is currently working on implementing the control-center region panel redesign in a branch. This uses exactly this feature.

In Ubuntu we usually do not use PackageKit itself, but aptdaemon and its PackageKit API compatibility shim python-aptdaemon.pkcompat. So I ported that plugin support for aptdaemon-pkcompat as well, so plugins work with either now. Ubuntu Precise got the new aptdaemon (0.43+bzr769-0ubuntu1) and language-selector (0.63) versions today, so you can start playing around with this now.

So how can you write your own plugins? This is a trivial, although rather nonsense example:

from packagekit import enums

def my_what_provides(apt_cache, provides_type, search):
    if provides_type in (enums.PROVIDES_CODEC, enums.PROVIDES_ANY):
        return [apt_cache["gstreamer-moo"]]
        raise NotImplementedError('cannot handle type ' + str(provides_type))

The function gets an apt.Cache object, one of enums.PROVIDES_* and the actual search type as described in the documentation (above dummy example does not actually use it). It then decides whether it can handle the request and return a list of apt.package.Package objects (i. e. values in an apt.Cache map), or raise a NotImplementedError otherwise.

You register the plugin through Python pkg-resources in your (this needs setuptools):



You can register arbitrarily many plugins, they will be all called and their resulting package lists joined.

All this will hopefully help a bit to push distro specifics to the lowest possible levels, and use upstream friendly and distribution agnostic APIs in your applications.

Read more

On my 8 hour train ride to Budapest last Sunday I finally worked on making libxklavier introspectable. Thanks to Sergey’s fast review the code now landed in trunk. I sent a couple of refinements to the bug report still, but those are mostly just icing on the cake, the main functionality of getting and setting keyboard layouts is working nicely now (see the example script).

Read more

Just took the plunge, using the excellent bandwidth and local mirror at UDS:

$ lsb_release -irc
Distributor ID: Ubuntu
Release: 12.04
Codename: precise

Nothing blew up in my face, so it seems today is a good day to die^Wupgrade.

Read more