Canonical Voices

Posts tagged with 'gnome'

pitti

Part of our efforts to reduce power consumption is to identify processes which keep waking up the disk even when the computer is idle. This already resulted in a few bug reports (and some fixes, too), but we only really just began with this.

Unfortunately there is no really good tool to trace file access events system-wide. powertop claims to, but its output is both very incomplete, and also wrong (e. g. it claims that read accesses are writes). strace gives you everything you do and don’t want to know about what’s going on, but is per-process, and attaching strace to all running and new processes is cumbersome. blktrace is system-wide, but operates at a way too low level for this task: its output has nothing to do any more with files or even inodes, just raw block numbers which are impossible to convert back to an inode and file path.

So I created a little tool called fatrace (“file access trace”, not “fat race” :-) ) which uses fanotify, a couple of /proc lookups and some glue to provide this. By default it monitors the whole system, i. e. all mounts (except the virtual ones like /proc, tmpfs, etc.), but you can also tell it to just consider the mount of the current directory. You can write the log into a file (stdout by default), and run it for a specified number of seconds. Optional time stamps and PID filters are also provided.

$ sudo fatrace
rsyslogd(967): W /var/log/auth.log
notify-osd(2264): O /usr/share/pixmaps/weechat.xpm
compiz(2001): R device 8:2 inode 658203
[...]

It shows the process name and pid, the event type (Rread, Write, Open, or Close), and the path. Sometimes its’ not possible to determine a path (usually because it’s a temporary file which already got deleted, and I suspect mmaps as well), in that case it shows the device and inode number; such programs then need closer inspection with strace.

If you run this in gnome-terminal, there is an annoying feedback loop, as gnome-terminal causes a disk access with each output line, which then causes another output line, ad infinitum. To fix this, you can either redirect output to a file (-o /tmp/trace) or ignore the PID of gnome-terminal (-p `pidof gnome-terminal`).

So to investigate which programs are keeping your disk spinning, run something like

  $ sudo fatrace -o /tmp/trace -s 60

and then do nothing until it finishes.

My next task will be to write an integration program which calls fatrace and powertop, and creates a nice little report out of that raw data, sorted by number of accesses and process name, and all that. But it might already help some folks as it is right now.

The code lives in bzr branch lp:fatrace (web view), you can just run make and sudo ./fatrace. I also uploaded a package to Ubuntu Precise, but it still needs to go through the NEW queue. I also made a 0.1 release, so you can just grab the release tarball if you prefer. Have a look at the manpage and --help, it should be pretty self-explanatory.

Read more
pitti

PackageKit has a “WhatProvides” API for mapping distribution independent concepts to particular package names. For example, you could ask “which packages provide a decoder for AC3 audio files?

$ pkcon what-provides  "gstreamer0.10(decoder-audio/ac3)"
[...]
Installed   	gstreamer0.10-plugins-good-0.10.30.2-2ubuntu2.amd64	GStreamer plugins from the "good" set
Available  	gstreamer0.10-plugins-ugly-0.10.18-3ubuntu4.amd64	GStreamer plugins from the "ugly" set

This is the kind of question your video player would ask the system if it encounters a video it cannot play. In reality they of course use the D-BUS or the library API, but it’s easier to demonstrate with the PackageKit command line client.

PackageKit provides a fair number of those concepts; I recently added LANGUAGE_SUPPORT for packages which provide dictionaries, spell checkers, and other language support for a given language or locale code.

However, PackageKit’s apt backend does not actually implement a lot of these (only CODEC and MODALIAS), and aptdaemons’s PackageKit compatibility API does not implement any. That might be because their upstreams do not know enough how to do the mapping for a particular distro/backend, because doing so involves distro specific code which should not go into upstreams, or simply because of the usual chicken-egg problem of app developers rather doing their own thing instead of using generic APIs.

So this got discussed between Sebastian Heinlein and me, and voila, there it is: it is now very easy to provide Python plugins for “what-provides” to implement any of the existing types. For example, language-selector now ships a plugin which implements LANGUAGE_SUPPORT, so you can ask “which packages do I need for Chinese in China” (i. e. simplified Chinese)?

$ pkcon what-provides "locale(zh_CN)"
[...]
Available   	firefox-locale-zh-hans-10.0+build1-0ubuntu1.all	Simplified Chinese language pack for Firefox
Available   	ibus-sunpinyin-2.0.3-2.amd64            	sunpinyin engine for ibus
Available   	language-pack-gnome-zh-hans-1:12.04+20120130.all	GNOME translation updates for language Simplified Chinese
Available   	ttf-arphic-ukai-0.2.20080216.1-1.all    	"AR PL UKai" Chinese Unicode TrueType font collection Kaiti style
[...]

Rodrigo Moya is currently working on implementing the control-center region panel redesign in a branch. This uses exactly this feature.

In Ubuntu we usually do not use PackageKit itself, but aptdaemon and its PackageKit API compatibility shim python-aptdaemon.pkcompat. So I ported that plugin support for aptdaemon-pkcompat as well, so plugins work with either now. Ubuntu Precise got the new aptdaemon (0.43+bzr769-0ubuntu1) and language-selector (0.63) versions today, so you can start playing around with this now.

So how can you write your own plugins? This is a trivial, although rather nonsense example:

from packagekit import enums

def my_what_provides(apt_cache, provides_type, search):
    if provides_type in (enums.PROVIDES_CODEC, enums.PROVIDES_ANY):
        return [apt_cache["gstreamer-moo"]]
    else:
        raise NotImplementedError('cannot handle type ' + str(provides_type))

The function gets an apt.Cache object, one of enums.PROVIDES_* and the actual search type as described in the documentation (above dummy example does not actually use it). It then decides whether it can handle the request and return a list of apt.package.Package objects (i. e. values in an apt.Cache map), or raise a NotImplementedError otherwise.

You register the plugin through Python pkg-resources in your setup.py (this needs setuptools):

   setup(
       [....]

       entry_points="""[packagekit.apt.plugins]
what_provides=my_plugin_module_name:my_what_provides
""",
       [...])

You can register arbitrarily many plugins, they will be all called and their resulting package lists joined.

All this will hopefully help a bit to push distro specifics to the lowest possible levels, and use upstream friendly and distribution agnostic APIs in your applications.

Read more
pitti

On my 8 hour train ride to Budapest last Sunday I finally worked on making libxklavier introspectable. Thanks to Sergey’s fast review the code now landed in trunk. I sent a couple of refinements to the bug report still, but those are mostly just icing on the cake, the main functionality of getting and setting keyboard layouts is working nicely now (see the example script).

Read more

Zeeshan sends along that he’s looking for help bring GNOME Boxes to Ubuntu and Debian.

If you read any of my previous blog entries, you must be now familiar with this ‘express installation’ concept we have in Boxes. Its pretty neat actually, you just set a few options at the beginning and then you can leave Boxes (or your machine) and when you are back, everything is setup for you automatically in a new box.

I have invested a lot of time/efforts on this already and will be spending a lot more time in future as well but I am just one man so can not possibly cover all operating systems out there. That is why I am asking for help from anyone who will be interested in adding express installation support for Ubuntu and Debian while I focus on Fedora and Windows variants. Oh and if you are interested in adding support for some other distribution/OS, that contribution will also be more than welcomed.

In any case, happy hacking!

If you’re interested in doing this (it would be great to get Boxes in 12.04) let me know. You’ll likely need to link up with the Desktop Team, I can help get you talking to the right people if you want to rock this.

Read more
pitti

Apport and the retracer bot in the Canonical data center have provided server-side automatic closing of duplicate crash report bugs for quite a long time. As we have only kept Apport crash detection enabled in the development release, we got away with this as bugs usually did not get so many duplicates that they became unmanageable. Also, the number of duplicates provided a nice hint to how urgent and widespread a crash actually was.

However, it’s time to end that era and provide something better now:

  • This probably caused a lot of frustration when a reporter of the crash spent time, bandwidth, and creativity to upload the crash data and create a description for it, only to find that it got closed as a duplicate 20 minutes later.
  • Some highly visible crashes sometimes generated up to a hundred duplicates in Launchpad, which was prone to timeouts, and needless catch-up by the retracers.
  • We plan to have a real crash database soon, and eventually want to keep Apport enabled in stable releases. This will raise the number of duplicates that we get by several magnitudes.
  • For common crashes we had to write manual bug patterns to avoid getting even more duplicates.

So with the just released Apport 1.90 we introduce client-side duplicate checking. So from now, when you report a crash, you are likely to see “We already know about this” right away, without having to upload or type anything, and you will get directed to the bug page. You should mark yourself as affected and/or subscribe to the bug, both to get a notification when it gets fixed, and also to properly raise the “hotness” of the bug to bubble up to developer attention.

For the technically interested, this is how we detect duplicates for the “signal” crashes like SIGSEGV (as opposed to e. g. Python crashes, where we always have a fully symbolic stack trace):
As we cannot rely on symbolic stack traces, and do not want to force every user to download tons of debug symbols, Apport now falls back to generating a “crash address signature” which combines the absolute addresses of the (non-symbolic) stack trace and the /proc/pid/maps mapping to a stack of libraries and the relative offsets within those, which is stable under ASLR for a given set of dependency versions. As the offsets are specific to the architecture, we form the signature as combination of the executable name, the signal number, the architecture, and the offset list. For example, the i386 signature of bug looks like this:

/usr/bin/rhythmbox:11:i686:/usr/lib/libgstpbutils-0.10.so.0.24.0+c284:/usr/lib/i386-linux-gnu/libgobject-2.0.so.0.3000.0+3337a:/usr/lib/i386-linux-gnu/libgobject-2.0.so.0.3000.0+8e0

As library dependencies can change, we have more than one architecture, and the faulty function can be called from different entry points, there can be many address signatures for a bug, so the database maintains an N:1 mapping. In its current form the signatures are taken as-is, which is much more strict than it needs to be. Once this works in principle, we can refine the matching to also detect duplicates from different entry points by reducing the part that needs to match to the common prefix of several signatures which were proven to be a duplicate by the retracer (which gets a fully symbolic stack trace).

The retracer bots now exports the current duplicate/address signature database to http://people.canonical.com/~ubuntu-archive/apport-duplicates in an indexed text format from where Apport clients can quickly check whether a bug is known.

For the Launchpad crash database implementation we actually check if the bug is readable by the reporter, i. e. it is private and the reporter is in a subscribed team, or the bug is public; if not, we let him report the bug anyway and duplicate it later through the existing server-side retracer, so that the reporter has a chance of getting subscribed to the bug. We also let the bug be filed if the currently existing symbolic stack trace is bad (tagged as apport-failed-retrace) or if a developer wants a new symbolic stack trace with the current libraries (tagged as apport-request-retrace).

As this is a major new feature, I decided that it’s time to call this Apport 2.0. This is the first public beta towards it, thus called 1.90. With Apport’s test driven and agile development the version numbers do not mean much anyway (the retracer bots in the data center always just run trunk, for example), so this is as good time as any to reset the rather large “.26″ minor version that we are at right now.

Read more
pitti

On a rather calm ten-hour flight to Orlando I once again did some pygobject, udisks, and Apport hacking (It’s scary how productive one can be when not constantly being interrupted by IRC, email, etc). One more visible change amongst these was finally fixing a five year old five-digit bug to integrate apport-retrace into the GUI, now that it does not potentially wreck your installation any more.

If the apport-retrace package is installed, the crash detail dialog will show a new “Examine locally” button:

Apport crash detail dialog

After clicking this, you can choose what do do exactly:

Retrace action dialog

I know this dialog is not a beauty, as it’s implemented using the ui_question_choice() API which is used by package hooks. That makes it work for all available UIs (GTK, KDE, CLI), though, and can easily be extended to have more actions. And if you get this far and want to stack traces, you are used to looking at eye-bleeding gibberish anyway..

Presumably the most useful (and default) action is to download all the debug symbols, open a Terminal, and put you into a GDB session with all these, and the core dump loaded, so that you can poke around the crashed program state with all symbols available.

But you can also run gdb without downloading debug symbols, or just update the .crash report file with a fully symbolic stack trace.

This works just as well in apport-cli, but not yet in the KDE version: Someone needs to implement the equivalent of the apport-gtk implementation to apport-kde and kde/bugreport.ui, i. e. show an “Examine locally” button if self.can_examine_locally() is true, and add an appropriate ui_run_terminal() method (which should be fairly similar to the GTK one, just with Qt/KDEish terminal emulators). But as Kubuntu does not currently use Apport (and also because I didn’t have all the dependencies installed on my laptop) I did not yet do this. Please catch me on IRC/mail/merge proposal if you want to work on this. If you look at above commit, the changes to the GtkBuilder file look huge, but that’s only because I haven’t touched it for ages and the current Glade shuffled the elements quite a bit; it just adds the button to the dialog.

For now this is all sitting in trunk, I’ll do a new upstream release and Ubuntu precise upload soon.

Happy debugging!

Read more
Ara

This year, the Desktop Summit happens in Berlin and although I am not that much involved with the desktop that I was previously it is a great opportunity as it happens locally :-)

I will be attending Saturday, Sunday and, thanks to my employer, Canonical, Monday as well, using one of the conference day paid leave that we have as part of the employee benefits. I will also be attending the parties, of course!

Really looking forward to this weekend and to seeing again some friends who are coming to the summit as well.

 

Read more

One of the (great) trends that browsers are doing these days is “getting out of the way”. That is, less “chrome” more space for content. I was curious to how we’ve been improving in this area, so I asked Jason to do some math, and here’s what we came up with.

So, given a desktop that you log in, how many pixels do we consume and how much do we leave for apps? Well, by default here’s how GNOME 2.x, 3.0, and Unity consume your pixels. These are the amount of pixels (broken down by resolution) that these three desktops use:

I measured Unity twice here. By default if there’s nothing in the way, we show you the launcher, if you move a window there or maximize, we get out of the way (the green bar). So, GNOME 2.x takes up a given amount of space no matter what. Unity takes more but gets out of your way once you start using it to about the same level as GNOME 3.0. Notice how both GNOME 3.0 and Unity are already giving the pixels back where they belong, to applications. :)

Next we have how much space we take up when working, for me I maximize my applications.  We maximized the window in GNOME 3.0 by dragging it to the top bar to measure it but didn’t take into account the window decorations and stuff. Still, much better across the board. I only measured Unity once because the launcher in this state goes away.

But wait a minute, doesn’t the application menu belong to the application? Let’s measure how much UI Unity consumes if we give the menu back to the application. So when you maximize an app the only UI Unity uses up is the home button, the window controls, and the indicators. There could still be dead space there in the menu, but that really depends on the length of the menu and per application, and I’m not going to go measure half the archive.

Caveats and Conclusions

a) GNOME 2.x is fat… :)

b) When you use them GNOME 3.0 and Unity are trending towards giving real estate back to applications. (I think this is good)

c) Unity does give the most space back, but remember that’s really all I’m measuring, this doesn’t imply that it’s better (or worse), and it also doesn’t take into account how we actually interact with the desktops, it’s just a raw measurement of pixels. Sorry guys, no flamebait here.

d) We didn’t measure how much space ayatana-scrollbars save you. This would be nice to know.

e) We didn’t take into account overlay-ish things like the dash or the overlay thing that GNOME Shell does. It could very well be that those UI interactions mean that you don’t have to care about those pixels (or care more), but that’s for an expert to figure out, my goal was just to figure out “Is it just me or are desktops following browser chrome trends?

f) We didn’t take into account full screening applications.

Here’s the spreadsheet if you want to mess with it, or add your favorite desktop. (I didn’t measure KDE)

Read more
pitti

As a followup action to my recent Talk about PyGI I now re-used my notes to provide some real wiki documentation.

It would be great if you could add package name info for Fedora/SUSE/etc., and perhaps add more example links for porting different kinds of software! Please also let me know if you have suggestions how to improve the structure of the page.

Read more
pitti

On next Monday this cycle’s Ubuntu Application Developer Week classes will start.

The topic that kept me busy most in this cycle was Python gobject-introspection, and porting pygtk2 apps to PyGI (see my initial steps and my report from the PyGI hackfest.)

To spread the love, there will be two talks about this next week: On Monday 17:00 UTC the very Tomeu Vizoso himself will explain what gobject-introspection (“GI”) is, why we need it, and how library developers use it to ship a good and useful GI binding (“typelib”) for application developers. I will then follow up on Tuesday 16:00 UTC about the app developer side, in particular how to use the GI typelibs in Python, and how to port PyGTK2 applications to PyGI.

For the most part these sessions are distribution neutral (we don’t have any special sauce for this in Debian/Ubuntu, it all happened right upstream :-) ); only a very small fraction of it (where I explain package names, etc.) will be specific to Debian/Ubuntu, but shouldn’t be hard to apply to other distributions as well.

So please feel invited to join, and bombard us with questions!

Read more
Daniel Holbach

I’ve been slacking a bit when it comes to DJing land picked it up this year again. Some of my mixtapes are on a part of my blog that’s not syndicated, but I set up a page with all the posts. I just posted a new one today.

If you have no plans tonight and you’re in Berlin, come to the GNOME3 Launch Lounge (Facebook event page) in c-base tonight. I’ll be playing there as well. :-)

Read more
pitti

GNOME 3.0 and Ubuntu Natty are currently undergoing a major architectural shift from GTK 2.0 to 3.0. Part of this is that the previous set of manually maintained language bindings, such as PyGTK, are being deprecated in favor of GObject Introspection, a really cool technology!

For us this means that we have to port all our PyGTK applications from PyGTK 2 to gobject-introspection and GTK 3.0 at the same time. I started with that for my own projects (Apport and Jockey) a few days ago, and along the way encountered a number of problems. They are being fixed (particular thanks to the quick responsiveness of John Palmieri!), so I guess after a few of those iterations, porting should actually become straight forward and solid.

So now I’m proud to announce Apport 1.16 which is now fully working with GTK 3.0 and pygobject-introspection. I just uploaded it to Ubuntu Natty, where it can get some wider testing. I also have a pygi/GTK3.0 branch for Jockey, which is also mostly working now, but it’s blocked on the availability of a GIR for AppIndicator. Once that lands, I’ll release and upload the Jockey as well.

For other people working on porting, these are the bugs and problems I’ve encountered:

  • [pygobject] GtkMessageDialog constructors did not work at all. This was fixed in pygobject 2.27, so it works fine in Natty, but there is little to no chance of making these work in Ubuntu 10.10.
  • [pygobject] Gtk.init_check() crashes (upstream bug). It’s not strictly necessary, though, just avoids crashes (and crash reports) when calling it without a $DISPLAY. I’ll put that back once that gets fixed.
  • [pygobject] You can’t pass unicode objects as method arguments or property values (upstream bug). Method arguments were recently fixed in upstream git head, and I backported it to the Natty package, so these work now. Property values are still outstanding. The workaround for both is to convert unicode values to bytes with .encode('UTF-8') everywhere.
  • [pygobject] pygi-convert.sh is a great help which automates most of the mechanical rewriting work and thus gets you 90% of the porting done automatically. It’s currently missing MessageDialog constants, and is a bit inconvenient to call. I sent two patches upstream (upstream bug) which improve this.
  • [GIR availability] libnotify did not build a GIR yet (upstream bug). This was fixed in upstream git head with some contributions of mine, and I uploaded it to Natty. This works quite well now, although add_action() crashes on passing the callback. This is still to be investigated.
  • [GIR availability] There is no Application Indicator GIR, as already mentioned. Our DX team and Ken VanDine are working on this, so this should get fixed soon.
  • [GTK] Many dialogs now scale in a very ugly manner: Instead of resizing the contents, the dialog just grows huge outer padding. This is due to a change of the default “fill” property, and apparently is not fully understood yet (upstream bug). As a workaround I explicitly added the fill property to the top-level GTKBox in my GTKBuilder .ui files. (Note that this happens in C as well, it’s not a GI specific bug.)
  • [GTK] Building GtkRadioButton groups is currently a bit inconvenient and requires special-casing for the first group member, due to some missing annotations. I sent a patch upstream which fixes that (upstream bug).
  • [GTK] After initial porting, a lot of my dialogs got way too narrow with wrapped labels and text, because GTK 3 changed the behaviour and handling of widget and window sizes. This is intended, not a bug, but it does require some adaptions in the GtkBuilder files and also in the code. The GTK 2 ? 3 migration guide has a section about it.

Despite those, I’m impressed how well gobject-introspection already works. It was relatively painless for me to generate a GIR from a libnotify (thanks to the annotations already being mostly correct for building the documentation) and use it from Python. In another couple of months this will be rock solid, and porting our bigger pygtk projects like ubiquity and software-center will hopefully become feasible.

Read more
pitti

(Update: Link to Tomeu’s blog post, repost for planet.gnome.org)

Last week I was in Prague to attend the GNOME/Python 2011 Hackfest for gobject-introspection, to which Tomeu Vizoso kindly invited me after I started working with PyGI some months ago. It happened at a place called brmlab which was quite the right environment for a bunch of 9 hackers: Some comfy couches and chairs, soldering irons, lots of old TV tubes, chips, and other electronics, a big Pirate flag, really good Wifi, plenty of Club Mate and Coke supplies, and not putting unnecessary effort into mundane things like wallpapers.

It was really nice to get to know the upstream experts John (J5) Palmieri and Tomeu Vizoso (check out Tomeu’s blog post for his summary and some really nice photos). When sitting together in a room, fully focussing on this area for a full week, it’s so much easier to just ask them about something and getting things done and into upstream than on IRC or bugzilla, where you don’t know each other personally. I certainly learned a lot this week (and not only how great Czech beer tastes :-) )!

So what did I do?

Application porting

After already having ported four Ubuntu PyGTK applications to GI before (apport, jockey, aptdaemon, and language-selector),
my main goal and occupation during this week was to start porting a bigger PyGTK application. I picked system-config-printer, as it’s two magnitudes bigger than the previous projects, exercises quite a lot more of the GTK GI bindings, and thus also exposes a lot more GTK annotation and pygobject bugs. This resulted in a new pygi s-c-p branch which has the first 100 rounds of “test, break, fix” iterations. It now at least starts, and you can do a number of things with it, but a lot of functionality is still broken.

As a kind of “finger exercise” and also to check for how well pygi-convert works for small projects now, I also ported computer-janitor. This went really well (I had it working after about 30 minutes), and also led me to finally fixing the unicode vs. str mess for GtkTreeView that you got so far with Python 2.x.

pygobject and GTK fixes

Porting system-config-printer and computer-janitor uncovered a lot of opportunities to improve pygi-convert.sh, a big “perl -e” kind of script to do the mechanical grunt work of the porting process. It doesn’t fix up changed signatures (such as adding missing arguments which were default arguments in PyGTK, or the ubiquitous “user_data” argument for signal handlers), but at least it gets a lot of namespaces, method, and constant names right.

I also fixed three annotation fixes in GTK+. We also collaboratively reviewed and tested Pavel’s annotation branch which helped to fix tons of problems, especially after Steve Frécinaux’s excellent reference leak fix, so if you play around with current pygobject git head, you really also have to use the current GTK+ git head.

Speaking of which, if you want to port applications and always stay on top of the pygobject/GTK development without having to clutter your package system with “make install”s of those, it works very well to have this in your ~/.bashrc:

export GI_TYPELIB_PATH=$HOME/projects/gtk/gtk:$HOME/projects/gtk/gdk
export PYTHONPATH=$HOME/projects/pygobject

Better GVariant/GDBus support

The GNOME world is moving from the old dbus-glib python bindings to GDBus, which is integrated into GLib. However, dbus-python exposed a really nice and convenient way of doing D-Bus calls, while using GDBus from Python was hideously complicated, especially for nontrivial arguments with empty or nested arrays:

from gi.repository import Gio, GLib
from gi._gi import variant_type_from_string

d = Gio.bus_get_sync(Gio.BusType.SESSION, None)
notify = Gio.DBusProxy.new_sync(d, 0, None, 'org.freedesktop.Notifications',
    '/org/freedesktop/Notifications', 'org.freedesktop.Notifications', None)

vb = GLib.VariantBuilder()
vb.init(variant_type_from_string('r'))
vb.add_value(GLib.Variant('s', 'test'))
vb.add_value(GLib.Variant('u', 1))
vb.add_value(GLib.Variant('s', 'gtk-ok'))
vb.add_value(GLib.Variant('s', 'Hello World!'))
vb.add_value(GLib.Variant('s', 'Subtext'))
# add an empty array
eavb = GLib.VariantBuilder()
eavb.init(variant_type_from_string('as'))
vb.add_value(eavb.end())
# add an empty dict
eavb = GLib.VariantBuilder()
eavb.init(variant_type_from_string('a{sv}'))
vb.add_value(eavb.end())
vb.add_value(GLib.Variant('i', 10000))
args = vb.end()

result = notify.call_sync('Notify', args, 0, -1, None)
id = result.get_child_value(0).get_uint32()
print id

So I went to making the GLib.Variant constructor work properly with nested types and boxed variants, adding Pythonic GVariant iterators and indexing (so that you can treat GVariant dictionaries/arrays/tuples just like their Python equivalents), and finally a Variant.unpack() method for converting the return value of a D-Bus call back into a native Python data type. This looks a lot friendlier now:

from gi.repository import Gio, GLib

d = Gio.bus_get_sync(Gio.BusType.SESSION, None)
notify = Gio.DBusProxy.new_sync(d, 0, None, 'org.freedesktop.Notifications',
    '/org/freedesktop/Notifications', 'org.freedesktop.Notifications', None)

args = GLib.Variant('(susssasa{sv}i)', ('test', 1, 'gtk-ok', 'Hello World!',
    'Subtext', [], {}, 10000))
result = notify.call_sync('Notify', args, 0, -1, None)
id = result.unpack()[0]
print id

I also prepared another patch in GNOME#640181 which will provide the icing on the cake, i. e. handle the variant building/unpacking transparently and make the explicit call_sync() unnecessary:

from gi.repository import Gio, GLib

d = Gio.bus_get_sync(Gio.BusType.SESSION, None)
notify = Gio.DBusProxy.new_sync(d, 0, None, 'org.freedesktop.Notifications',
    '/org/freedesktop/Notifications', 'org.freedesktop.Notifications', None)

result = notify.Notify('(susssasa{sv}i)', 'test', 1, 'gtk-ok', 'Hello World!',
            'Subtext', [], {}, 10000)
print result[0]

I hope that I can get this reviewed and land this soon.

Thanks to our sponsors!

Many thanks to the GNOME Foundation and Collabora for sponsoring this event!

Read more
kenvandine

For some time now I have been struggling with getting Gwibber working reliably with facebook.  Since Gwibber was included by default in Ubuntu, usage has gone way up and we quickly exceeded our API request allocation with Facebook.  Facebook allocations are per application, not per user, which means Facebook blocks API requests for everyone, not just the users which are refreshing too often, etc.

I have been desperately attempting to contact anyone from facebook to help figure out the best way to solve the problem and have been mostly ignored.  I did get one reply from their developer relations stating that they don’t “support embedding video content in our website at this time”.  Clearly they didn’t even read my email :-(

I understand their need for allocations and to throttle as needed, however the numbers really don’t add up.  They are really skewed, there is one API request that we make which is way over the allocation, but the others are barely even on the chart.

Facebook just announced “Operation Developer Love” where they state they will stop ignoring developers and triage bugs.  Please help us get to the bottom of this by voting on this bug.

Read more
pitti

It’s been a decade ago when I did my first steps with contributing to Free Software, about seven years when I joined Debian, and about 6 with Canonical and Ubuntu. Time for some reflection what I have done over these years!

Distribution Packaging and Maintenance

My first sponsored Debian upload ever was cracklib2, which seriously needed some love and was looking for a new maintainer. So in that upload I managed to close all outstanding bugs. Thanks to my mentor Martin Godisch about providing a lot of guidance for this!

Since then I’ve maintained various packages, where the most popular ones are certainly the free database server “PostgreSQL” (see next section) and the e-book management software “Calibre”.

“Maintaining” by and large means “making it really easy to get and use this software”. This decomposes to:

  • Packaging it in a way that a simple apt-get install makes the software work out of the box (as far as possible)
  • Provide a default configuration/customizations so that it integrates and plays well with the rest of the system; this includes the paths and permissions for log files, log rotation, debconf, configuration file standards, etc.
  • Be the front line for bug reports from users, sort, answer, and de-duplicate them, and either fix them myself, or forward useful bugs to upstream.
  • Providing security updates for stable releases
  • To some extent, help with the development of the software; this gets mostly driven by user demand, and of course my personal interests.

In August 2004 I got employed by Canonical to work full time on Ubuntu, which pretty much turned a hobby into a profession. I never regretted this in the past years, it’s an awesome job to do!

In principle I’m doing the same thing in Ubuntu as well: Bring stuff from developers to the people out there. Except with a different focus, in Ubuntu my daily bread and butter is the GNOME desktop and stuff around (and immediately below) it. And even though after a long day of bug triaging and debugging I feel a bit low-hearted (“50.000 bugs away from perfection”), when I take a step back and see how much the usage of Free Software in the world has grown since 2004, I am very proud of being part of Ubuntu, which certainly has its fair contribution and share in this growth. So what seemed like a crazy idea from Mark back in 2004 actually has made a remarkable progress.

In the beginning of Ubuntu I mostly sent back patches to the Debian bug tracker, but this evolved quite a bit on both sides: These days I try to keep “my” packages in sync and commit stuff directly to Debian, which works very well with e. g. the pkg-utopia team, which is responsible for HAL, udisks, upower, PolicyKit, and related packages. At this point I want to thank Michael Biebl for being such an awesome guy on the Debian side! Also, it seems that Debian has moved a fair bit away from the strong “Big Maintainer Lock” towards team based maintenance, so these days it is easier than ever to commit stuff directly to Debian for a lot of packages, without much fuss.

PostgreSQL

I have done a handful of changes to PostgreSQL, but these mostly concerned easy packaging and crash fixes, nothing out of the ordinary. I’m not really a PostgreSQL upstream developer.

The thing I am really proud of is the postgresql-common package, which is a very nice example what a distro can provide on top of upstream: If you install the upstream tarball, you have to manually care for creating clusters, providing a sensible configuration for them, set up SSL, set up log rotation, etc. With postgresql-common, this is all done automatically. The biggest feature it provides is a robust and automatic way of upgrading between major releases with the pg_upgradecluster tool, which takes care of a dozen corner cases and the nontrivial process of dumping the old cluster and reloading the new one. Also, you can effortlessy run several instances of the same version in parallel, so that you can e. g. have a production and a development instance, or try the new 9.0 RC1 while still running the 8.4 production one. (more details)

Crash Reporting with Apport

This has been a pet peeve of mine pretty much from day 1. Back in the old days, crashes in software were a pain to track down: many crashes are hard to reproduce, it takes ages to get useful information from bug reporters, and a lot of data cannot be recovered any more when you try to reproduce and analyze a crash after the fact.

With the growing demand for QA from both Canonical and Ubuntu, in 2006 I finally got some time to start Apport, which would make all this a lot easier: It intercepts crashes as they happen, collects the data that we as a developer need, and makes it very easy for the user to submit them as a bug report. This is accompanied by a backend service (called “retracers”) which would reprocess the bug reports by taking the core dump, reproducing a chroot with the packages and versions that the reporter had, installing the debugging symbols, and re-running gdb, to produce a fully symbolic stack trace.

See this bug report for how this looks like. Since then, tons of crashes were fixed, way more than we could ever have done “the old way” with asking users to rebuild with “-g -O0″, running gdb, etc.

By today, Apport has grown quite a bit: rich bug reports, per-package hooks, automatic duplication of crashes, interactive GUI elements in hooks, etc.

Plumbing Development

Handling hotpluggable hardware has always interested me, since the day when I got my first USB stick and it was ridiculously hard (from an user perspective) to use it:

    $ su -
    # mount -t vfat -o uid=1000 /dev/sda1 /mnt

My first go at this was to write pmount which would allow normal users to mount hotpluggable storage without root privileges and worrying about mount paths and options, and then integrate it into GNOME and HAL. Personally I abandoned it years ago, but it seems other people still use it, so I’m glad that Vincent Fourmond took over the maintenance now.

Since then, the entire stack evolved quite a bit: HAL grew to something useful and rather secure, and finally into some monstrous unmaintainable beast, which is why it was declared dead in 2008, and replaced with the “U” stack: udev, udisks, upower, etc. I enjoy hacking on that a lot, and since it’s part of my Desktop Team Tech Lead/Developer role in Canonical, I can spend some company time on it. so far I worked on bug fixes, small new features, and writing a rather comprehensive test suite for udisks (see my udisks commits so far). I also did a fair share of porting stuff away from HAL to the new stack, including some permanent commitments like maintaining the keymaps in udev.

GNOME

Before I started with Canonical I was never much of a GUI person: I was fully content with fvwm and a few xterms around it. But as an Ubuntu developer I do dogfooding, and thus I switched to GNOME for my day to day work. It didn’t take long before I really fell in love with it!

Similar to my Debian packages, my upstream involvement with GNOME is mostly integration and bug fixing. As already explained, we get a looooot of bug reports, so my focus is mostly on bug fixing. To date, I sent 93 patches to bugzilla. Since January 2010 I became a committer, so that it’s easier for me to get patches upstream.

Debugging problems and fixing bugs is a pretty tedious task, but I still enjoy the rewarding nice feeling when you finally tracked down something and can close a bug with 50 duplicates, and you have made people’s life a little bit easier from now on.

Read more

Allison Randall has blogged about her new role in Ubuntu; working at Canonical as the Technical Architect for Ubuntu. One thing which I think is awesome is how she mentions people who encouraged her:

To give credit where credit is due, there have been 4 great influences on my career over the years, mentors, friends, people who believed in me, encouraged me to dream big dreams and try big things, who taught me that I’m better, smarter, wiser, more dynamic, and resilient than I ever imagined. In alphabetical order: Damian Conway, Greg Kroah-Hartman, Mark Shuttleworth, and Nathan Torkington. Thanks guys, I wouldn’t be here without you!

Everyone has mentors in Free Software, so Allison’s post inspired me to talk about mine. People who made you who you are. I hope this encourages you to talk about the people who inspired you.

While Allison’s post motivated me, this next month I get to celebrate 3 years at Canonical and I would like to thank people who have motivated me to do what I do. Think of them as the root of me:

  • Dave Camp - Dave is one of the older Ximian folk, a former co-maintainer of Nautilus, and later he worked on Mozilla. He taught me to ignore the haters, and rock on by your bad self. He’s a hacker’s hacker, a dude who puts his head down and skates. He doesn’t care about fame, fortune, or any of that jazz, he’s just a dude. He is also an amazing guitar player.
  • Luis Villa - Not much to say here, he’s brought so many of us here that I have no words that could ever be kind enough to explain what Luis has done for so many of us.
  • Jeff Waugh and Benjamin “Mako” Hill - I am going to mush them together, since at the time they were the collective first “Ubuntu Community managers”. Jeff for being the spark plug of motivation and getting me to a UDS, and Mako for being the Free Software advocate who leads by example, not by shoving the GPL down people’s throat. I would love to hear much more from both of you. Both of you believed in me from the beginning, and I will always be grateful.
  • Asa Dotzler - many years ago when I first started wondering what Mozilla was he took the time to explain open source to me. I also love that he flames Linux for what it is, maybe someday we’ll get our act together. :) I’m not even sure if he’s looking at Linux these days but I’ll always appreciate his first interaction with me.

These people pointed me in the right direction, and these are the ones who focused me into a fine instrument of Ubuntu laser-destruction. If you hate me, then it’s probably these people’s fault:

  • Luke Kanies from Puppet Labs (at the time from Reductive Labs), who convinced me to think about applying for my current job at Canonical.
  • Oliver Grawert, who made me actually do that or he promised to punch me in the face.
  • Daniel Holbach. The name mentioned by Jono Bacon when he said “I am starting a new team” that made me apply for the job on the spot and totally not care what the consequences are. He has that effect on people.
  • … and of course the rest of you on the community team (and I just don’t mean Canonical folk) who have been supportive of me over the years, even when I quit the team and had a temper tantrum a few years ago.)

And on top of that I’ve got the old folks, the mentors who are just awesome by being there, they’re all good friends and good mentors. I am pretty sure that everyone who has worked with these people are not surprised:

  • Chris Blizzard - old school GNOME, now at Mozilla. He builds airplanes ffs.
  • Vincent Untz - old school GNOME, now at Novell. He builds ice cream ffs.
  • Miguel de Icaza - old school GNOME, now at Novell. He builds stack exchanges ffs.
  • Ryan Lortie - old school GNOME, now at Codethink. He builds dconf ffs.

I’d also like to take a moment to thank the new people — those of you who are new here and want to rock. Spend some time researching my generation’s heroes, and learn from what they have to say.

Hah, I just thought “Some day this entire mess will all be yours”, but I don’t know if that’s a nice thing to say or not, so heh to you.

Read more
?????? ?????

The last few months we’ve been doing a lot of work to enable direct import of translations from different upstream VCS systems. For now, we’ve focused on getting one very important case right first (GNOME), and then we’ll extend it to supporting other upstreams as well.

How are we going to do it? First off, we’ve split it all into two separate stages:

  • get upstream translations into Launchpad
  • push upstream translations from Launchpad into Ubuntu

For some upstreams, getting them into Launchpad is trivial: they might already be hosted in Launchpad. For majority of them, however, it means pulling from different VCS systems. Thanks to Launchpad Code and Bazaar teams, getting the code in the form of bazaar branch is not that big a deal. However, when pulling translations from a VCS instead of getting them from tarballs means one slight complication. Translation templates (POT files) won’t be there, and we’ll have to regenerate them.

Regenerating templates differs from project to project. And doing it should be considered an unsafe operation. So, in the first step we are only going to support intltool-based modules, and generation of templates will happen inside a sandboxed environment. This will enable us to import upstream translations directly into read-only Launchpad projects: this is marked with green-coloured arrows on the diagram.

After that is done, we’ll start pushing all these translations directly into Ubuntu (blue-coloured arrows), minimizing the time it takes for translations to get from upstream translators to Ubuntu users.

I’ve written a more thorough explanation in my personal blog, so check it out.

Parts of this will be rolled out this cycle, but more will come in the coming months.

Read more
pitti

I finally listened to Sebastien Bacher and applied for GNOME commit rights yesterday, after hassling Seb once more about committing an approved patch for me. Surprisingly, it only took some 4 hours until my application was approved and my account created, wow! Apparently 71 patches are enough. :-)

With my new powers, I fixed a crash in gdm, and applied two stragglers into gvfs’ build system today.

More to come!

Read more
pitti

In the merciless vendetta for purging hal we now reached another major milestone: gvfs, GNOME’s file system layer which handles USB storage as well as virtual file systems for libgphoto2 cameras, Bluetooth devices, audio CDs, or ftp/sftp/cifs mounts, is now fully ported to libgudev and doesn’t need hal at all any more. These long nights of porting weren’t in vain, after all \o/.

Now I just need to hassle David Zeuthen to apply the patches soon. Of course I couldn’t wait and already uploaded them to Karmic, so please test the hell out of it and let me know about problems.

https://wiki.ubuntu.com/Halsectomy is a little greener once again. :-)

Read more
James Henstridge

In my last post, I said I had trouble getting Rygel’s tracker backend to function and assumed that it was expecting an older version of the API.  It turns out I was incorrect and the problem was due in part to Ubuntu specific changes to the Tracker package and the unusual way Rygel was trying to talk to Tracker.

The Tracker packages in Ubuntu remove the D-Bus service activation file for the “org.freedesktop.Tracker” bus name so that if the user has not chosen to run the service (or has killed it), it won’t be automatically activated.  Unfortunately, instead of just calling a Tracker D-Bus method, Rygel was trying to manually activate Tracker via a StartServiceByName() call.  This would fail even if Tracker was running, hence my assumption that it was a tracker API version problem.

This problem will be fixed in the next Rygel release: it will call a method on Tracker directly to see if it is available.  With that problem out of the way, I was able to try out the backend.  It was providing a lot more metadata to the PS3 so more files were playable, which was good.  Browsing folders was also much quicker than the folder back end.  There were a few problems though:

  1. Files are exposed in one of three folders: “All Images”, “All Music” or “All Videos”.  With even a moderate sized music collection, this is unmangeable.  It wasn’t clear what order the files were being displayed in either.
  2. There was quite a long delay before video playback starts.

When the folder back end fixes the metadata and speed issues, I’d be inclined to use it over the tracker back end.

Video Transcoding

Getting video transcoding working turned out to require a newer GStreamer (0.10.23), the “unstripped” ffmpeg libraries and the “bad” GStreamer plugins package from multiverse.  With those installed, things worked pretty well.  With these dependencies encoded in the packaging, it’d be pretty painless to get it set up.  Certainly much easier than setting things up in MediaTomb’s configuration file.

Read more