Canonical Voices

Posts tagged with 'ubuntu'

pitti

The unstoppable PostgreSQL team just announced the first release candidate of 9.2, with several bug fixes since the Beta 4. If you haven’t tested 9.2 yet, now is the time! Remember that you can run a copy of your 8.4 or 9.2 cluster in parallel for testing with pg_upgradecluster.

If you use Debian, 9.2rc1 will be available in experimental in a few hours. For Ubuntu, you can get packages for all supported releases from my PostgreSQL backports PPA as usual.

Enjoy!

Read more
pitti

I just released Apport 2.5 with a bunch of new features and some bug fixes.

By default you cannot report bugs and crashes to packages from PPAs, as they are not Ubuntu packages. Some packages like Unity or UbuntuOne define their own crash database which reports bugs against the project instead. This has been a bit cumbersome in the past, as these packages needed to ship a /etc/apport/crashdb.conf.d/ snippet. This has become much easier, package hooks can define a new crash database directly now (#551330):

def add_info(report, ui):
   if determine_whether_to_report_to_upstream:
       report['CrashDB'] = '{ "impl": "launchpad", "project": "picsaw" }'

(Documented in package-hooks.txt)

Apport now also looks for package hooks in /opt (#1020503) if the executable path or a file in the package is somewhere below /opt (it tries all intermediate directories).

With these two, we should have much better support for filing bugs against ARB packages.

This version also finally drops the usage of gksu and moves to PolicyKit. Now we only have one package left in the default install (update-notifier) which uses it. Almost there!

Read more
pitti

I just released PyGObject 3.3.90, for GNOME 3.5.90.

This is now working correctly on big-endian 64 bit machines such as powerpc64, and fixes marshalling for GParamSpec attributes and return values, as well as a few small bug fixes.

Thanks to all contributors!

Complete list of changes:

  • Implement marshalling for GParamSpec (Mathieu Duponchelle) (#681565)
  • Fix erronous import statements for Python 3.3 (Simon Feltman) (#682051)
  • Do not fail tests if pyflakes or pep8 are not installed (Martin Pitt)
  • Fix PEP-8 whitespace checking and issues in the code (Martin Pitt)
  • Fix unmarshalling of gssize (David Malcolm) (#680693)
  • Fix various endianess errors (David Malcolm) (#680692)
  • Gtk overrides: Add TreeModelSort.__init__(self, model) (Simon Feltman) (#681477)
  • Convert Gtk.CellRendererState in the pygi-convert script (Manuel Quiñones) (#681596)

Read more
pitti

Yesterday, GUADEC hosted a PyGObject hackfest. I was really happy to see so many participants, and a lot of whom who are rather new to the project. I originally feared that it would just be the core crew of four people, as this is not exactly the shiniest part of GNOME development.

So I did not work on the stuff I was planning for, but instead walked around and provided mentoring, help, and patch review. Unfortunately I do not know all the results from the participants, hopefully they will blog some details themselves. But this is what I was involved in:

  • Manuel Quiñones added an gtk_tree_view_column_set_attributes() override (the original C function uses varargs and thus is not introspectable). Most time was spent figuring out an appropriate test case.
  • I showed Didier Roche some tricks about porting a pygtk application to PyGI/GTK3. He gave a shot to porting Meld, but unfortunately it uses a lot of pygtk hacks/tricks, most of which are obsolete now. So this proved too big a project for one day eventually :-(
  • Paolo and I guided Marta Maria Casetti, one of this year’s GNOME GSoC students, through her first pygobject patch. The test case still needs some love (again, nothing regarding GtkTreeView is easy), but the actual patch is good. Thanks Marta for participating, and not getting intimidated by all the new stuff!
  • While working on above patch, Marta encountered a rather curious TypeError: Expected Gtk.TreeViewColumn, but got GObjectMeta when writing the override. What seemed to be a trivial problem at first quickly turned into an one-hour debugging session involving grandmaster John Palmieri and me, with others chipping in as well. In the end it (of course!) turned out to be a trivial four-character change in Marta’s patch, but it was fun to get to understand the problem (a loong-forgotten special case of overrides resolution in overrides code). Now pygobject gives a proper error message which is actually helpful, i. e. which argument causes the problem and which module/class/method is provided, which should prevent us from being misguided into the totally wrong direction the next time this happens.
  • John Stowers got the Windows build working again, and showed off the gtk-demo under Windows. This is really amazing, I hope we can get that into trunk soon and not let it bitrot again for so long. Thanks!
  • Simon and Manuel worked on porting some Sugar extensions. Together with Paolo we also discussed the GStreamer 1.0 API a bit, which parts can become API additions and which need to become overrides.
  • Michal Hruby debugged a leak in the handling of GVariant arrays when using libdee.

Thanks everyone for participating! I hope everyone enjoyed it and got to learn a new thing or two. See you at the next one!

PyGObject hackfest at GUADEC 2012

PyGObject hackfest at GUADEC 2012

Read more
pitti

I have had the pleasure of attending GUADEC in full this year. TL;DR: A lot of great presentations + lots of hall conversations about QA stuff + the obligatory be{er,ach} = ?.

Last year I just went to the hackfest, and I never made it to any previous one, so GUADEC 2012 was a kind of first-time experience for me. It was great to put some faces and personal impressions to a lot of the people I have worked with for many years, as well as catching up with others that I did meet before.

I discussed various hardware/software testing stuff with Colin Walters, Matthias Clasen, Lennart Poettering, Bertrand Lorentz, and others, so I have a better idea how to proceed with automated testing in plumbing and GNOME now. It was also really nice to meet my fellow PyGObject co-maintainer Paolo Borelli, as well as seeing Simon Schampier and Ignacio Casal Quinteiro again.

No need to replicate the whole schedule here (see for yourself on the interwebs), so I just want to point out some personal highlights in the presentations:

  • Jacob Appelbaum’s keynote about Tor brought up some surprising facts about how the project has outgrown its past performance problems and how useful it was during e. g. the Arab revolution
  • .

  • Philip Whitnall’s presented Bendy Bus, a tool to mock D-Bus services for both unit and fuzz testing. He successfully used it to find and replicate bugs in Evolution (by mocking evolution-data-server) as well as libfolks (by mocking the telepathy daemons). It should work just as well to mock system services like upower or NetworkManager to test the UI bits that use it. This is a topic which has been on my wishlist for a long time already, so I’m happy that there is already an existing solution out there. We might have to add some small features to it, but it’s by and large what I had in mind, and in the discussion afterwards Philip said he’d appreciate patches against it.
  • Christophe Fergeau showed how to easily do Windows builds and installers from GNOME tarballs with MinGW-w64, without having to actually touch/use Windows (using cross-building and running tests etc. under wine). I found it surprising how easy that actually is, and it should not be hard to integrate that in a jhbuild-like setup, so that it does not keep breaking every time.
  • Colin Walters gave an introduction to OSTree, a project to build bootable images from kernel/plumbing/desktop upstream git heads on a daily basis. This is mostly to avoid the long delays that we otherwise have with doing upstream releases, packaging them, and getting them into a form that can safely be tested by users. In an afterwards discussion we threw some ideas around how we can integrate existing and future tests into this (something in spirit like our autopkgtest). This will be the area where I’ll put most focus on in the next time.
  • Adam Dingle of yorba fame shared his thoughts about how we can crowdfunding of Free Software Projects work in practice, comparing efforts like codefoundry and kickstarter. Of course he does not have a solution for this yet, but he raised some interesting concerns and it spun off lots of good discussions over lunchtime.
  • Last but not least, the sport event on Saturday evening was awesome! In hindsight I was happy to not have signed up for soccer, as people like Bastian or Jordi played this really seriously. I participated in the Basketball competition instead, which was the right mix of fun and competition without seriously trying to hurt each other. :-)

There were a lot of other good ones, some of them technical and some amusing and enlightening, such as Frederico’s review of the history of GNOME.

On Monday I prepared and participated in a PyGObject hackfest, but I’ll blog about this separately.

I want to thank all presenters for the excellent program, as well as the tireless GUADEC organizer team for making everything so smooth and working flawlessly! Great Job, and see you again in Strasbourg or Brno!

Read more
pitti

I started to collect some easy PyGObject bugs which are appropriate for the PyGObject hackfest at GUADEC on July 30th. These are bugs which do not need a lot of previous knowlege and are excellent starters for new contributors, such as adding overrides, fixing build system issues, etc.

I also created an initial idea pool/agenda/coordination page, where participants can add or signup for things to work on.

Feel free to add your own topics! I’m really looking forward to GUADEC and the hackfest, see you there!

GUADEC 2012

Read more
pitti

I released PyGObject 3.3.4. This is mostly a bug fix only release to fix existing API. Highlights are that lists of GVariants and other corner cases are now working correctly when being passed from C to Python, and that calling help() on a GI module now does something sensible.

Thanks to all contributors!

Complete list of changes:

  • pygi-convert.sh: Drop bogus filter_new() conversion (Martin Pitt) (#679999)
  • Fix help() for GI modules (Martin Pitt) (#679804)
  • Skip gi.CallbackInfo objects from a module’s dir() (Martin Pitt) (#679804)
  • Fix __path__ module attribute (Martin Pitt)
  • pygi-convert.sh: Fix some child ? getChild() false positives (Joe R. Nassimian) (#680004)
  • Fix array handling for interfaces, properties, and signals (Mikkel Kamstrup Erlandsen) (#667244)
  • Add conversion of the Gdk.PropMode constants to pygi-convert.sh script (Manuel Quiñones) (#679775)
  • Add the same rules for pack_start to convert pack_end (Manuel Quiñones) (#679760)
  • Add error-checking for the case where _arg_cache_new() fails (Dave Malcolm) (#678914)
  • Add conversion of the Gdk.NotifyType constants to pygi-convert.sh script (Manuel Quiñones) (#679754)
  • Fix PyObject_Repr and PyObject_Str reference leaks (Simon Feltman) (#675857)
  • [API add] Gtk overrides: Add TreePath.__len__() (Martin Pitt) (#679199)
  • GLib.Variant: Fix repr(), add proper str() (Martin Pitt) (#679336)
  • m4/python.m4: Update Python version list (Martin Pitt)
  • Remove “label” property from Gtk.MenuItem if it is not set (Micah Carrick) (#670575)

Read more
pitti

I just received confirmation that my request for a PyGObject hackfest has been approved by the GUADEC organizers.

If you are developing GObject-introspection based Python applications and have some problems with PyGObject, this is the time and place to get to know each other, getting bugs fixed, learn about pygobject’s innards, or update libraries to become introspectable. I will prepare a list of easy things to look into if you are interested in learning about and getting involved in PyGObject’s development.

See you on July 30th in A Coruña!

GUADEC Badge

Read more
pitti

I released PyGObject 3.3.3.

The most notable changes are that you can now access methods (and other identifiers) which are Python keywords, PyGObject automatically escapes them now by appending a ‘_’. For example, you can now call myGdkWindow.raise_() or GLib.Thread.yield_() instead of having to resort to the previous workaround getattr(myGdkWindow, 'raise')().

This version also restores the deprecated get_data() and set_data() methods. They were never really meant to be used from Python programs, they can potentially mess up your program and cause crashes, and do not give you anything that regular Python object properties would not already provide in a much safer way (i. e. just write my_obj.foo = 'bar' instead of my_obj.set_data('foo', 'bar')). Apparently some software projects are using them, so they will now raise a deprecation warning and be removed for the GNOME 3.8 cycle instead.

Thanks to all contributors!

Complete list of changes:

  • Remove obsolete release-tag make target (Martin Pitt)
  • Do not do any python calls when GObjects are destroyed after the python interpreter has been finalized (Simon Schampijer) (#678046)
  • Do not change constructor-only “type” Window property (Martin Pitt) (#678510)
  • Escape identifiers which are Python keywords (Martin Pitt) (#676746)
  • Fix code for PEP-8 violations detected by the latest pep8 checker. (Martin Pitt)
  • Fix crash in GLib.find_program_in_path() (Martin Pitt) (#678119)
  • Revert “Do not bind gobject_get_data() and gobject_set_data()” (Martin Pitt) (#641944)
  • GVariant: Raise proper TypeError on invalid tuple input (David Keijser) (#678317)

Update:Just released 3.3.3.1 to fix a regresssion from the keyword escaping patch. It also escaped enum and flags names, but as they are translated to upper case they are never keywords.

Read more
pitti

A few weeks ago I wrote about my new role as an upstream QA engineer. I have now officially been in that role since June. Quite expectedly I had (and still have) some backlog from my previous Desktop engineer role, but I have had plenty of time to work on automatic tests and some test technology now. If you are interested in the daily details, you can look at the ramblings on my G+ page; in a nutshell I worked on integration tests for udisks2 (mostly upstream now), a mock polkit API, and a small enhancement of the scsi_debug kernel module. On the distro QA side I got the integration tests of udisks2, upower, PostgreSQL, Apport, and ubuntu-drivers-common working and added to our Jenkins autopkgtest runner, where they are executed whenever the particular package or any of its dependencies get updated. This already uncovered a surprising number of actual bugs, so I’m happy that this system starts being useful after the initial hump of getting the tests to run properly in that environment.

In that previous blog post I mentioned that Canonical will hire a second person for an upstream QA engineer role. I am pleased that the job posting is now online, so if you are familiar with how the Linux plumbing and desktop stacks work, are frantic about testing, like to work with the Linux, plumbing, GNOME, and other FOSS communities, know your way around jhbuild, Jenkins, and similar technologies, and would like to explore new possibilities like applying static code checking or creating APIs to fake hardware, please have a look at the role description! Please feel free to contact me on IRC (pitti on Freenode) or by email if you have further questions about the role.

Read more
pitti

I just released PyGObject 3.3.2, (almost) in time for tomorrow’s GNOME 3.5.2 release. No API breaks or new features this time, just lots of bug fixes and some minor API completions. My personal favorite is making closure calls work with GVariant arguments, which I finally figured out after over half a year; this finally unblocks making GDBus fully introspectable with not too much additional work, only that in the meantime dbus-python was ported to Python 3 so that the need for it is actually a lot smaller now.

Thanks to all contributors!

Complete list of changes:

  • foreign: Register cairo.Path and cairo.FontOptions foreign structs (Bastian Winkler) (#677388)
  • Check types in GBoxed assignments (Marien Zwart) (#676603)
  • [API add] Gtk overrides: Add TreeModelRow.get_previous() (Bastian Winkler) (#677389)
  • [API add] Add missing GObject.TYPE_VARIANT (Bastian Winkler) (#677387)
  • Fix boxed type equality (Jasper St. Pierre) (#677249)
  • Fix TestProperties.testBoxed test (Jose Rostagno) (#676644)
  • Fix handling of by-reference structs as out parameters (Carlos Garnacho) (#653151)
  • tests: Add more vfunc checks for GIMarshallingTestsObject (Martin Pitt)
  • Test caller-allocated GValue out parameter (Martin Pitt) (#653151)
  • GObject.bind_property: Support transform functions (Bastian Winkler) (#676169)
  • Fix lookup of vfuncs in parent classes (Carlos Garnacho) (#672864)
  • tests/test_properties.py: Fix whitespace (Martin Pitt)
  • gi: Support zero-terminated arrays with length arguments (Jasper St. Pierre) (#677124)
  • [API add] Add GObject.bind_property method (Simon Feltman) (#675582)
  • pygtkcompat: Correctly set flags (Jose Rostagno) (#675911)
  • Gtk overrides: Implement __delitem__ on TreeModel (Jose Rostagno) (#675892)
  • Gdk Color override should support red/green/blue_float properties (Simon Feltman) (#675579)
  • Support marshalling of GVariants for closures (Martin Pitt) (#656554)
  • _pygi_argument_from_object(): Check for compatible data type (Martin Pitt)
  • pygtkcompat: Fix color conversion (Martin Pitt)
  • test_gi: Check setting properties in constructor (Martin Pitt)
  • Support getting and setting GStrv properties (Martin Pitt)
  • Support defining GStrv properties from Python (Martin Pitt)
  • Add GObject.TYPE_STRV constant (Martin Pitt)
  • Unref GVariants when destroying the wrapper (Martin Pitt) (#675472)
  • Fix TestArrayGVariant test cases (Martin Pitt)
  • pygtkcompat: Add gdk.pixbuf_get_formats compat code (Jose Rostagno) (#675489)
  • pygtkcompat: Add some more compat functions (Jose Rostagno) (#675489)
  • Fix tests for Python 3 (Martin Pitt)
  • Fix building with –disable-cairo (Martin Pitt)
  • tests: Fix deprecated assertions (Martin Pitt)
  • Run tests with MALLOC_PERTURB_ (Martin Pitt)

Read more
pitti

New PostgreSQL microreleases with two security fixes and several bug fixes was just announced publically.

I spent the morning with the packaging orgy for Debian unstable and experimental (now uploaded), Debian Wheezy (update sent to security team), Ubuntu hardy, lucid, natty, oneiric, precise (LP #1008317) and my backports PPA.

I tested these fairly thoroughly, but please let me know if you encounter any problem with these.

Read more
pitti

As I wrote two weeks ago, I consider the QA related changes in Ubuntu 12.04 a great success. But while we will continue and even extend our efforts there, this is not where the ball stops: it’s great to have the feedback cycle between “break it” and “notice the bug” reduced from potentially a few months to one day in many cases, but wouldn’t it be cool to reduce that to a few minutes, and also put the machinery right at where stuff really happens — into the upstream trunks? If for every commit to PyGObject, GTK, NetworkManager, udisks, D-BUS, telepathy, gvfs, etc. we’d immediately build and test all reverse dependencies and the committer would be told about regressions?

I have had the desire to work on automated tests in Linux Plumbing and GNOME for quite a while now. Also, after 8 years of doing distribution work of packaging and processes (tech lead, release engineering/management, stable release updates, etc.) I wanted to shift my focus towards technology development. So I’ve been looking for a new role for some time now.

It seems that time is finally there: At the recent UDS, Mark announced that we will extend our QA efforts to upstream. I am very happy that in two weeks I can now move into a role to make this happen: Developing technology to make testing easier, work with our key upstreams to set up test suites and reporting, and I also can do some general development in areas that are near and dear to my heart, like udev/systemd, udisks, pygobject, etc. This work will be following the upstream conventions for infrastructure and development policies. In particular, it is not bound by Canonical copyright license agreements.

I have a bunch of random ideas what to work on, such as:

  • Making it possible/easier to write tests with fake hardware. E. g. in the upower integration tests that I wrote a while ago there is some code to create a fake sysfs tree which should really go into udev itself, available from C and introspection and be greatly extended. Also, it’s currently not possible to simulate a uevent that way, that’s something I’d like to add. Right now you can only set up /sys, start your daemon, and check the state after the coldplugging phase.
  • Interview some GNOME developers what kind of bugs/regressions/code they have most trouble with and what/how they would like to test. Then write a test suite with a few working and one non-working case (bugzilla should help with finding these), discuss the structure with the maintainer again, find some ways to make the tests radically simpler by adding/enhancing the API available from gudev/glib/gtk, etc. E. g. in the tests for apport-gtk I noticed that while it’s possible to do automatic testing of GUI applications it is still way harder than it should and needs to be. I guess that’s the main reason why there are hardly any GUI tests in GNOME?
  • I’ve heard from several people that it would be nice to be able to generate some mock wifi/ethernet/modem adapters to be able to automatically test NetworkManager and the like. As network devices are handled specially in Linux, not in the usual /dev and sysfs manner, they are not easy to fake. It probably needs a kernel module similar to scsi_debug, which fakes enough of the properties and behaviour of particular nmetwork card devices to be useful for testing. One could certainly provide a pipe or a regular bridge device at the other end to actually talk to the application through the fake device. (NB this is just an idea, I haven’t looked into details at all yet).
  • For some GUI tests it would be much easier if there was a very simple way of providing mocks/stubs for D-BUS services like udisks or NetworkManager than having to set up the actual daemons, coerce them into some corner-case behaviour, and needing root privileges for the test due to that. There seems to be some prior art in Ruby, but I’d really like to see this in D-BUS itself (perhaps a new library there?), and/or having this in GDBus where it would even be useful for Python or JavaScript tests through gobject-introspection.
  • There are nice tools like the Clang static code analyzer out there. I’d like to play with those and see how we can use it without generating a lot of false positives.
  • Robustify jhbuild to be able to keep up with building everything largely unattended. Right now you need to blow away and rebuild your tree way too often, due to brittle autotools or undeclared dependencies, and if we want to run this automatically in Jenkins it needs to be able to recover by itself. It should be able to keep up with the daily development, automatically starting build/test jobs for all reverse dependencies for a module that just has changed (and for basic libraries like GLib or GTK that’s going to be a lot), and perhaps send out mail notifications when a commit breaks something else. This also needs some discussion first, about how/where to do the notifications, etc.

Other ideas will emerge, and I hope lots of you have their own ideas what we can do. So please talk to me! We’ll also look for a second person to work in that area, so that we have some capacity and also the possibility to bounce ideas and code reviews between each other.

Read more
pitti

I just uploaded Apport 2.1 to Quantal. A big change in that version is that the whole code now works with both Python 2 and 3, except for the launchpadlib crash database backend (as we do not yet have a python3-launchpadlib package).

I took some care that apport report objects get along with both strings (unicode type in Python 2) and byte arrays (str type in Python 2) in values, so most package hooks should still work. However, now is the time to check whether they also work with Python 3, to make the impending transition to Python 3 easier.

However, you need to watch out if you use projects or scripts which directly use python-apport to process reports: The open(), write(), and write_mime() methods now require the passed file descriptors to be open in binary mode. You will get an exception otherwise.

A common pattern so far has been code like

  report = apport.Report()
  report.load(open('myfile.crash'))

This needs to be changed to

  report = apport.Report()
  with open('myfile.crash', 'rb') as f:
      report.load(f)

The “with” context is not strictly required, but it takes care of timely closing the files again. This avoids ResourceWarning spew when you run this in test suites or enable warnings.

Read more
pitti

The first Beta of the upcoming PostgreSQL 9.2 was released yesterday (see announcement). Your humble maintainer has now created packages for you to test. Please give them a whirl, and report any problems/regressions that you may see to the PostgreSQL developers, so that we can have a rock solid 9.2 release.

Remember, with the postgresql-common infrastructure you can use pg_upgradecluster to create a 9.2 cluster from your existing 8.4/9.1 cluster and run them both in parallel without endangering your data.

For Debian the package is currently waiting in the NEW queue, I expect them to go into experimental in a day or two. For Ubuntu 12.04 LTS you can get packages from my usual PostgreSQL backports PPA. Note that you need at least postgresql-common version 0.130, which is available in Debian unstable and the PPA now.

I (or rather, the postgresql-common test suite) found one regression: Upgrades do not keep the current value of sequences, but reset them to their default value. I reported this upstream and will provide updated packages as soon as this is fixed.

Read more
pitti

This announcement comes very late (a week after release), but better late than never..

The first PyGObject 3.3 series release is now out, with lots of yummy fixes and improvements. Dieter, Sebastian, and I went through a round of bugzilla spring cleaning to clean up old bugs, fix simple bugs, and apply good patches that were waiting, so as a result the patch queue is now almost empty and PyGObject works better than ever.

There was also quite some work on the test suite: it became a lot stricter and robust, and now also enforces PEP8 compatibility and absence of pyflake errors of the code.

One small but handy new feature is that the freeze_notify() and handler_block() methods are now context managers, i. e. they automatically call the corresponding thaw_notify()/handler_unblock() at the end of the with statement in an exception-safe way. ((#672324)

There are almost no API changes in this release, so it should work fine with GNOME 3.4 and applications developed with pygobject 3.2. The one exception is the removal of the Gobject.get_data() and Gobject.set_data() methods. They were prone to errors and crashes as they are not safely bindable, and in Python you can and should just use normal Python object attributes instead.

Complete list of changes:

  • GSettings: allow extra keyword arguments (Giovanni Campagna) (#675105)
  • pygtkcompat: Correct Userlist module use (Jose Rostagno) (#675084)
  • Add release-news make rule (Martin Pitt)
  • Add “make check.nemiver” target (Martin Pitt)
  • Test flags and enums in GHash values (Martin Pitt) (#637466)
  • tests: Activate test_hash_in and apply workaround (Martin Pitt) (#666636)
  • Add special case for Gdk.Atom array entries from Python (Martin Pitt) (#661709)
  • test_gdbus: Call GetConnectionUnixProcessID() with correct signature (Martin Pitt) (#667954)
  • Add test case for Gtk.ListStore custom sort (Martin Pitt) (#674475)
  • GTK overrides: Add missing keyword arguments (Martin Pitt) (#660018)
  • Add missing override for TreeModel.iter_previous() (Martin Pitt) (#660018)
  • pygi-convert.py: Drop obsolete drag method conversions (Martin Pitt) (#652860)
  • tests: Replace deprecated assertEquals() with assertEqual() (Martin Pitt)
  • Plug tiny leak in constant_info_get_value (Paolo Borelli) (#642754)
  • Fix len_arg_index for array arguments (Bastian Winkler) (#674271)
  • Support defining GType properties from Python (Martin Pitt) (#674351)
  • Handle GType properties correctly (Bastian Winkler) (#674351)
  • Add missing GObject.TYPE_GTYPE (Martin Pitt)
  • Fix test_mainloop.py for Python 3 (Martin Pitt)
  • Make callback exception propagation test stricter (Martin Pitt) (#616279)
  • Add context management to freeze_notify() and handler_block(). (Simon Feltman) (#672324)
  • Add support for GFlags properties (Martin Pitt) (#620943)
  • Wrap GLib.Source.is_destroyed() method (Martin Pitt) (#524719)
  • Fix error message when trying to override a non-GI class (Martin Pitt) (#646667)
  • Fix segfault when accessing __grefcount__ before creating the GObject (Steve Frécinaux) (#640434)
  • Do not bind gobject_get_data() and gobject_set_data() (Steve Frécinaux) (#641944)
  • Add test case for multiple GLib.MainLoop instances (Martin Pitt) (#663068)
  • Add a ccallback type which is used to invoke callbacks passed to a vfunc (John (J5) Palmieri) (#644926)
  • Regression test: marshalling GValues in GHashTable (Alberto Mardegan) (#668903)
  • Update .gitignore (Martin Pitt)
  • Fix “distcheck” and tests with out-of-tree builds (Martin Pitt)
  • Add a pep8 check to the makefile (Johan Dahlin) (#672627)
  • PEP8 whitespace fixes (Johan Dahlin) (#672627)
  • PEP8: Remove trailing ; (Johan Dahlin) (#672627)
  • tests: Replace deprecated Python API (Martin Pitt)
  • Fail tests if they use or encounter deprecations (Martin Pitt)
  • Do not run tests in two phases any more (Martin Pitt)
  • test_overrides: Find local gsettings schema with current glib (Martin Pitt)
  • Add GtkComboBoxEntry compatibility (Paolo Borelli) (#672589)
  • Correct review comments from Martin (Johan Dahlin) (#672578)
  • Correct pyflakes warnings/errors (Johan Dahlin) (#672578)
  • Make tests fail on CRITICAL logs, too, and apply to all tests (Martin Pitt)
  • Support marshalling GI_TYPE_TAG_INTERFACE (Alberto Mardegan) (#668903)
  • Fix warnings on None values in added tree/list store rows (Martin Pitt) (#672463)
  • pygtkcompat test: Properly clean up PixbufLoader (Martin Pitt)

Read more
pitti

Half a year ago I blogged about the changed expectancies and processes to improve quality of the development release which we discussed at the UDS in Orlando: A promise that we don’t break the development version, regressions are not to be tolerated, acceptance criteria for Canonical upstreams. For that we introduced the Stable+1 team, actually did some reversions of broken packages, our QA team set up rigorous daily installation image and upgrade tests, and the code development process for Unity and related project was changed to enforce buildability and passing automatic tests with each and every change to trunk.

To be honest I was still a tad sceptic back then when this was planned. These were a lot of changes for one cycle, the stable+1 team was a considerable resource investment (starting with three people fulltime in the first few months), and not to the least our friends in the DX team felt thwarted because they had to sit down for a long time developing tests, and then changing their habits and practices for development.

So was all that effort worth it?

One word: OMGCRYOUTLOUDYES!!!!

Just a random sample of goodness that this brought:

  • It was nice to not have to sit down for an hour every cople of days to figure out how to get back my desktop after the daily dist-upgrade bricked it.
  • Unity, compiz, and friends were remarkably stable. I still remember the previous cycles where every new version got differently crashy, broke virtual workspaces, and what not. The worst thing that happened this cycle is eternally breaking keybindings (or changing them around), but at least those usually had obvious workarounds.
  • As a result of those, I think we had at least one, maybe two magnitudes more testers of the daily development release than in previous cycles. So we got a lot of good bug reports and also patch contributions for smaller issues in Precise which we otherwise would not have discovered.
  • The daily dist-upgrade tests tremendously helped to uncover packaging problems which would break real-world upgrades out there by the dozens. It took months to fix the hardest one: upgrading 10.04 LTS to 12.04 LTS with all universe packages offered in software-center. This beast takes 13 hours to run, so nobody really did manual tests like that in the past cycles.
  • Due to the daily automatic CD image builds we dramatically reduced both the cost of fixing regressions as well as the emergency hackathons during milestone preparations. It is a lot easier to unbreak e. g. LVM setup or OEM install modes on our images when the regression happened just a day before than discovering it two days before a milestone is due, as again nobody tests these less common modes very often.
  • So as a result, I really think the investments into QA and the stable+1 teams already paid off twofold by giving us more time to work on the less critical fixes, avoiding lots of user frustration about broken upgrades, and generally making the daily development a lot more enjoyable. Or, as Rick Spencer puts it: Velocity, velocity, velocity!

    Despite these improvements, there are still some improvements I’m looking forward to in the next cycles: Thanks to Colin Watson we can now use -proposed as a proper staging area, and used this feature rather extensively in the past month. From my point of view, 90% of the remaining daily dist-upgrade failures were due to packages building on different architectures at vastly different times, or failing on some, but not all architectures (“arch skew”). This is something you cannot really predict or guard against as a developer when you upload large and potentially harmful packages directly to the development release, so uploading them to the staging area and letting everything build there will reduce the breakage to zero. This was successfully demonstrated with Unity, GTK, and other packages where arch skew pretty much always causes people to hose their desktop, as well as daily CD images not working.

    I’m also looking forward to combining the staging area with lots of automatic tests against reverse dependencies (e. g. testing the installer against a new GTK or pygobject before it lands), something we just barely tipped our toes in.

    I can’t imagine how we were ever able to develop our new releases the old way. :-)

    Precise Pangolin^W^WUbuntu 12.04, I’m proud of you! Go out and amaze people!

    Read more
pitti

I just released a new pygobject version 3.1.92, for this week’s GNOME 3.3.92. This was my first-ever GNOME release (yay!), so please bear with me.

One highlight of this release is the new pygtkcompat module, contributed by Johan Dahlin. It provides backwards compatibility to pygtk far beyond to what the Gtk overrrides do, and also includes some shims for the old static webkit, gudev, and other modules. You can, and have to, enable them individually:

import gi.pygtkcompat

# enable "gobject" and "glib" modules
gi.pygtkcompat.enable()
gi.pygtkcompat.enable_gtk(version='3.0')

import glib
import gtk

Now you can use gtk.Window(), glib.timeout_add() etc. as before, and these will be transparently be converted into their modern GI counterparts. Please note that this is still in its infancy, and also mostly meant to ease the porting to GI. It’s not something we’ll keep forever.

Thanks to Michel Dänzer this release now also works properly on big-endian machines.

I mostly worked on fixing the calls of methods which take a list of GValues as arguments, such as Gtk.ListStore.insert_with_valuesv() and similar functions, and made the override API for tree models (append() etc. with providing row data) atomic wrt. the signals it sends out.

I want to thank Johan and Paolo for the nice teamwork with reviewing each other’s patches. That’s open source at its best!

Complete list of changes:

  • Correct Gtk.TreePath.__iter__ to work with Python 3 (Johan Dahlin)
  • Fix test_everything.TestSignals.test_object_param_signal test case (Martin Pitt)
  • Add a PyGTK compatibility layer (Johan Dahlin)
  • pygtkcompat: Remove first argument for get_origin() (Johan Dahlin)
  • Fix pygtkcompat.py to work with Python 3 (Martin Pitt)
  • GtkViewport: Add a default values for the adjustment constructor parameters (Johan Dahlin)
  • GtkIconSet: Add a default value for the pixbuf constructor parameter (Johan Dahlin)
  • PangoLayout: Add a default value for set_markup() (Johan Dahlin)
  • Gtk[HV]Scrollbar: Add a default value for the adjustment constructor parameter (Johan Dahlin)
  • GtkToolButton: Add a default value for the stock_id constructor parameter (Johan Dahlin)
  • GtkIconView: Add a default value for the model constructor parameter (Johan Dahlin)
  • Add a default value for column in Gtk.TreeView.get_cell_area() (Johan Dahlin)
  • Atomic inserts in Gtk.{List,Tree}Store overrides (Martin Pitt)
  • Fix Gtk.Button constructor to accept use_stock parameter (Martin Pitt)
  • Correct bad rebase, remove duplicate Window (Johan Dahlin)
  • Add bw-compatible arguments to Gtk.Adjustment (Johan Dahlin)
  • GtkTreePath: make it iterable (Johan Dahlin)
  • Add a default argument to TreeModelFilter.set_visible_func() (Johan Dahlin)
  • Add a default argument to Gtk.TreeView.set_cursor (Johan Dahlin)
  • Add a default argument to Pango.Context.get_metrics() (Johan Dahlin)
  • Fix double-freeing GValues in arrays (Martin Pitt)
  • Renamed “property” class to “Property” (Simon Feltman)
  • Fix Python to C marshalling of GValue arrays (Martin Pitt)
  • Correct the Gtk.Window hierarchy (Johan Dahlin)
  • Renamed getter/setter instance attributes to fget/fset respectively. (Simon Feltman)
  • Add Gtk.Arrow/Gtk.Window constructor override (Johan Dahlin)
  • Fix marshalling to/from Python to work on big endian machines. (Michel Dänzer)
  • Use gi_cclosure_marshal_generic instead of duplicating it. (Michel Dänzer)
  • Override Gtk.TreeView.get_visible_range to fix return (René Stadler)
  • Plug memory leak in _is_union_member (Paolo Borelli)
  • tests: Split TestInterfaces into separate tests (Sebastian Pölsterl)
  • README: Update current maintainers (Martin Pitt)

Read more
pitti

Part of our efforts to reduce power consumption in Ubuntu is to provide an easy tool to hunt down which programs and devices are to blame for inordinate power consumption. powertop’s interactive mode is pretty good for this if you are sitting in a train and want to tweak some knobs to max out battery life, but we need something more reproducible and noninteractive for developers who want to file proper bug reports.

So I wrote a little script power-usage-report which calls fatrace for measuring file access activity from programs, and powertop-1.13 to measure process and device wakeups, clean up and sort their ouput, and generate a report which is appropriate to attach to bug reports, send around, put into Jenkins for measuring daily progress, etc. It is now part of fatrace version 0.4, so today’s Precise upgrades will have it.

The output has several sections for disk access (which prevent the disk from spinning down), wakeups (causing CPU power usage), and device activity. Disk/wakeups are sorted in descending order by process:

$ sudo power-usage-report
Measurement will begin in 5 seconds. Please make sure that the
computer is idle, i. e. do not press keys, start or operate programs, and that
programs are not busy with active tasks other than the one you want to examine.
Starting measurement for 60 seconds...
Measurement complete. Generating report...
======= unity-panel-ser: 5 file access events ======
/usr/share/zoneinfo/UTC: 1 reads
/etc/timezone:
/usr/share/zoneinfo/posix/Europe/Berlin: 1 reads
/etc/localtime: 3 reads

======= gnome-settings-: 1 file access events ======
/etc/fstab: 1 reads

======= telepathy-gabbl: 1 file access events ======
/home/martin/.cache/wocky/caps/caps-cache.db: 1 reads

====== Wakeups ======
  30,9% ( 52,0)   compiz
  16,3% ( 27,4)   [iwlwifi] 
  12,5% ( 21,0)   [i915] 
   3,7% (  6,3)   [ahci] 
   2,3% (  3,9)   swapper/3
   1,2% (  2,0)   gvfs-afc-volume
[...]

====== Devices ======
An audio device is active 100,0% of the time:
hwC0D0 Conexant CX20585 

Recent USB suspend statistics
Active  Device name
100,0%	USB device 1-1.5.4.4 : USB Mouse (A4Tech)
100,0%	/sys/bus/usb/devices/1-1.5.4.2
100,0%	USB device 1-1.5.4 : Kinesis Keyboard Hub (PI Engineering)
  0,0%	USB device 1-1.5.2 : USB2.0 Hub Controller (NEC Corporation)

[...]

You can redirect output to a file, of course. The top header (“Starting measurement..” etc.) will go to stderr and thus not be part of the redirected output.

Read more
pitti

Part of our efforts to reduce power consumption is to identify processes which keep waking up the disk even when the computer is idle. This already resulted in a few bug reports (and some fixes, too), but we only really just began with this.

Unfortunately there is no really good tool to trace file access events system-wide. powertop claims to, but its output is both very incomplete, and also wrong (e. g. it claims that read accesses are writes). strace gives you everything you do and don’t want to know about what’s going on, but is per-process, and attaching strace to all running and new processes is cumbersome. blktrace is system-wide, but operates at a way too low level for this task: its output has nothing to do any more with files or even inodes, just raw block numbers which are impossible to convert back to an inode and file path.

So I created a little tool called fatrace (“file access trace”, not “fat race” :-) ) which uses fanotify, a couple of /proc lookups and some glue to provide this. By default it monitors the whole system, i. e. all mounts (except the virtual ones like /proc, tmpfs, etc.), but you can also tell it to just consider the mount of the current directory. You can write the log into a file (stdout by default), and run it for a specified number of seconds. Optional time stamps and PID filters are also provided.

$ sudo fatrace
rsyslogd(967): W /var/log/auth.log
notify-osd(2264): O /usr/share/pixmaps/weechat.xpm
compiz(2001): R device 8:2 inode 658203
[...]

It shows the process name and pid, the event type (Rread, Write, Open, or Close), and the path. Sometimes its’ not possible to determine a path (usually because it’s a temporary file which already got deleted, and I suspect mmaps as well), in that case it shows the device and inode number; such programs then need closer inspection with strace.

If you run this in gnome-terminal, there is an annoying feedback loop, as gnome-terminal causes a disk access with each output line, which then causes another output line, ad infinitum. To fix this, you can either redirect output to a file (-o /tmp/trace) or ignore the PID of gnome-terminal (-p `pidof gnome-terminal`).

So to investigate which programs are keeping your disk spinning, run something like

  $ sudo fatrace -o /tmp/trace -s 60

and then do nothing until it finishes.

My next task will be to write an integration program which calls fatrace and powertop, and creates a nice little report out of that raw data, sorted by number of accesses and process name, and all that. But it might already help some folks as it is right now.

The code lives in bzr branch lp:fatrace (web view), you can just run make and sudo ./fatrace. I also uploaded a package to Ubuntu Precise, but it still needs to go through the NEW queue. I also made a 0.1 release, so you can just grab the release tarball if you prefer. Have a look at the manpage and --help, it should be pretty self-explanatory.

Read more