Canonical Voices

Posts tagged with 'debug'

Kyle Nitzsche

Noting that this week I delivered three hour long sessions of interest to those interested in writing HTML5 apps for Ubuntu:

Read more

The current default D-BUS configuration (at least on Ubuntu) disallows monitoring method calls on the system D-BUS (dbus-monitor --system), which makes debugging rather cumbersome; this has worked years ago, but apparently got changed for security reasons. It took me a half an hour to figure out how to enable this for debugging, and as this has annoyingly little Google juice (I didn’t find any solution), let’s add some.

The trick seems to be to set a global policy to be able to eavesdrop any method call after the individual /etc/dbus-1/system.d/*.conf files applied their restrictions, for which there is already a convenient facility. Create a file /etc/dbus-1/system-local.conf with

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE busconfig PUBLIC
  "-//freedesktop//DTD D-BUS Bus Configuration 1.0//EN"

  <policy user="root">
    <!-- Allow everything to be sent -->
    <allow send_destination="*" eavesdrop="true"/>
    <!-- Allow everything to be received -->
    <allow eavesdrop="true"/>
    <allow send_type="method_call"/>

Then sudo dbus-monitor --system displays everything. Needless to say that you don’t want this file on any production system!

Does anyone know an easier way? My first naive stab was to run dbus-monitor as root, but that doesn’t make any difference at all.

Update: Turns out this is already described in a better way at Yay me for not finding that.. I updated above recipe to limit access to root, which is much better indeed.

Read more

Paul Wise poked me this morning about uploading fatrace (“file access trace”, see the original announcement for details) to Debian, thanks for the reminder!

So I filed an Intent To Package, and will upload it in a few days, unless some discussion evolves.

I also took the opportunity to do some modernization: The power-usage-report script now uses the current PowerTop 2.x instead of the old 1.13, uses Python 3 now, and includes the “process device activity” in the report. I released this as 0.5. The actual fatrace binary didn’t change its behaviour, it just got some code optimizations; thanks to Yann Droneaud for those.

Read more
Kyle Nitzsche

dbg versus dbgsym

Will the real debug package please stand up

While mucking around with stack traces, valgrind, and apport-valgrind, I stumbled across two types of debug symbol [1] packages:
  • dbg packages, for example: empathy-dbg
  • dbgsym packages, for example: empathy-dbgsym
Being rather new to this area, I was not sure which to use. But, I wanted to valgrind empathy, so I looked further.

The package Description fields [2] shed no light on which to use:

Description-en: GNOME multi-protocol chat and call client (debug symbols) Instant messaging program supporting text, voice, video, file transfers and inter-application communication over many different protocols, including: AIM, MSN, Google Talk (Jabber/XMPP), Facebook, Yahoo!, Salut, Gadu-Gadu, Groupwise, ICQ and QQ. This package contains the Empathy IM application and account manager.
Description: debug symbols for package empathy Instant messaging program supporting text, voice, video, file transfers and inter-application communication over many different protocols, including: AIM, MSN, Google Talk (Jabber/XMPP), Facebook, Yahoo!, Salut, Gadu-Gadu, Groupwise, ICQ and QQ. This package contains the Empathy IM application and account manager.
Turning to the internet, I found many others had asked the same question, and they usually got answers like this:

You can use either the -dbg or the -dbgsym packages, but you can't use both.
Still, no guidance as to which to use. And anyway, I wanted to understand what was going on at a deeper level. Investigating further, I found that dbg packages are hosted in the standard archive (, whereas dbgsym packages are hosted in a special archive:

It also turns out that the dbgsym packages on com do not appear to be strictly normal debian packages:
  • Normal debian packages end in ".deb" (for example: empathy-dbg_3.6.0.3-0ubuntu1_amd64.deb)
  • The packages on end in ".ddeb" (empathy-dbgsym_3.6.3-0ubuntu2_amd64.ddeb)
This ".ddeb" extension reminded me of ".udeb" packages, which I have encountered in the context of live-build installations. Are these dbgsym/ddeb packages also intended for some narrowly defined use case?

This warranted further poking around. Here are my findings.

dbg packages

dbg packages are the traditional debian method for providing debug symbols separately from the normally installed binary packages.

dbg packages are created by upstream packagers and flow naturally into the standard Ubuntu archives. They are normal debian packages (that end in ".deb").

There is usually only one binary dbg package per source package. That is, there are usually not separate dbg packages for individual binary packages, there's just one named after the source package, like empathy-dbg.

They are pretty easy to make these days. The normal way is described here.

So, thanks to a lot of upstream work, the Ubuntu standard archive already has lots of these dbg packages. You can see a list of them with something like this:

$ apt-cache search dbg | less

These dbg packages work as expected: if you install, for example, empathy-dbg, and then generate an empathy stack trace (with valgrind or gdb, for example):
  • The symbols it contains should be converted to useful function names (instead of being question marks)
  • The source file names should be displayed (instead of paths to shared object (.so) and other binary files)
So let's take a moment to appreciate the work upstream maintainers have done!

But, not every package in Ubuntu has a dbg package. And, even those that do may have created them in different ways, and possibly with different naming conventions, which makes them difficult (or impossible) to find mechanistically.

This is where dbgsym packages come in.

dbgsym packages

Consider the Ubuntu archive: thousands of binary packages (more than 7000 in the Ubuntu 12.10 Quantal main archive component alone).

That's a lot of different people in teams maintaining a lot of packages.

Inevitably, some packages that arguably should provide binary debug symbol packages do not. Or they do, but in a slightly different way, for example, the package name may not follow the standard pattern.

This wonderful diversity adds up to a concrete problem: the complete set of debug symbols required for debugging and creating useful stack traces of any arbitrary packaged C executable (ELF file) are not reliably present in standard Ubuntu archives.

Suppose I need to check something for memory leaks using apport-valgrind or debug it with gdb and I hit one of these missing bits?

Automation \o/

Fortunately, there is a systematic (and automatic) solution (which is almost fully working).

The idea is to automate the generation of debug symbols at package build time (whether or not the package maintainers took steps to create dbg packages). For each binary package that installs an executable for which symbols may be helpful, automatically create a new BINARYPACKAGE-dbgysm package, and let it install the appropriate set of debug symbol binaries. Make it conflict with the dbg package to avoid file collisions. Then, publish them in

That's two steps:
  1. Creating the dbgsym binary packages (this part seems to be fully implemented)
  2. Publishing them to (this part may not be fully implemented)

Binary package oriented

The dbgsym approach has a finer granularity than the usual dbg approach: it is binary package oriented. That is, a dbgysm package is created for each appropriate binary package (each that installs an ELF), whereas with the traditional, manual dbg approach, there is usually one dbg package created for all binary packages.

So there are often many dbgsym packages per source package with this approach, which results in many more dbgsym packages than dbg packages, a good thing: you only need to download and install the parts you want.

Conflicts - a closer look

These automatically generated binary dbgsym packages install some of the same files [3] as the corresponding dbg package, by design. But, without additional packaging steps, this would amount to a packaging error (since no single file may be installed by two binary packages without special steps to resolve this collision).

This is automatically handled by setting the dbgsym package to conflict with the dbg package(s). (Conflicting packages cannot both be fully installed at the same time.) This explains why I found so many statements saying that "dbg and dbgsym packages cannot both be installed at the same time." It is literally not possible due to these intentional conflicts settings.

How dbgsym packages are created

dbgsym packages are automatically created during a normal source package binary build by the pkg-create-dbgsym package, if it is installed. That is, if you install pkg-create-dbgsym, whenever a debian source package is built (for example with debuild in the source tree), pkg-create-dbgsym scripts create the dbgysm binary packages and set them to conflict as appropriate. (Thanks to Martin Pitt and other debian developers.) These steps are in addition to the normal activities of a build, so you also generate non-debug binary packages.

For example, after installing pkg-create-dbgsym, I built the empathy source package, and here are the binary packages that were created [4]. Lots of dbgsym ddebs!

ddeb: Huh? What?

These dbgsym binary package files end in ".ddeb", not ".deb", so what's up with that?

As far as I can tell, this is merely a convenience for archive maintenance. That is: a .ddeb is exactly the same as a .deb, except that its extension is different. This serves the purpose of allowing archive management tools to find ddeb binary packages and treat them differently, in this case:
  • Posting them on 
  • Not posting them on

Enough already: which to use?

So, after this exploration, I vote for dbgsym packages over dbg packages for the following reasons
  • The trend is towards automatic generation of dbgsym packages and automatic posting of them to separate ddebs archive like
  • Such ddeb archives should contain more debug symbol packages than the traditional dbg packages because they are automatically created for all appropriate packages (instead of relying on maintainers to add a dbg package) so you get more
  • Such ddeb archives certainly should contain equivalents for the existing dbg packages as well, so you lose nothing
  • If you want, you can just get the debug symbols you need, since dbgsym packages are more granular 
This all means that if you use dbgsym packages, your stack traces and debugging experiences are more likely to provide you the data you need to get your work done, with less bloat.

Also, manually created dbg packages may disappear over time. Why? No need for package maintainers to think about debug packages. And the normal archives get smaller as debug symbol packages disappear.

Using dbgsym

As noted, the dbgsym packages are published on Ubuntu systems do not know this archive by default.

You can easily add it to your "software sources". For example, the following line adds for Quantal, main to your /etc/apt/sources.list file:

deb quantal main

Then run sudo apt-get update, and you should be all set to go.

For example, check whether empathy-dbgsym is available for installation:

$ apt-cache policy empathy-dbgsym
  Version table:
 *** 0
        500 quantal/main amd64 Packages
        500 quantal/main amd64 Packages
        100 /var/lib/dpkg/status

There it is! (I happen to have this one installed.)

How about the dbgsym version of empathy's binary package account-plugin-aim?

$ apt-cache policy account-plugin-aim-dbgsym
  Installed: (none)
  Version table: 0
        500 quantal/main amd64 Packages

There it is!

A caveat

It appears that not all dbgsym binary packages that one might expect are actually present on

For example, when I built empathy on a Quantal amd64 system with pgk-create-dbgsym installed, the expected dbgsym binary packages were created, including, for example, one for the account-plugin-irc binary package, and here it is:


But, I don't see a dbgsym package for account-plugin-irc published in the Ubuntu ddebs quantal main archive. The same is true for some (but by no means all) other binary packages deriving from empathy, like account-plugin-icq-dbgsym

This would seem to be a hiccup in the publishing part of the automation, not the dbgsym generation part.


[1] Debug symbols fill in stack traces to make them more human readable by substituting C function names and C source code file name for question marks and library file names. See apport-valgrind for details.

[2] You can display a package Description for any package your system's apt knows about with apt-cache show PACKAGE, and look for the Description field.

[3] Conflicting with the dbg package(s), an example using account-plugin-aim as an example:

The empathy source package has the accounts-plugin-aim binary package. It installs the following file: /usr/lib/libaccount-plugin-1.0/providers/ (as shown by dpkg -S  /usr/lib/libaccount-plugin-1.0/providers/

The empathy-dbg package installs a debug symbol version of this file in a debug directory: /usr/lib/debug/usr/lib/libaccount-plugin-1.0/providers/ There is no conflict yet, since the paths are different.

The automatically generated dbgsym packages for empathy include account-plugin-aim-dbgsym. This package installs the same debug .so file as we saw for the empathy-dbg package:  /usr/lib/debug/usr/lib/libaccount-plugin-1.0/providers/

That's a conflict. Two packages (empathy-dbg and account-plugin-aim-dbgsym) install the same file.

This is expected and handled by making the dbgysm binary packages conflict with the dbg package (if any), which it does:

$ apt-cache show account-plugin-aim-dbgsym | grep Conflicts
Conflicts: empathy-dbg 

[4] With pkg-create-dbgsym installed, building empathy creates all these debs and dbgsym/ddebs:


Read more

Part of our efforts to reduce power consumption in Ubuntu is to provide an easy tool to hunt down which programs and devices are to blame for inordinate power consumption. powertop’s interactive mode is pretty good for this if you are sitting in a train and want to tweak some knobs to max out battery life, but we need something more reproducible and noninteractive for developers who want to file proper bug reports.

So I wrote a little script power-usage-report which calls fatrace for measuring file access activity from programs, and powertop-1.13 to measure process and device wakeups, clean up and sort their ouput, and generate a report which is appropriate to attach to bug reports, send around, put into Jenkins for measuring daily progress, etc. It is now part of fatrace version 0.4, so today’s Precise upgrades will have it.

The output has several sections for disk access (which prevent the disk from spinning down), wakeups (causing CPU power usage), and device activity. Disk/wakeups are sorted in descending order by process:

$ sudo power-usage-report
Measurement will begin in 5 seconds. Please make sure that the
computer is idle, i. e. do not press keys, start or operate programs, and that
programs are not busy with active tasks other than the one you want to examine.
Starting measurement for 60 seconds...
Measurement complete. Generating report...
======= unity-panel-ser: 5 file access events ======
/usr/share/zoneinfo/UTC: 1 reads
/usr/share/zoneinfo/posix/Europe/Berlin: 1 reads
/etc/localtime: 3 reads

======= gnome-settings-: 1 file access events ======
/etc/fstab: 1 reads

======= telepathy-gabbl: 1 file access events ======
/home/martin/.cache/wocky/caps/caps-cache.db: 1 reads

====== Wakeups ======
  30,9% ( 52,0)   compiz
  16,3% ( 27,4)   [iwlwifi] 
  12,5% ( 21,0)   [i915] 
   3,7% (  6,3)   [ahci] 
   2,3% (  3,9)   swapper/3
   1,2% (  2,0)   gvfs-afc-volume

====== Devices ======
An audio device is active 100,0% of the time:
hwC0D0 Conexant CX20585 

Recent USB suspend statistics
Active  Device name
100,0%	USB device 1- : USB Mouse (A4Tech)
100,0%	/sys/bus/usb/devices/1-
100,0%	USB device 1-1.5.4 : Kinesis Keyboard Hub (PI Engineering)
  0,0%	USB device 1-1.5.2 : USB2.0 Hub Controller (NEC Corporation)


You can redirect output to a file, of course. The top header (“Starting measurement..” etc.) will go to stderr and thus not be part of the redirected output.

Read more

Part of our efforts to reduce power consumption is to identify processes which keep waking up the disk even when the computer is idle. This already resulted in a few bug reports (and some fixes, too), but we only really just began with this.

Unfortunately there is no really good tool to trace file access events system-wide. powertop claims to, but its output is both very incomplete, and also wrong (e. g. it claims that read accesses are writes). strace gives you everything you do and don’t want to know about what’s going on, but is per-process, and attaching strace to all running and new processes is cumbersome. blktrace is system-wide, but operates at a way too low level for this task: its output has nothing to do any more with files or even inodes, just raw block numbers which are impossible to convert back to an inode and file path.

So I created a little tool called fatrace (“file access trace”, not “fat race” :-) ) which uses fanotify, a couple of /proc lookups and some glue to provide this. By default it monitors the whole system, i. e. all mounts (except the virtual ones like /proc, tmpfs, etc.), but you can also tell it to just consider the mount of the current directory. You can write the log into a file (stdout by default), and run it for a specified number of seconds. Optional time stamps and PID filters are also provided.

$ sudo fatrace
rsyslogd(967): W /var/log/auth.log
notify-osd(2264): O /usr/share/pixmaps/weechat.xpm
compiz(2001): R device 8:2 inode 658203

It shows the process name and pid, the event type (Rread, Write, Open, or Close), and the path. Sometimes its’ not possible to determine a path (usually because it’s a temporary file which already got deleted, and I suspect mmaps as well), in that case it shows the device and inode number; such programs then need closer inspection with strace.

If you run this in gnome-terminal, there is an annoying feedback loop, as gnome-terminal causes a disk access with each output line, which then causes another output line, ad infinitum. To fix this, you can either redirect output to a file (-o /tmp/trace) or ignore the PID of gnome-terminal (-p `pidof gnome-terminal`).

So to investigate which programs are keeping your disk spinning, run something like

  $ sudo fatrace -o /tmp/trace -s 60

and then do nothing until it finishes.

My next task will be to write an integration program which calls fatrace and powertop, and creates a nice little report out of that raw data, sorted by number of accesses and process name, and all that. But it might already help some folks as it is right now.

The code lives in bzr branch lp:fatrace (web view), you can just run make and sudo ./fatrace. I also uploaded a package to Ubuntu Precise, but it still needs to go through the NEW queue. I also made a 0.1 release, so you can just grab the release tarball if you prefer. Have a look at the manpage and --help, it should be pretty self-explanatory.

Read more