Canonical Voices

Posts tagged with 'tools'

Colin Ian King

Counting code size with SLOCCount

David A. Wheeler's SLOCCount is a useful tool for counting lines of code in a software project.  It is simple to use, just provide it with the path to the source code and let it grind through all the source files.  The resulting output is a break down of code line count for each type of source based on the programming language.

SLOCCount also estimates development time in person-years as well as the number of developers and the cost to develop.  One can override the defaults and specify parameters such as costs per person, overhead and effort to make it match to your development model.

Of course, like all tools that produce metrics it can be abused, for example using it as a meaningless metric of programmer productivity.  Counting lines of code does not really measure project complexity, a vexing bug that took 2 days to figure out and resulted in a 1 line fix is obviously more expensive than a poorly written 500 line function that introduces a no new noticeable functionality.   As a rule of thumb, SLOCCount is a useful tool to get an idea of the size of a project and some idea of the cost to develop it.   There are of course more complex ways to examine project source code, such as cyclomatic complexity metrics, and there are specific tools such as Panopticode that do this.

As a small exercise, I gave SLOCCount the task of counting the lines of code in the Linux kernel from version 2.6.12 to 3.6 and used the default settings to produce an estimated cost to develop each version.


It is interesting to see that the rate of code being added seemed to increase around the 2.6.28 release.   So what about the estimated cost to develop?..


This is of course pure conjecture.  The total lines of code does not consider the code of some patches that remove code and assumes that the cost is directly related to lines of code.  Also, code complexity makes some lines of code far more expensive to develop than others.   It is interesting to see that each release is adding an average of 184,000 lines of code per release which SLOCCount estimates to cost about $8.14 million dollars or ~44.24 dollars per line of code; not sure how realistic that really is.

Anyhow, SLOCCount is easy to use and provides some very useful rule-of-thumb analysis on project size and costs.

Read more
Matt Fischer

If you’re rolling your own Ubuntu edition, usually a localized one, then you typically need to configure a bunch of settings. Previously this was done by creating a config package and overwriting settings files manually. This was a pain and also required knowledge of which settings to modify and where and how they were stored. Ubuntu-defaults-builder solves most of these issues for anyone rolling their own Ubuntu by abstracting the way settings are saved for you, you edit a simple text file and let the tools generate the settings you need.

Here’s How it Works

After installing ubuntu-defaults-builder, you create your config area by running ubuntu-defaults-template [package-name]. Package-name should be something like ubuntu-defaults-germany or ubuntu-defaults-mydistro. This creates a directory named [package-name]. Inside that directory are simple config files that you can modify:

drwxr-xr-x 12 1000 1000 4096 Mar 23 21:08 .
drwxrwxr-x 4 1000 1000 4096 Mar 23 21:05 ..
drwxr-xr-x 2 1000 1000 4096 Mar 25 04:11 debian
-rw-r--r-- 1 1000 1000 799 Mar 22 21:38 depends.txt
drwxr-xr-x 2 1000 1000 4096 Mar 23 02:34 desktop
drwxr-xr-x 2 1000 1000 4096 Mar 23 02:44 hooks
drwxr-xr-x 2 1000 1000 4096 Mar 22 21:38 i18n
drwxr-xr-x 4 1000 1000 4096 Mar 22 21:41 icons
drwxr-xr-x 2 1000 1000 4096 Mar 22 21:41 multimedia
-rw-r--r-- 1 1000 1000 871 Mar 22 21:38 recommends.txt
drwxr-xr-x 2 1000 1000 4096 Mar 23 20:53 unity
drwxr-xr-x 2 1000 1000 4096 Mar 22 21:40 webbrowser

The files are not “real” conf files, but are used by ubuntu-defaults-builder to create gconf and gsettings overrides. Most of the conf files are simple and self-documenting. Look at /unity/launchers.txt for example, in that file you specify what default launchers you want in Unity.

root@caprica:~/Projects/# cat ubuntu-defaults-fred/unity/launchers.txt
@clear
nautilus-home
firefox

Now, just modify all these files how you would like them. Finally you can build your defaults package with dpkg-buildpackage. After you do that, take a look at the .deb with dpkg -c and you’ll see that your settings have become overrides for anything that uses gconf or dconf/gsettings:

cat /usr/share/glib-2.0/schemas/ubuntu-defaults-fred.gschema.override
[com.canonical.Unity.Launcher]
favorites=['nautilus-home.desktop', 'firefox.desktop']

What Can You Configure?

ubuntu-defaults-builder, as of 0.27, lets you modify the following settings:

  • depends.txt – a list of packages to make dependencies of the defaults package, this is helpful when building the image.
  • recommends.txt – a list of packages to make recommends of the defaults package, this is helpful when building the image.
  • desktop/ directory – set the background, default MIME handlers, default user session, and adding custom desktop files.
  • hooks/chroot – extra hooks to be run during image build (advanced)
  • i18n/ directory – set default keyboard layout, language, and which language packs to include in the image
  • icons/ directory – where to place custom icons, for general use or for any custom .desktop files
  • multimedia/radiostations.txt – customize the radio stations list for rhythmbox and banshee
  • unity/launchers.txt – set the default list of launchers
  • webbrowser/ directory – append bookmarks to the bookmarks menu or toolbar, set the default search engine or startpage. Currently only Firefox is supported.

Testing

At this point you probably want to test it in a VM. However, due to the way that dconf settings work, the simplest method to ensure your changes will be seen is to test like this:

1) Copy the defaults package to the VM.
2) Install the defaults package
3) Create a new user
4) Logout
5) Login as the new user. This will ensure that all the default settings take precedence.

You could also use your VM’s snapshot model and save yourself from ending up with 50 user accounts if you do a lot of testing.

Once your defaults package is good, you can build an image with it using the command ubuntu-defaults-image. This tool allows you to specify a locale, add packages or sources, etc and will generate an install image for you to do final testing on.

Limitations

Ubuntu-defaults-builder is a great tool, but it has limitations. First and foremost is that you cannot change everything you might want to using it. Unless you’re going to write a patch for ubuntu-defaults-builder, you are limited to what the tool knows about. The tool has specialized routines for almost every setting it knows how to to. If you want to set other settings, for example, some advanced lightdm.conf settings (such as disabling guest or hiding the user list), you will have to override it using the old way of dropping in a new lightdm.conf file (or using /usr/lib/lightdm/lightdm-set-defaults if that will do what you want). For settings stored in gsettings/dconf you will have to write your own gsettings override files and use the override_dh_installgsettings rule in debian/rules.

When you are making your own overrides, be careful that you’re not interacting with the overrides that are built by ubuntu-defaults-builder. For example, in my case, I get two overrides, the one starting with 50 is from the dh_installgsettings step during the build:

-rw-r–r– 1 root root 68 Mar 23 20:18 50_ubuntu-defaults-fred.gschema.override
-rw-r–r– 1 root root 146 Mar 23 21:06 ubuntu-defaults-fred.gschema.override

I do not know the order the two files will be handled in, so I just try not to let them overlap settings. The easiest way to do this is to use ubuntu-defaults-builder’s own mechanisms whenever possible and then to double-check your output .deb file.

More Info

The spec for ubuntu-defaults-builder, which has additional details on what it does, how, and why, is available here.

Read more
Colin Ian King

Firmware Test Suite Live (fwts-live) is a USB live image that will automatically boot and run the Firmware Test Suite (fwts) - it will run on legacy BIOS and also UEFI firmware (x86_64) bit systems.

fwts-live will run a range of fwts tests and store the results on the USB stick - these can be reviewed while running fwts-live or at a later time on another computer if required.

To install fwts-live on to a USB first download either a 32 or 64 bit image from http://odm.ubuntu.com/fwts-live/ and then uncompress the image using:

 bunzip2 fwts-live-*.img.bz2  

Next insert a USB stick into your machine and unmount it. Now one has to copy the fwts-live image to the USB stick - one can find the USB device using:

 dmesg | tail -10 | grep Attached  
 [ 2525.654620] sd 6:0:0:0: [sdb] Attached SCSI removable disk  

..so the above example it is /dev/sdb, and copy using:

 sudo dd if=fwts-live-oneiric-*.img of=/dev/sdb  
 sync  

..and then remove the USB stick.

To run, insert the USB stick into the machine you want to test and then boot the machine.  This will start up fwts-live and then you will be shown a set of options - to either run all the fwts batch tests, to select individual tests to run, or abort testing and shutdown.


If you chose to run all the fwts batch tests then fwts will automatically run through a series of tests which will take a few minutes to complete:


and when complete one can chose to view the results log:


if "Yes" is selected then one can view the results. The cursor up/down and page up/down keys can be used to navigate the results log file.  When you have completed viewing the results log, fwts-live will inform you where the results have been saved on the USB stick (so that one can review them later by plugging the USB stick into a different machine).


A full user guide to fwts-live is available at: https://wiki.ubuntu.com/HardwareEnablementTeam/Documentation/FirmwareTestSuiteLive

To help interpret any errors or warnings found by fwts we recommend visiting  fwts reference guide - this is has comprehensive description of each test and detailed explanations of warnings and error messages.

Below is a demo of fwts-live running inside QEMU:

 
 
Kudos to Chris Van Hoof for producing fwts-live

Read more
Colin Ian King

Monitoring /proc/timer_stats

The /proc/timer_stats interface allows one to check on timer usage in a Linux system and hence detect any misuse of timers that can cause excessive wake up events (and also waste power).  /proc/timer_stats reports the process id (pid) of a task that initialised the timer, the name of the task, the name of the function that initialised the timer and the name of the timer callback function.  To enable timer sampling, write "1\n" to /proc/timer_stats and to disable write "0\n".

While this interface is simple to use, collecting multiple samples over a long period of time to monitor overall system behaviour takes a little more effort.   To help with this, I've written a very simple tool called eventstat that calculates the rate of events per second and can dump the data in a .csv (comma separated values) format for importing into a spreadsheet such as LibreOffice for further analysis (such as graphing).

In its basic form, eventstat will run ad infinitum and can be halted by control-C. One can also specify the sample period and number of samples to gather, for example:

 sudo eventstat 10 60  

.. this gathers samples every 10 seconds for 60 samples (which equates to 10 minutes).

The -t option specifies an events/second threshold to discard events less than this threshold, for example:  sudo cpustat -t 10 will show events running at 10Hz or higher.

To dump the samples into a .csv file, use the -r option followed by the name of the .csv file.  If you just want to collect just the samples into a .csv file and not see the statistics during the run, use also the -q option, e.g.

 sudo eventstat -q -r event-report.csv


With eventstat you can quickly identify rouge processes that cause a high frequency of wake ups.   Arguably one can do this with tools such as PowerTop, but eventstat was written to allow one to collect the event statistics over a very long period of time and then help to analyse or graph the data in tools such as Libre Office spreadsheet.


The source is available in the following git repository:  git://kernel.ubuntu.com/cking/eventstat.git and in my power management tools PPA: https://launchpad.net/~colin-king/+archive/powermanagement

In an ideal world, application developers should check their code with tools like eventstat or PowerTop to ensure that the application is not misbehaving and causing excessive wake ups especially because abuse of timers could be happening in the supporting libraries that applications may be using.

Read more
Colin Ian King

UEFI  Compatibility Support Module (CSM) provides compatibility support for traditional legacy BIOS.  This allows allows the booting an operating system that requires a traditional option ROM support, such as BIOS Int 10h video calls.

While looking at boot and runtime misbehaviour on UEFI systems I would like to know if CSM is enabled or not, but the question is how does one detect CSM support?   Well, making the assumption that CSM is generally enabled to support Int 10h video calls, we look for any video option ROMs and see if the real mode Int 10h vector is set to jump to a handler in one of the ROMs.  

Option ROMs are found in the region 0xc0000 to 0xe0000 and normally the video option ROM is found at 0xc0000.  Option ROMs are found on 512 byte boundaries with a header bytes containing 0x55, 0xaa and ROM length (divided by 512) so we just mmap in 0xc0000..xe0000 and then scan the memory for headers to locate option ROM images.  

My assumption for CSM being enabled is that Int 10h vectors into one of these option ROMs, and we can assume it is a video option ROM if it contains the string "VGA" somewhere in the ROM image.  Yes, it is a hack, but it seems to work on the range of UEFI enabled systems I've so far used.

For reference, I've put the code in my debug-code git repository and available for anyone to use.


Read more
Colin Ian King

Some problems are a little challenging to debug and require sometimes a bit of lateral thinking to solve.   One particular issue is when suspend/resume locks up and one has no idea where or why because the console has is suspended and any debug messages just don't appear.

In the past I've had to use techniques like flashing keyboard LEDs, making the PC speaker beep or even forcible rebooting the machine at known points to be able to get some idea of roughly where a hang has occurred.   This is fine, but it is tedious since we can only emit a few bits of state per iteration.   Saving state is difficult since when a machine locks up one has to reboot it and one looses debug state.   One technique is to squirrel away debug state in the real time clock (RTC) which allows one to store twenty or so bits of state, which is still quite tough going.

One project I've been working on is to use the power of system tap to instrument the entire suspend/resume code paths - every time a function is entered a hash of the name is generated and stored in the RTC.  If the machine hangs, one can then grab this hash out of the RTC can compare this to the known function names in /proc/kallsyms, and hopefully this will give some idea of where we got to before the machine hung.

However, what would be really useful is the ability to print out more debug state during suspend/resume in real time.   Normally I approach this by using a USB/serial cable and capturing console messages via this mechanism.  However, once USB is suspended, this provides no more information.

One solution I'm now using is with Kamal Mostafa's minimodem.  This wonderful tool is an implementation of a software modem and can send and receive data by emulating a Bell-type or RTTY FSK modem.  It allows me to transmit characters at 110 to 300 baud over a standard PC speaker and reliably receive them on a host machine.  If the wind is in the right direction, one can transmit at higher speeds with an audio cable plugged in the headphone jack of the transmitter and into the microphone socket on the receiver if hardware allows.

The 8254 Programmable Interval-timer on a PC can be used to generate a square wave at a predefined frequency and can be connected to the PC speaker to emit a beep.  Sending data using the speaker to minimodem is a case of sending a 500ms leader tone, then emitting characters.  Each character has a 1 baud space tone, followed by 8 bits (least significant bit first) with a zero being a 1 baud space tone and a 1 being represented by a 1 baud mark tone, and the a trailing bunch of stop bits.

So using a prototype driver written by Kamal, I tweaked the code and put it into my suspend/resume SystemTap script and now I can dump out messages over the PC speaker and decode them using minimodem.  300 baud may not be speedy, but I am able to now instrument and trace through the entire suspend/resume path.

The SystemTap scripts are "work-in-progress" (i.e. if it breaks you keep the pieces), but can be found in my pmdebug git repo git://kernel.ubuntu.com/cking/pmdebug.git.  The README file gives a quick run down of how to use this script and I have written up a full set of instructions.

The caveat to this is that one requires a PC where one can beep the PC speaker using the PIT.  Lots of modern machines seem to either have this disabled, or the volume somehow under the control of the Intel HDA audio driver.  Anyhow, kudos to Kamal for providing minimodem and giving me the prototype kernel driver to allow me to plug this into a SystemTap scrip.


Read more
Colin Ian King

Today my colleague Chris Van Hoof pointed me to a Gource visualization of the work I've been doing on the Firmware Test Suite.  Gource animates the software development sources as a tree with the root in the centre of the display and directories as branches and source files as leaves.


Static pictures do this no justice. I've uploaded an mp4 video of the entire software development history of fwts so you can see Gource in action.

To generate the video, the following incantation was used:

 gource -s 0.03 --auto-skip-seconds 0.1 --file-idle-time 500 \  
 --multi-sampling -1280x720 --stop-at-end \  
 --output-ppm-stream - | ffmpeg -y -r 24 \  
 -f image2pipe -vcodec ppm -i - -b 2048K fwts.mp4  

..kudos to Chris for this rune.


Read more
Colin Ian King

Dumping UEFI variables

UEFI variables in Linux can be found in /sys/firmware/efi/vars on UEFI firmware based machine, however, the raw variable data is in a binary format and hence not in a human readable form.   The Ubuntu Natty firmware test suite contains the uefidump tool to extract and decode the binary data into a more human readable form.

To run, use:

sudo fwts uefidump -


and you will see something similar to the following:

Name: AuthVarKeyDatabase.
  GUID: aaf32c78-947b-439a-a180-2e144ec37792
  Attr: 0x17 (NonVolatile,BootServ,RunTime).
  Size: 1 bytes of data.
  Data: 0000: 00                                               .

Name: Boot0000.
  GUID: 8be4df61-93ca-11d2-aa0d-00e098032b8c
  Attr: 0x7 (NonVolatile,BootServ,RunTime).
  Active: Yes
  Info: Primary Master Harddisk
  Path: \BIOS(2,0,Primary Master Harddisk).

Name: Boot0001.
  GUID: 8be4df61-93ca-11d2-aa0d-00e098032b8c
  Attr: 0x7 (NonVolatile,BootServ,RunTime).
  Active: Yes
  Info: EFI Internal Shell
  Path: \Unknown-MEDIA-DEV-PATH(0x7)\Unknown-MEDIA-DEV-PATH(0x6).

Name: Boot0003.
  GUID: 8be4df61-93ca-11d2-aa0d-00e098032b8c
  Attr: 0x7 (NonVolatile,BootServ,RunTime).
  Active: Yes
  Info: ubuntu
  Path: \HARDDRIVE(1,22,9897,0f52a6e132775546,ab,f6)\FILE('\EFI\ubuntu\grubx64.efi').

Name: Boot0004.
  GUID: 8be4df61-93ca-11d2-aa0d-00e098032b8c
  Attr: 0x7 (NonVolatile,BootServ,RunTime).
  Active: Yes
  Info: EFI DVD/CDROM
  Path: \ACPI(0xa0341d0,0x0)\PCI(0x2,0x1f)\ATAPI(0x0,0x1,0x0).

Name: BootOptionSupport.
  GUID: 8be4df61-93ca-11d2-aa0d-00e098032b8c
  Attr: 0x6 (BootServ,RunTime).
  BootOptionSupport: 0x0303.

Name: BootOrder.
  GUID: 8be4df61-93ca-11d2-aa0d-00e098032b8c
  Attr: 0x7 (NonVolatile,BootServ,RunTime).
  Boot Order: 0x0003,0x0000,0x0001,0x0004,0x0005,0x0006.

Name: ConIn.
  GUID: 8be4df61-93ca-11d2-aa0d-00e098032b8c
  Attr: 0x7 (NonVolatile,BootServ,RunTime).
  Device Path: \ACPI(0xa0341d0,0x0)\PCI(0x0,0x1f)\ACPI(0x50141d0,0x0)\UART(115200 baud,8,1,1)\VENDOR(11d2f9be-0c9a-9000-273f-c14d7f010400)\USBCLASS(0xffff,0xffff,0x3,0x1,0x1).

Name: ConInDev.
  GUID: 8be4df61-93ca-11d2-aa0d-00e098032b8c
  Attr: 0x6 (BootServ,RunTime).
  Device Path: \ACPI(0xa0341d0,0x0)\PCI(0x0,0x1f)\ACPI(0x50141d0,0x0)\UART(115200 baud,8,1,1)\VENDOR(11d2f9be-0c9a-9000-273f-c14d7fff0400).

Name: Setup.
  GUID: 038bcef0-21e2-49d1-a47c-b7257296b980
  Attr: 0x7 (NonVolatile,BootServ,RunTime).
  Size: 114 bytes of data.
  Data: 0000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  Data: 0010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  Data: 0020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  Data: 0030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  Data: 0040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  Data: 0050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  Data: 0060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  Data: 0070: 01 00   
..

The tool will try to decode the binary data, however, if it cannot identify the variable type it will resort to doing a hex dump of the data instead.


Read more
Colin Ian King

The Firmware Test Suite (fwts) is still a relatively new tool and hence this cycle I've still been adding some features and fixing bugs.  I've been running fwts against large data sets to soak test the tool to catch a lot of stupid corner cases (e.g. broken ACPI tables). Also, I am focused on getting some better documentation written (this is still "work in progress").

New tests for the Oneiric 11.10 release are as follows:

mpcheck:
    Sanity check tables against the MultiProcessor Specification (MPS). For more information about MPS, see the wikipedia MPS page.

mpdump:
    Dump annotated MPS tables.

msr:
    Sanity check Model Specific Registers across all CPUs. Does some form of MSR default field sanity checking.

s3power:
    Very simple suspend (S3) power checks.  This puts the machine into suspend and attempts to measure the power consumed while suspended. Generally this test gives more accurate results the longer one suspends the machine.  Your mileage may vary on this test.

ebdadump:
     Hex dump of the Extended BIOS Data Area.

In addition to the above, the fwts "method" test is now expanded to evaluate and exercise over 90 ACPI objects and methods.

One can also join the fwts mailing list by going to the project page and subscribing.


Read more
Colin Ian King

The dwarves package contains a set of useful tools that use the DWARF information placed in the ELF binaries by the compiler. Utilities in the dwarves package include:

pahole: This will find alignment holes in structs and classes in languages such as C/C++.  With careful repacking one can achieve better cache hits.  I could have done with this when optimising some code a few years back...

codiff:  This is a diff like tool use to compare the effect a change in source code can create on the compiled code.

pfunct:  This displays information about functions, inlines, goto labels, function size, number of variables and much more.

pdwtags: A DWARF information pretty-printer.

pglobal:  Dump out global symbols.

prefcnt:  A DWARF tags usage count.

dtagnames: Will lists tag names.

So, using pglobal, I was able to quickly check which variables I had made global (or accidentally not made them static!) on some code that I was developing as follows:

pglobal -v progname

and the same for functions:

pglobal -f progname

Easy!  Obviously these tools only work if the DWARF information is not stripped out.

All in all, these are really useful tools to know about and will help me in producing better code in the future.


Read more
Colin Ian King

Fragmentation on ext2/ext3/ext4 filesystems

If you want know how fragmented a file is on an ext2/ext3/ext4 filesystem there are a couple of methods of finding out.  One method is to use hdparm, but one needs CAP_SYS_RAWIO capability to do so, hence run with sudo:

sudo hdparm --fibmap /boot/initrd.img-2.6.38-9-generic

/boot/initrd.img-2.6.38-9-generic:
 filesystem blocksize 4096, begins at LBA 24000512; assuming 512 byte sectors.
 byte_offset  begin_LBA    end_LBA    sectors
           0   35747840   35764223      16384
     8388608   39942144   39958527      16384
    16777216   40954664   40957423       2760

Alternatively, one can use the filefrag utility (part of the e2fsprogs package) for reporting the number of extents:

filefrag /boot/initrd.img-2.6.38-9-generic
/boot/initrd.img-2.6.38-9-generic: 3 extents found

..or more verbosely:

filefrag -v /boot/initrd.img-2.6.38-9-generic
Filesystem type is: ef53
File size of /boot/initrd.img-2.6.38-9-generic is 18189027 (4441 blocks, blocksize 4096)
 ext logical physical expected length flags
   0       0  1468416            2048
   1    2048  1992704  1470463   2048
   2    4096  2119269  1994751    345 eof
/boot/initrd.img-2.6.38-9-generic: 3 extents found

Well, that's useful. So, going one step further, how many free extents are available on the filesystem? Well, e2freefrag is the tool for this:

sudo e2freefrag /dev/sda1
Device: /dev/sda1
Blocksize: 4096 bytes
Total blocks: 2999808
Free blocks: 1669815 (55.7%)

Min. free extent: 4 KB
Max. free extent: 1370924 KB
Avg. free extent: 6160 KB
Num. free extent: 1084

HISTOGRAM OF FREE EXTENT SIZES:
Extent Size Range :  Free extents   Free Blocks  Percent
    4K...    8K-  :           450           450    0.03%
    8K...   16K-  :            97           227    0.01%
   16K...   32K-  :            99           527    0.03%
   32K...   64K-  :           134          1475    0.09%
   64K...  128K-  :            96          2193    0.13%
  128K...  256K-  :            50          2235    0.13%
  256K...  512K-  :            30          2643    0.16%
  512K... 1024K-  :            36          6523    0.39%
    1M...    2M-  :            36         13125    0.79%
    2M...    4M-  :            14          9197    0.55%
    4M...    8M-  :            15         18841    1.13%
    8M...   16M-  :             8         17515    1.05%
   16M...   32M-  :             7         38276    2.29%
   32M...   64M-  :             3         39409    2.36%
   64M...  128M-  :             3         67865    4.06%
  512M... 1024M-  :             4        802922   48.08%
    1G...    2G-  :             2        646392   38.71%

..thus reminding me to do some housekeeping and remove some junk from my file system... :-)


Read more
Colin Ian King

The Linux PCI core driver provides a useful (and probably overlooked) sysfs interface to read PCI ROM resources.  A PCI device that has a ROM resource will have a "rom" sysfs file associated with it, writing anything other than 0 to it will enable one to then read the ROM image from this file.

For example, on my laptop, to find PCI devices that have ROM images associated with them I used:

find /sys/devices -name "rom"
/sys/devices/pci0000:00/0000:00:02.0/rom

and this corresponds to my Integrated  Graphics Controller:

lspci | grep 02.0
00:02.0 VGA compatible controller: Intel Corporation Mobile GM965/GL960 Integrated Graphics Controller (primary) (rev 0c)

To dump the ROM I used:

echo 1 | sudo tee /sys/devices/pci0000\:00/0000\:00\:02.0/rom
sudo cat /sys/devices/pci0000\:00/0000\:00\:02.0/rom > vbios.rom

To disassemble this I used ndisasm:

sudo apt-get install nasm
ndisasm -k 0,3 vbios.rom | less

..and just use strings on the ROM image to dump out interesting text, e.g.

strings vbios.rom
000000000000
00IBM VGA Compatible BIOS.
PCIR
(00`
*@0p
H?@0b
..


..and then used a tool like bvi to edit the ROM.


Read more
Colin Ian King

Editing binaries using bvi

I don't want to start a vi vs emacs holy war, but I have found a wonderful binary editor based on the vi command set.  The bvi editor is great for tweaking binary files - one can tab between the hex and ASCII representations of the binary data to re-edit the data.

To install use:

sudo apt-get install bvi

As well as basic vi commands, bvi contains bit-level operations to and/or/xor/not/shift bits too.   One has to use ':set memmove' to insert/delete data as this is not enabled by default.

Anyhow, install, consult the man page and get editing binary files!


Read more
Colin Ian King

SystemTap is a very useful and powerful tool that enables one to insert kernel debug into a running kernel.  Today I wanted to inspect the I/O read/write operations occurring when running some ACPI AML, so it was a case of hacking up a few lines of system tap to dump out the relevant state (e.g. which port being accessed, width of the I/O operation in bits and value being written or read).

So instead of spinning a bespoke kernel with a few lines of debug in, I use SystemTap to quickly write a re-usable script.  Simple and easy!

I've put the SystemTap script in my git repository for any who are interested.


Read more
Colin Ian King

Normally when I see a problem to do with CPU, I/O, network or memory resource hogging I turn to my trusty tools such as vmstat, iostate or ifstat to check system behaviour.

I stumbled upon dstat today, which boasts to be "a versatile replacement for vmstat, iostat and ifstat.".  So let's see what it can do. First, install with:

sudo apt-get install dstat

Running dstat with no arguments displays CPU stats (user,system,idle,wait,hardware interrupts, software interrupts), Disk I/O stats (read/write), Network stats (receive,send), Paging stats (in/out) and System stats (interrupts, context switches).   Not bad at all. 

Dstat even highlights in colour the active values, so you don't miss relevant deltas in the statistics being churned out by the tool.  Colours can be disabled with the --nocolor option.

But there is more. There are a plethora of options to show various system activities, such as:

-l       load average
-m       memory stats (used, buffers, cache, free)
-p       process stats (runnable, uninterruptible, new)
--aio    aio stats (asynchronous I/O)
--fs     filesystem stats (open files, inodes)
--ipc    ipc stats (message queue, semaphores, shared memory)
--socket socket stats (total, tcp, udp, raw, ip-fragments)
--tcp    TCP stats (listen, established, syn, time_wait, close)
--udp    UDP stats (listen, active)
--vm          virtual memory stats (hard pagefaults, soft pagefaults, allocated, free)

..to name but a few.  As it is, this already is more functional that vmstat, iostat and ifstat.   But that's not all - dstat is extensible by the use of dstat plugins and the packaged version of dstat comes shipped with a few already, for example:

--battery          battery capacity
--cpufreq          CPU frequency
--top-bio-adv  top block I/O activity by process
--top-cpu-adv  most expensive CPU process
--top-io      most expensive I/O process
--top-latency  process with highest latency

..and many more besides - consult the manual page for dstat to see more.

So, dstat really is a Swiss army knife - a useful tool to get to know and use whenever you need to quickly spot misbehaving processes or devices.


Read more
Colin Ian King

The hwloc (hardware locality) package contains the useful tool lstopo. To install use:

sudo apt-get install hwloc

By default, lstopo will display a logical view of the system caches and CPU cores, for example:


To get a non-graphical output use:

lstopo -

Machine (1820MB) + Socket #0 + L3 #0 (3072KB)
  L2 #0 (256KB) + L1 #0 (32KB) + Core #0
    PU #0 (phys=0)
    PU #1 (phys=2)
  L2 #1 (256KB) + L1 #1 (32KB) + Core #1
    PU #2 (phys=1)
    PU #3 (phys=3)

lstopo is also able to output the toplogy image in a variety of formats (Xfig, PDF, Postscript, PNG, SVG and XML) by specifying the output filename and extension, e.g.

lstopo topology.pdf

For more information, consult the manual for hwloc and lstopo.


Read more
Colin Ian King

hardinfo - look at hardware information in a GUI

Leandro Pereira has created hardinfo, a useful user-friendly GTK+ utility to browse the hardware information on one's machine. It also includes a bunch of benchmarking tools too.

To install, simply use:

apt-get install hardinfo

and run with Applications->System Tools->System Profiler and Benchmark or run from the command line using:

hardinfo

The tool presents information from /proc and /sys in a very easy to user and clean GUI. The view is broken into four categories:

Computer: OS, kernel modules, filesystem, display, environment variables, users.
Devices: CPU, memory, PCI, USB, printers, battery, sensors, input, storage, DMI, resources.
Network: Interfaces, IP connections, routing tables, ARP table, DNS servers, statistics, shares.
Benchmarks: Blowfish, CryptoHash, Fibonacci, CPU N-Queens, CPU FFT, FPU Raytracing.

Below are some sample views of my machine using hardinfo:




Quite a handy tool.


Read more
Colin Ian King

Perf tool now in Ubuntu Lucid.

A few months ago I blogged about the perf tool and how useful it is to drill down into kernel and user space applications to analyse performance bottle necks. Well, thanks to Andy Whitcroft, the perf tool is now available in Ubuntu Lucid. Perf needs to be in lock-step with the kernel, which complicated the packaging of this tool.

To install, use:

sudo apt-get install linux-tools

and this installs the perf, perf-stat, perf-top, perf-record, perf-report and perf-list tools.

The author of perf, Ingo Molnar has written some basic instructions on driving perf in the tools/perf/Documentation/example.txt file. They should give one a feel of how to drive these tools.

For example, to examine google-chrome:

perf record google-chrome

... run a test and then exit, and then generate a summary of activity:


perf report

One can examine specific events in the system too. To get a list of available events use:

perf list

For example, to measure CPU cycles, number of instructions, context switches and kmallocs on google-chrome one uses:

perf stat -e cpu-cycles -e instructions -e context-switches -e kmem:kmalloc google-chrome

Performance counter stats for 'google-chrome':

15576472832 cycles # 0.000 M/sec
13466330277 instructions # 0.865 IPC
40791 context-switches # 0.000 M/sec
38602 kmem:kmalloc # 0.000 M/sec

18.062301718 seconds time elapsed

One more quick example, this time we record a call-graph (stack chain/backtrace) of the dd command:

perf record -g dd if=/dev/zero of=/dev/null bs=1M count=4096
and get the call graph using:

# Samples: 12584 # # Overhead Command Shared Object Symbol # ........ ............... ............................... ...... # 95.96% dd [kernel] [k] __clear_user | |--99.69%-- read_zero | vfs_read | sys_read | system_call_fastpath | __read --0.31%-- [...]
1.37% dd [kernel] [k] read_zero 0.87% dd [kernel] [k] _cond_resched | |--50.91%-- read_zero | vfs_read | sys_read | system_call_fastpath | __read
..etc.

I recommend reading Ingo's example.txt and playing around with this tool. It is very powerful and allows one to drill down and examine system performance right down to the instruction level.


Read more
Colin Ian King

Saving power on my HPMini (revisited on Lucid)

A while ago I blogged about power saving on my HPMini. Well, today I'm revisiting this using today's Lucid daily and I'm pleased to see that I can save a small amount of power with Lucid compared to Karmic. However, the difference is small - the inaccuracies of estimating power consumption using ACPI may be more significant.

My steps were as follows:

0. Run powertop for about 15 minutes on battery power and note down the recommended power saving tricks.

1. Blacklist Bluetooth, since I don't use this and it really sucks a load of power. To do so, add the following to /etc/modprobe.d/blacklist.conf

blacklist btusb

2. Enable HDA audio powersaving:

echo 1 > /sys/module/snd_hda_intel/parameters/power_save

3. Increase dirty page writeback time:

echo 1500 > /proc/sys/vm/dirty_writeback_centisecs

4. Disable the webcam driver (not sure exactly if this saves much power), add the following to /etc/modprobe.d/blacklist.conf

blacklist uvcvideo

5. Turn down the screen brightness (this saves 0.3 Watts)

6. Disable desktop effects to save some power used by compositing.

7. Disable cursor blinking on the gnome terminal to save 2 wakeups a second, using:

gconftool-2 --type string --set /apps/gnome-terminal/profiles/Default/cursor_blink_mode off

I managed to push the power consumption down to ~6.5 Watts which is an improvement of nearly 1 Watt from my tests on Karmic. Not bad, but I don't fully trust the data from the battery/ACPI and I really need to see how much extra battery life these tweaks give me when I'm using the machine on my travels.

On an idle machine I'm seeing > ~99.0% C4 residency, which is quite acceptable. Running powertop is always leads to insights to saving power, so kudos to Intel for this wonderful utility.


Read more
Colin Ian King

HDA Analyzer

The HDA Analyzer tool that allows one to look at the raw HD-audio control data in an easy to user GUI. The instructions on how to download and use this tool are described at the HDA Analyzer page - they are as follows:

1. Fetch:


2. Run:

python run.py

And browse...


This certainly takes the pain out of looking at the control information.


Read more