Canonical Voices

Colin Ian King

NumaTop: A NUMA system monitoring tool

NumaTop is a useful tool developed by Intel for monitoring runtime memory locality and analysis of processes on Non-Uniform Memory Access (NUMA) systems.  NumaTop can identify potential NUMA related performance bottlenecks and hence help one to re-balance memory/CPU allocations to maximise the potential of a NUMA system.

Initial "Top" like process view

One can select specific processes and drill down and characteristics such as memory latencies or call chains to see where code is hot.

Observing a specific process..
..and observing memory latencies
Observing per Node CPU and memory statistics
The tool uses perf to collect deeper system statistics and hence needs to be run with root privileges will only run on NUMA systems. I've recently packaged NumaTop and it is now available in Ubuntu Wily 15.10 and the source is available on github.

Read more
Colin Ian King

light-weight process stats with cpustat

A while ago I was working on identifying busy processes on small Ubuntu devices and required a tool that could look at per process stats (from /proc/$pid/stat) in a fast and efficient way with minimal overhead.   There are plenty of tools such as "top" and "atop" that can show per-process CPU utilisation stats, but most of these aren't useful on really slow low-power devices as they consume several tens of megacycles collecting and displaying the results.

I developed cpustat to be compact and efficient, as well as provide enough stats to allow me to easily identify CPU sucking processes.   To optimise the code, I used tools such as perf to identify code hotspots as well as valgrind's cachegrind to identify poorly designed cache inefficient data structures.

The majority of the savings were in the parsing of data from /proc - originally I used simple fscanf() style parsing; over several optimisation rounds I ended up with hand-crafted numeric and string scanning parsing that saved several hundred thousand cycles per iteration.

I also made some optimisations by tweaking the hash table sizes to match the input data more appropriately.  Also, by careful re-use of heap allocations, I was able to reduce malloc()/free() calls and save some heap management overhead.

Some very frequent string look-ups were replaced with hash lookups and frequently accessed data was duplicated rather than referenced indirectly to keep data local to reduce cache stalls and hence speed up data comparison lookup time.

The source has been statically checked by CoverityScan, cppcheck and also clang's scan-build to check for bugs introduced in the optimisation steps.

Example of cpustat
cpustat is now available in Ubuntu 15.10 Wily Werewolf.   Visit the cpustat project page for more details.

Read more
Colin Ian King

Tweaking the thermald configuration file

The Intel Thermal deamon (aka thermald) actively monitors thermal sensors and will modify cooling controls to try to keep the hardware cool.   By default, thermald will run in a "zero-configuration" mode and attempt to use the available CPU Digital Thermal Sensor(s) (DTS) to sense the temperature and use the P-state driver, Running Average Power Limit (RAPL), PowerClamp and cpufreq to control cooling.

Some systems may not work well in the default mode, perhaps the machine just runs too hot and one would like to tweak the settings to kick in passive or active cooling at a lower temperature than the default configuration. Thermald has a configuration file /etc/thermald/thermal-conf.xml that allows fine tuning of thermald. Essentially one declares the thermal sensors on the machine and a set of thermal zone controls that read these thermal sensors and inform thermald the policy to control cooling when specific temperature thresholds are crossed.

For an example, I've picked on an old Acer Aspire One (AMD C-60). Let's see the sensors for this machine:

find /sys/class/hwmon/* -exec echo -n "{}: " \; -exec cat {}/name \;
/sys/class/hwmon/hwmon0: radeon
/sys/class/hwmon/hwmon1: k10temp
one can use tools such as sensors (from the lm-sensors package) to get an idea of the high and critical trip points for these:
$ sudo apt-get install lm-sensors
$ sensors
Adapter: PCI adapter
temp1: +60.0°C (crit = +120.0°C, hyst = +90.0°C)

Adapter: PCI adapter
temp1: +60.5°C (high = +70.0°C)
(crit = +115.0°C, hyst = +107.5°C)

So, in this simple example, I will just use the CPU sensor k10temp (from /sys/class/hwmon/hwmon1) as my thermald CPU temperature sensor. Next, I need to define a policy on what to do when this sensor reaches a specific high temperature threshold. In this example, I want to trigger passive (non-fan) cooling by adjusting the CPU frequency using cpufreq and also the ACPI processor sysfs cooling controls when we reach 85 degrees C. I require thermald to control both cooling methods to run together in parallel with 60% of the influence to come from cpufreq and 40% from the ACPI processor cooling controls. My thermald config file for this is as follows:
<Name>Aspire One</Name>
<Type>cpu package</Type>
One can observe this working by starting thermald in verbose debug mode:
$ sudo thermald --no-daemon --loglevel=debug
it is worth exercising the machine (I use stress-ng --cpu 0) to ramp up the load and temperature to observe how thermald is working. Once one is happy with the results, one can then start thermald using:
$ sudo systemctl start thermald
More examples can be found in the thermald manual page:
$ man thermal-conf.xml 

Read more
Colin Ian King

static code analysis (revisited)

A while ago I was extolling the virtues of static analysis tools such as cppcheck, smatch and CoverityScan for C and C++ projects.  I've recently added to this armoury the clang analyser scan-build, which has been most helpful in finding even more obscure bugs that the previous three did not catch.

Using scan-build is very simple indeed, install clang and then in your source tree just build your project with scan-build, e.g. for a project built by make, use:

scan-build make
..and at the end of a build one will see a summary message:
scan-build make
scan-build: 366 bugs found.
scan-build: Run 'scan-view /tmp/scan-build-2015-09-08-094505-16657-1' 
to examine bug reports.
scan-build: The analyzer encountered problems on some source files.
scan-build: Preprocessed versions of these sources were deposited in 
scan-build: Please consider submitting a bug report using these files:

..and running scan-view will show the issues found.  For an example of the kind of results scan-build can find, I ran it against a systemd build (head commit 4df0514d299e349ce1d0649209155b9e83a23539). 

As one can see, scan-build is a powerful and easy to use open-source static analyser.  I heartily recommend using it on every C and C++ project.

Read more
Colin Ian King

Monitoring temperatures with psensor

While doing some thermal debugging this weekend I stumbled upon the rather useful temperature monitoring utility "Psensor".   I configured it to update stats every second and according to perf it was only using 0.02 CPU's worth of compute, so it seems relatively lightweight and shouldn't contribute to warming the machine up!

I like the min/max values being clearly shown and also the ability to change graph colours and toggle data on or off.  Quick, easy and effective.  Not sure why I haven't found this tool earlier, but I wish I had!

Read more
Colin Ian King

Identifying Suspend/Resume delays

The Intel SuspendResume project aims to help identify delays in suspend and resume.  After seeing it demonstrated by Len Brown (Intel) at this years Linux Plumbers conference I gave it a quick spin and was delighted to see how easy it is to use.

The project has some excellent "getting started" documentation describing how to configure a system and run the suspend resume analysis script which should be read before diving in too deep.

For the impatient, one can do try it out using the following:

git clone
cd suspendresume
sudo ./

..and manually resume once after the machine has completed a successful suspend.

This will create a directory containing dumps of the kernel log and ftrace output as well as an html web page that one can read into your favourite web browser to view the results.  One can zoom in/out of the web page to drill down and see where the delays are occurring, an example from the SuspendResume project page is shown below:

example webpage (from

It is a useful project, kudos to Intel for producing it.  I thoroughly recommend using it to identify the delays in suspend/resume.

Read more
Colin Ian King

The Canonical Hardware Enablement Team and myself are continuing the work to add more ACPI table tests to the Firmware Test Suite (fwts).  The latest 15.08.00 release added sanity checks for the following tables:

The release also added a test for the ACPI _CPC revision 2 control method and we updated the ACPICA core to version 20150717.

Our aim is to continue to add support for existing and new ACPI tables to make fwts a comprehensive firmware test tool.  For more information about fwts, please refer to the fwts jump start wiki page.

Read more
Colin Ian King

stress-ng adds more features

Since I last wrote about perf being added to stress-ng back in the end of May I have been busy in my spare time adding more features to stress-ng.

New stressors include:
  • ptrace - traces a child process performing many simple system calls
  • sigsuspend - sends SIGUSR1 signals to multiple children waiting on sigsuspend(2)
  • sigpending - checks if SIGUSR1 signals are pending on a process that alternatively masks and umasks this signal
  • mmapfork - rapidly spawn multiple child processes that try to allocate a chunk of free memory (and try to avoid swapping). Each process then uses  madvise(2) to hints before and after the memory is memset and then the child dies.
  • quota - exercise various quotactl(2) Q_GET* commands
  • sockpair - client/server socket I/O using socket pair and random sized I/O
  • getrandom - exercise the new getrandom(2) system call
  • numa -  migrates a memory mapped buffer and processes around NUMA modes, exercising migrate_pages(2), mbind(2) and move_pages(2).
  • wcs - exercises libc wide character string functions (thanks to Christian Ehrhardt for this contribution).
 ..and I have added some improvements too:
  • --yaml option to dump stats from --metrics, --perf, -tz into a YAML structured log.
  • made the --aggressive option more aggressive by forcing more CPU migrations and context switches.
I have also added a thermal zone stats gathering option --tz to see how warm the machine is getting when running a test.  For example:

... where x86_pkg_temp is the CPU package temperature and acpitz are the two ACPI thermal zones on my desktop.

Stress-ng is being used to run stress test various kernels across a range of Ubuntu devices, such as phone, desktop and server.   Thrashing a system with hundreds of processes and a lot of low memory pressure is just one method of checking that kernel and daemons can handle a mix of demanding work loads.

stress-ng 0.04.12 is now available in Ubuntu Wily.   See the stress-ng project page for more details.

Read more
Colin Ian King

New ACPI table tests in fwts 15.07.00

The Canonical Hardware Enablement Team and myself have been working on some significant improvements and changes to the Firmware Test Suite 15.07.00 over the past several weeks.  This cycle has been focused on adding more ACPI table testing support:

1. Added ACPI table tests:

  • BERT (Boot Error Record Table)
  • BGRT (Boot Graphics Resource Table)
  • BOOT (Boot Table)
  • CPEP (Corrected Platform Error Polling Table)
  • CSRT (Core System Resource Table)
  • DBG2 (Debug Port Table 2)
  • DBGP (Debug Port Table)
  • ECDT (Embedded Controller Boot Resources Table)
  • ERST (Error Record Serialization Table)
  • FACS (Firmware ACPI Control Structure)
  • HEST (Hardware Error Source Table)
  • LPIT (Low Power Idle Table test)
  • MSDM (Microsoft Data Management Table)
  • SLIC (Software Licensing Description Table)
  • SLIT (System Locality Distance Information)
  • SPCR (Serial Port Console Redirection Table)
  • SPMI (Service Processor Management Interface Description Table)
  • SRAT (System Resource Affinity Table)
  • TCPA (Trusted Computing Platform Alliance Capabilities Table)
  • UEFI (UEFI Data Table)
  • WAET (Windows ACPI Emulated Devices Table)
  • XENV (Xen Environment Table)
2. Moved the following tests out of the generic "acpitables" test into their own ACPI table tests:
  • FADT (Fixed ACPI Description Table)
  • HPET (HPET IA-PC High Precision Event Timer Table)
  • GTDT (GTDT Generic Timer Description Table)
  • MADT (Multiple APIC Description Table)
  • RSDP (Root System Description Pointer)
  • RSDT (Root System Description Table)
  • SBST (Smart Battery Specification Table)
  • XSDT (Extended System Description Table)
3. Updated ACPICA to version 20150616 and also 20150619 (ACPICA is used for the assembler/dissassembler and execution engine).

4. Renamed the --uefi and --acpi options to --uefitests and --acpitests respectively.

5. Improved fwts built-time regression tests.  To ensure future changes don't break fwts, we have added more regression tests to sanity check fwts ACPI table tests. Quality matters to us.

This release also incorporates some important bug fixes too, such making the acpidump dump file loading parser more robust, updating the SMM Communication fields on the UEFI table and fixing a segfault in the regular expression kernel log scanner on 32 bit systems.

For the next release of fwts, we are planning to continue to add table more tests from ACPI 5.x and ACPI 6.0 to get full coverage.

As ever, like all releases, for more details please consult the change log and the release notes.

    Read more
    Colin Ian King

    Firmware Related Blogs

    More often than not, I'm looking at ACPI and UEFI related issues, so I was glad to see that  Vincent Zimmer has collected up various useful links to blogs that are Firmware Related.   Very useful, thanks Vincent!

    Read more
    Colin Ian King

    Static code analysis on kernel source

    Since 2014 I have been running static code analysis using tools such as cppcheck and smatch against the Linux kernel source on a regular basis to catch bugs that creep into the kernel.   After each cppcheck run I then diff the logs and get a list of deltas on the error and warning messages, and I periodically review these to filter out false positives and I end up with a list of bugs that need some attention.

    Bugs such as allocations returning NULL pointers without checks, memory leaks, duplicate memory frees and uninitialized variables are easy to find with static analyzers and generally just require generally one or two line fixes.

    So what are the overall trends like?

    Warnings and error messages from cppcheck have been dropping over time and "portable warnings" have been steadily increasing.  "Portable warnings" are mainly from arithmetic on void * pointers (which GCC handles has byte sized but is not legal C), and these are slowly increasing over time.   Note that there is some variation in the results as I use the latest versions of cppcheck, and occasionally it finds a lot of false positives and then this gets fixed in later versions of cppcheck.

    Comparing it to the growth in kernel size the drop overall warning and error message trends from cppcheck aren't so bad considering the kernel has grown by nearly 11% over the time I have been running the static analysis.

    Kernel source growth over time
    Since each warning or error reported has to be carefully scrutinized to determine if they are false positives (and this takes a lot of effort and time), I've not yet been able determine the exact false positive rates on these stats.  Compared to the actual lines of code, cppcheck is finding ~1 error per 15K lines of source.

    It would be interesting to run this analysis on commercial static analyzers such as Coverity and see how the stats compare.  As it stands, cppcheck is doing it's bit in detecting errors and helping engineers to improve code quality.

    Read more
    Colin Ian King

    Powerstat and thermal zones

    Last night I was mulling over an overheating laptop issue that was reported by a user that turned out to be fluff and dust clogging up the fan rather than the intel_pstate driver being broken.

    While it is a relief that the kernel driver is not at fault, it still bothered me that this kind of issue should be very simple to diagnose but I overlooked the obvious.   When solving these issues it is very easy to doubt that the complex part of a system is working correctly (e.g. a kernel driver) rather than the simpler part (e.g. the fan not working efficiently).  Normally, I try to apply Occam's Razor which in the simplest form can be phrased as:

    "when you have two competing theories that make exactly the same predictions, the simpler one is the better."

    ..e.g. in this case, the fan is clogged up.

    Fortunately, laptops invariably provide Thermal Zone information that can be monitored and hence one can correlate CPU activity with the temperature of various components of a laptop.  So last night I added Thermal Zone sampling to powerstat 0.02.00 which is enabled with the new -t option.

    powerstat -tfR 0.5
    Running for 60.0 seconds (120 samples at 0.5 second intervals).
    Power measurements will start in 0 seconds time.

    Time User Nice Sys Idle IO Run Ctxt/s IRQ/s Watts x86_pk acpitz CPU Freq
    11:13:15 5.1 0.0 2.1 92.8 0.0 1 7902 1152 7.97 62.00 63.00 1.93 GHz
    11:13:16 3.9 0.0 2.5 93.1 0.5 1 7168 960 7.64 63.00 63.00 2.73 GHz
    11:13:16 1.0 0.0 2.0 96.9 0.0 1 7014 950 7.20 63.00 63.00 2.61 GHz
    11:13:17 2.0 0.0 3.0 94.5 0.5 1 6950 960 6.76 64.00 63.00 2.60 GHz
    11:13:17 3.0 0.0 3.0 93.9 0.0 1 6738 994 6.21 63.00 63.00 1.68 GHz
    11:13:18 3.5 0.0 2.5 93.6 0.5 1 6976 948 7.08 64.00 63.00 2.29 GHz

    ..the -t option now shows x86_pk (x86 CPU package temperature) and acpitz (APCI thermal zone) temperature readings in degrees Celsius.

    Now this is where the fun begins.  I ran powerstat for 60 seconds at 2 samples per second and then imported the data into LibreOffice.  To easily show corrleations between CPU load, power consumption, temperature and CPU frequency I normalized the data so that the lowest values were 0.0 and the highest were 1.0 and produced the following graph:

    One can see that the CPU frequency (green) scales with the the CPU load (blue) and so does the CPU power (orange).   CPU temperature (yellow) jumps up quickly when the CPU is loaded and then steadily increases.  Meanwhile, the ACPI thermal zone (purple) trails the CPU load because it takes time for the machine to warm up and then cool down (it takes time for a fan to pump out the heat from the machine).

    So, next time a laptop runs hot, running powerstat will capture the activity and correlating temperature with CPU activity should allow one to see if the overheating is related to a real CPU frequency scaling issue or a clogged up fan (or broken heat pipe!).

    Read more
    Colin Ian King

    Snooping on I/O using iosnoop

    A while ago I blogged about Brendan Gregg's excellent book for tracking down performance issues titled "Systems Performance, Enterprise and the Cloud".   Brendan has also produced a useful I/O diagnostic bash script iosnoop that uses ftrace to gather block device I/O events in real time.

    The following example snoops on I/O for 1 second:

    $ sudo iosnoop 1
    Tracing block I/O for 1 seconds (buffered)...
    kworker/u16:2 650 W 8,0 441077032 28672 1.46
    kworker/u16:2 650 W 8,0 441077024 4096 1.45
    kworker/u16:2 650 W 8,0 364810624 462848 1.35
    kworker/u16:2 650 W 8,0 364810240 69632 1.34

    And the next example snoops and shows start and end time stamps:
    $ sudo iosnoop -ts  
    Tracing block I/O. Ctrl-C to end.
    35253.062020 35253.063148 jbd2/sda1-211 211 WS 8,0 29737200 53248 1.13
    35253.063210 35253.063261 jbd2/sda1-211 211 FWS 8,0 18446744073709551615 0 0.05
    35253.063282 35253.063616 <idle> 0 WS 8,0 29737304 4096 0.33
    35253.063650 35253.063688 gawk 551 FWS 8,0 18446744073709551615 0 0.04
    35253.766711 35253.767158 kworker/u16:0 305 W 8,0 433580264 4096 0.45
    35253.766778 35253.767258 kworker/0:1H 321 FWS 8,0 18446744073709551615 0 0.48
    35253.767289 35253.767635 <idle> 0 WS 8,0 273358464 4096 0.35
    35253.767309 35253.767654 <idle> 0 W 8,0 118371312 4096 0.35
    35253.767648 35253.767741 <idle> 0 FWS 8,0 18446744073709551615 0 0.09
    Ending tracing...
    One needs to run the tool as root as it uses ftrace. There are a selection of filtering options, such as showing I/O from a specific device, I/O issues of a specific I/O type, selecting I/O on a specific PID or a specific name. iosnoop also can display the I/O completion times, start times and Queue insertion I/O start time. On Ubuntu, iosnoop can be installed using:
    sudo apt-get install perf-tools-unstable
    A useful I/O analysis tool indeed. For more details, install the tool and read the iosnoop man page.

    Read more
    Colin Ian King

    During some spare moments I've added a couple of minor CPU related enhancements to powerstat.    The new -c option gathers CPU C-state activity over the run and shows a summary at the end, for example:

     C-State  Resident   Count Latency
    C7-IVB 75.239% 102315 87
    C6-IVB 0.004% 60 80
    C3-IVB 0.138% 2892 59
    C1E-IVB 1.150% 7599 10
    C1-IVB 0.948% 4611 1
    POLL 0.000% 3 0
    C0 22.521%
    The above example shows that my Ivybridge i5-3210M spent ~75% of the time in the deepest C7 sleep state and ~22.5% of the time in the fully operating C0 state.

    A new -f option gathers CPU frequency statistics across all the on-line CPUs and displays the running average.   This provides an "instantaneous" view of the current CPU frequencies rather than a running average between the last sample, so beware that just gathering statistics using powerstat can cause CPU activity which of course can change CPU frequency.

    For a simple test, I ran powerstat for a short 250 second run and normalised the CPU Core Power, CPU Load and CPU Frequency stats so that the data ranges are 0..1 so I can plot all three stats and easily compare them:

    One can easily see the correlation between CPU Frequency, CPU Load and CPU core power consumed just from the powerstat data.

    Powerstat tries to be as lightweight and as small as possible to minimize the impact on system behaviour.  My hope is that adding these extra CPU instrumentation features adds more useful functionality without adding a larger system impact.  I've instrumented powerstat with perf and I believe that the overhead is sufficiently small to justify these changes.

    These two new features will be landing in powerstat 0.01.40 in Ubuntu Wily.

    Read more
    Colin Ian King

    Over the last few weeks I have been toying with the idea of adding more performance monitoring to stress-ng so one can see how much a stress test impacts on the CPU. The obvious choice to get such low level data is via Linux perf events using perf_event_open(2).

    The man page for perf_event_open() provides plenty of information to get perf working from userspace, however, I was a bit stumped when I used several hardware perf events and then ran out of hardware Perf Monitoring Units (PMUs) resulting in some strange event counter readings. I discovered that when one runs out of PMUs, perf will multiplex event counting and so the perf counters need to be scaled by multiplying by PERF_FORMAT_TOTAL_TIME_ENABLED and divided by PERF_FORMAT_TOTAL_TIME_RUNNING.

    Once I had figured this out, it was relatively plain sailing to get perf working in stress-ng.  So stress-ng V0.04.04 now supports the --perf option that just enables perf monitoring on each stress test being run, it is as simple as that. For multiple instances of a stress test, stress-ng will sum all the perf counters of each processes running the stress-test to provide an overall total.

    The following example will run the stress-ng cache stress test.  The first run enables cache flushing and so fetches of data will cause cache misses.  The second run has no cache flushing and hence has far lower cache miss rate.

    Note how the cache-flushing not only causes a far higher cache miss rate, but also reduces the effective number of instructions per cycle being executed and hence reduces the throughput (as one would expect).  With cache-flushing enabled I was seeing only 17.53 bogo ops per second compared to the 35.97 bogo ops per second with cache-flushing disabled.

    The perf stats are enlightening. I still find it incredible that my laptop has so much computing power.  Some of the more compute bound stressors (such as the stress-ng bitops cpu stressor) are hitting over 20 billion instructions per second on my machine, which is rather impressive.  It seems that gcc optimization and the x86 superscaler micro-ops are working efficiently with some of these stress tests.

    My hope is that the integrated perf monitoring in stress-ng will be instructive when comparing results on different processor architectures across the range of stress-ng stress tests.

    Read more
    Colin Ian King

    As simple experiment, I thought it would be interesting to investigate stress-ng compiled with GCC 4.9.1 and GCC 5.1.1 in terms of computational improvement and power consumption on various CPU stress methods.   The stress-ng CPU stress test contains various different mixes of integer, floating point, bit operations and logic operations that can be used for processor loading, so it makes a useful test to see how well the code gets optimized with GCC.

    Stress-ng provides a "bogo-ops" mechanism to measure a "unit of operation", normally this is just a count of the number of operations performed in a unit of time, hence allowing us to compare the relative performance of each stress method when compiled with different versions of GCC.  Running each stress method for a relatively long time (a few minutes) on an idle machine allows us to get a fairly stable and accurate measurement of bogo-ops per second.  Tests were run on a Lenovo x230 with an i5-3210M CPU.

    The first chart below shows the relative improvement in bogo-ops per second between the two versions of GCC.  A value of n indicates GCC 5.1.1 is n times faster  in terms of bogo-ops per second than GCC 4.9.1, hence values less than 1.0 show that GCC 5.1.1 has regressed in performance.

    It appears that int64, int32, int16, int8 and rand show some remarkable improvements with GCC 5.1.1; these all perform various integer operations (add, subtract, multiply, divide, xor, and, or, shift).

    In contrast, hamming, hanoi, parity and sieve show degraded performance with GCC 5.1.1.  Hanoi just exercises recursion of a function with a few arguments and some memory load/stores.  Hamming, parity and sieve exercise bit twiddling operations and memory load/stores.

    Further to just measuring computation, I used the Intel RAPL CPU package power measurements (using powerstat) to next measure the power consumed and then compute bogo ops per Watt for stress-ng built with GCC 4.9.1 and 5.1.1.  I then compared the relative improvement of 5.1.1 compared to 4.9.1:
    The chart above shows the same kind of characteristics as the first chart, but in terms of computational improvement per Watt.  Note that there are even better improvements in relative terms for the integer and rand CPU stress methods.  For example, the rand stress method shows a 1.6 x improvement in terms of computation per second and a 2.1 x improvement in terms of computation per Watt comparing GCC 4.9.1 with 5.1.1.

    It seems that benchmarking performance in terms of just compute improvements really should take into consideration the power consumption too to get a better idea of how compiler optimization improvements.  Compute-per-watt rather than compute-per-second should perhaps be the preferred benchmark in the modern high-density compute farms.

    Of course, these comparisons are just with one specific x86 micro-architecture,  so one would expect different results for different x86 CPUs..  I guess that is for another weekend to test if I get time.

    Read more
    Colin Ian King

    comparing cpuburn and stress-ng

    The cpuburn package contains several hand crafted assembler "burn" programs to load x86 processors and to maximize heat production to stress a system.  This also is the intention of the stress-ng "cpu" stress test which contains a variety of methods to stress CPUs with a wide range of instruction mixes.   Stress-ng is written in C and relies on the the compiler to generate efficient code to hopefully load the CPU.  So how does stress-ng compared to the hand crafted cpuburn suite of programs on modern processors?

    Since there is a correlation between power consumed and heat generated, I took the liberty to measure the CPU package power consumption measures using the Intel RAPL interface as one way of comparing cpuburn and stress-ng.  Recent versions of powerstat supports RAPL, so I ran each stressor for 120 seconds and took CPU package power measurements every 4 seconds over this interval with powerstat.

    So, the cpuburn "burn" programs do well, however, some of the stress-ng CPU stress methods seem to do better.   The best stress-ng CPU methods are: ackermann, callfunc, hanoi, decimal128, dither, int128decimal128, trig and zeta.  It appears that ackermann, callfunc and hanoi do well because these are very localised deeply recursive function calls, so I expect register save/restores and some stack activity is the main power consumer.  The rest exercise the integer and floating point units and memory load/stores.

    As it stands, a handful of stress-ng CPU stressors aren't as good as cpuburn. What is noticeable is that burnBX on an i3120M seems to do rather well in terms of loading the CPU.

    One conclusion to draw from this is that modern C compilers such as gcc (in this case, gcc 4.9.2) with a suitably chosen mix of stores, loads and integer/floating point operations can outperform hand written assembler in terms of loading the full CPU package.  When I have a little more time, I will try and repeat this experiment with clang and gcc 5

    Read more
    Colin Ian King

    An on-going background project of mine is to add various interesting system stress tests to stress-ng.  Over the past several months I've been looking at the ways to exercise various less used or obscure system calls just to add more kernel coverage to the tool.

    • rlimit - generate tens of thousands of SIGXFSZ and many SIGXCPU signals
    • itimer - exercise ITIMER_PROF and generate SIGPROF signals
    • mlock - lock and unlock pages with mlock()/munlock()
    • timerfd - exercise rapid CLOCK_REALTIME events by select() and read() on a timerfd.
    • memfd - exercise anonymous populated page memory mapping and unmappoing using memfd.
    • more aggressive affinity stressor changes to force more CPU IPIs
    • hdd - add readv/writev I/O option
    • tee - tee data between a writer and reader process using tee()
    • crypt - encrypt data with MD5, SHA-256 and SHA-512 using libcrypt
    • mmapmany - perform tens of thousands of memory maps/unmaps to exhaust the per-process mapping limit.
    • zombie - fill up process table with tens of thousands of zombie processes
    • str - heavily exercise a range of glibc string functions
    • xattr - exercise file extended attributes
    • readahead - random reads with readaheads
    • vm - add a rowhammer memory stressor well as extra per-stressor configuration settings and a lot of code clean up and bug fixing.

    I've recently been using stress-ng to exercise various kernels on a range of hardware and it has been useful in forcing bugs, especially with the memory specific stressors that seem to trip low memory corner cases.

    stress-ng 0.04.01 will be soon available in Ubuntu 15.10 Wily Werewolf.  Visit the stress-ng project page for more details.

    Read more
    Colin Ian King

    powerstat improvements with RAPL

    The Linux Running Average Power Limit (RAPL) interface was introduced about 2 years ago in the Linux kernel and allows userspace to read the power consumption from various x86 System-on-a-Chip (SoC) power domains.  The power domains range from the SoC package, CPU core, DRAM controller and graphics power plane.

    It appears that the Intel energy status MSRs can be read very rapidly and the resolution is exceptionally good; however, reading the MSR too frequently will consume some power when using the RAPL interface.

    I've improved powerstat to now use the RAPL interface with a new -R option (to measure just the total package power consumption).  A new -D option will show all the RAPL domain measurements available.  RAPL measurements are very responsive and one can easily correlate power spikes with bursts of system activity.

    Finally, I have added a basic histogram output with the new -H option. This will plot histograms of the power measurements and CPU load from the stats gathered during the powerstat run.

    Powerstat 0.01.37 is available in Ubuntu 15.10 Wily Werewolf and the source is available from the git repository.

    Read more
    Colin Ian King

    Over the past year or more I was focused on identifying power consuming processes on various mobile devices.  One of many strategies to reducing power is to remove unnecessary file system activity, such as extraneous logging, repeated file writes, unnecessary file re-reads and to reduce metadata updates.

    Fnotifystat is a utility I wrote to help identify such file system activity. My desire was to make the tool as small as possible for small embedded devices and to be relatively flexible without the need of using perf just in case the target device did not have perf built into the kernel by default.

    By default, fnotifystat will dump out every second any file system open/close/read/write operations across all mounted file systems, however, one can specify the delay in seconds and the number of times to dump out statistics.   fnotifystat uses the fanotify(7) interface to get file activity across the system, hence it needs to be run with CAP_SYS_ADMIN capability.

    An open(2), read(2)/write(2) and close(2) sequence by a process can produce multiple events, so fnotifystat has a -m option to merge events and hence reduce the amount of output.  A verbose -v option will output all file events if one desires to see the full system activity.

    If one desires to just monitor a specific collection of processes, one can specify a list of the process ID(s) or process names using the -p option, for example:

    sudo fnotifystat -p firefox,thunderbird

    fnotifystat catch events on all mounted file systems, but one can restrict that by specifying just path(s) one is interested in using the -i (include) option, for example:

    sudo fnotifystat -i /proc

    ..and one can exclude paths using the -x option.

    More information and examples can be found on the fnotifystat project page and the manual also contains more details and some examples too.

    Fnotifystat 0.01.10 is available in Ubuntu Vivid Vervet 15.04 and can also be installed for older releases from my power management tools PPA.

    Read more