Canonical Voices

Posts tagged with 'canonical'

Michael Hall

Snaps are a great way to get the most up to date applications on your desktop without putting the security or stability or your system at risk. I’ve been snapping up a bunch of things lately and the potential this new paradigm offers is going to be revolutionary. Unfortunately nothing comes for free, and the security of snaps comes with some necessary tradeoffs like isolation and confinement, which reduces some of the power and flexibility we’ve become used to as Linux users.

But now the developers of the snappy system (snapd, snap-confine and snapcraft) are giving us back some of that missing flexibility in the form of a new “content” interface which allows you to share files (executables, libraries, or data) between the snap packages that you develop. I decided to take this new interface for a test drive using one of the applications I had recently snapped: Geany, my editor of choice. Geany has the ability to load plugins to extend it’s functionality, and infact has a set of plugins available in a separate Github repository from the application itself.

I already had a working snap for Geany, so the next thing I had to do was create a snap for the plugins. Like Geany itself, the plugins are hosted on GitHub and have a nice build configuration already, so turning it into a snap was pretty trivial. I used the autotools plugin in Snapcraft to pull the git source and build all of the available plugins. Because my Geany snap was built with Gtk+ 3, I had to build the plugins for the same toolkit, but other than that I didn’t have to do anything special.

 plugin: autotools
 source-type: git
 configflags: [--enable-gtk3=yes --enable-all-plugins]

Now that I had a geany.snap and geany-plugins.snap, the next step was to get them working together. Specifically I wanted Geany to be able to see and load the plugin files from the plugins snap, so it was really just a one-way sharing. To do this I had to create both a slot and a plug using the content interface. Usually when you’re building snap you only use plugs, such as network or x11, because you are consuming services provided by the core OS. In those cases also you just have to provide the interface name in the list of plugs, because the interface and the plug have the same name.

But with the content interface you need to do more than that. Because different snaps will provide different content, and a single snap can provide multiple kinds of content, you have to define a new name that is specific to what content you are sharing. So in my geany-plugins snapcraft.yaml I defined a new kind of content that I called geany-plugins-all (because it contains all the geany plugins in the snap), and I put that into a slot called geany-plugins-slot which is how we will refer to it later. I told snapcraft that this new slot was using the content interface, and then finally told it what content to share across that interface, which for geany-plugins was the entire snap’s content.

 content: geany-plugins-all
 interface: content
 - /

With that I had one half of the content interface defined. I had a geany-plugins.snap that was able to share all of it’s content with another snap. The next step was to implement the plug half of the interface in my existing geany.snap. This time instead of using a slots: section I would define a plugs: section, with a new plug named geany-plugins-plug and again specifying the interface to be content just like in the slot. Here again I had to specify the content by name, which had to match the geany-plugins-all that was used in the slot. The names of the plug and slot are only relevant to the user who needs to connect them, it’s this content name that snapd uses to make sure they can be connected in the first place. Finally I had to give the plug a target directory for where the shared content will be put. I chose a directory called plugins, and when the snaps are connected the geany-plugins.snap content will be bind-mounted into this directory in the geany.snap

 content: geany-plugins-all
 default-provider: geany-plugins
 interface: content
 target: plugins

Lastly I needed to tell snapcraft which app would use this interface. Since the Geany snap only has one, I added it there.

 command: gtk-launch geany
 plugs: [x11, unity7, home, geany-plugins-plug]

Once the snaps were built, I could install them and the new plug and slot were automatically connected

$ snap interfaces
Slot                             Plug
geany-plugins:geany-plugins-slot geany:geany-plugins-plug

Now that put the plugins into the application’s snap space, but it wasn’t enough for Geany to actually find them. To do that I used Geany’s Extra plugin path preferences to point it to the location of the shared plugin files.

Screenshot from 2016-08-30 16-27-12

After doing that, I could open the Plugin manager and see all of the newly shared plugins. Not all of them work, and some assume specific install locations or access to other parts of the filesystem that they won’t have being in a snap. The Geany developers warned me about that, but the ones I really wanted appear to work.

Screenshot from 2016-08-30 16-29-54

Read more

Occasionally I find myself processing input data which arrives as a stream, like data from files or from a socket, but that has a known structure that can be modeled with C types. For instance, let’s say we are receiving from a socket a parcel that consists on a header of one byte, and a payload that is an integer. A naive way to handle this is the following (simplified for readability) code snippet:

int main(void)
    int fd;
    char *buff;
    struct sockaddr_in addr;
    int vint;
    char vchar;

    fd = socket(AF_INET, SOCK_STREAM, 0);
    buff = malloc(BUFF_SIZE);
    /* Init socket address */
    connect(fd, (struct sockaddr *) &addr, sizeof(addr));

    read(fd, buff, BUFF_SIZE);

    vchar = buff[0];
    vint  = *(int *) &buff[1];
    /* Do something with extracted data, free resources */
    return 0;

Here we get the raw data with a read() call, we read the first byte, then we read an integer by taking a pointer to the second read byte and casting it to a pointer to an integer. (for this example we are assuming that the integer inserted in the stream has the same size and endianness as the CPU ones).

There is a big issue with this: the cast to int *, which is undefined behavior according to the C standard 1. And it is because things can go wrong in at least two ways, first due to pointer aliasing rules, second due to type alignment.

Strict pointer aliasing tells the compiler that it can assume that pointers to different types point to different places in memory. This allows some optimizations, like reordering. Therefore, we could be in trouble if, say, we take &buff[1] into a char * and use it to write a byte in that location, as reordering could hit us. So just do not do that. Let’s also hope that we have a compiler that is not completely insane and does not move our reading by int pointer before the read() system call. We could also disable strict aliasing if we are using GCC with option -fno-strict-aliasing, which by the way is something that the Linux kernel does. At any rate, this is a complex subject and I will not dig into it this time.

We will concentrate in this article on how to solve the other problem, that is, how to access safely types that are not stored in memory in their natural alignment.

The C Standard-Compliant Solution

Before moving further, keep in mind that it is always possible to be strictly compliant with the standard and access safely memory without breaking language rules or using compiler or machine specific tricks. In the example, we could retrieve vint by doing

    vint  =   buff[1] + (buff[2] << 8)
            + (buff[3] << 16) + (buff[4] << 24);

(supposing stored data is little endian).

The issue here is performance: we are implicitly transforming four bytes to integers, then we have to bit-shift three of them, and finally we have to add them up. Note however that this is what we want if data and CPU have different endianness.

Doing Unaligned Memory Accesses

In all machine architectures there is a natural alignment for the different data types. This alignment is usually the size of the types, for instance in 32 bits architectures the alignment for integers is 4, for doubles it is 8, etc. If instances of these types are not stored in memory positions that are multiple of their alignment, we are talking about unaligned access. If we try to access unaligned data either of these can happen:

  • The hardware let’s us access it – but always at a performance penalty.
  • An exception is triggered by the CPU. This type of exception is called bus error 2.

We might be willing to accept the performance penalty 3, which is mitigated by CPU caches and not that noticeable in certain architectures like x86-64 , but we certainly do not want our program to crash. How possible is this? To be honest it is not something I have seen that often. Therefore, as a first analysis step, I checked how easy it was to get bus errors. To do so, I created the following C++ program, access1.cpp (I could not resist to use templates here to reduce the code size):

#include <iostream>
#include <typeinfo>
#include <cstring>

using namespace std;

template <typename T>
void print_unaligned(char *ptr)
    T *val = reinterpret_cast<T *>(ptr);

    cout << "Type is \"" << typeid(T).name()
         << "\" with size " << sizeof(T) << endl;
    cout << val << " *val: " << *val << endl;

int main(void)
    char *mem = new char[128];

    memset(mem, 0, 128);

    print_unaligned<int>(mem + 1);
    print_unaligned<long long>(mem);
    print_unaligned<long long>(mem + 1);
    print_unaligned<long double>(mem);
    print_unaligned<long double>(mem + 1);

    delete[] mem;
    return 0;

The program allocates memory using new char[], which as malloc() in C is guaranteed to allocate memory with the same alignment as the strictest fundamental type. After zeroing the memory, we access mem and mem + 1 by casting to different pointer types, knowing that the second address is odd, and therefore unaligned except for char * access.

I compiled the file with g++ on my laptop, ran it, and got

$ g++ access1.cpp -o access1
$ file access1
access1: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/, for GNU/Linux 2.6.32, BuildID[sha1]=09d0fb19340a10941eef4c3dc4d6eb29383e717d, not stripped
$ ./access1
Type is "i" with size 4
0x16c3c20 *val: 0
Type is "i" with size 4
0x16c3c21 *val: 0
Type is "x" with size 8
0x16c3c20 *val: 0
Type is "x" with size 8
0x16c3c21 *val: 0
Type is "e" with size 16
0x16c3c20 *val: 0
Type is "e" with size 16
0x16c3c21 *val: 0

No error for x86-64. This was expected as Intel architecture is known to support unaligned access by hardware, at a performance penalty (which is apparently quite small these days, see 4).

The second try was with an ARM CPU, compiling for arm-32:

$ g++ access1.cpp -o access1
$ file access1
access1: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=8c3c3e7d77fddd5f95d18dbffe37d67edc716a1c, not stripped
$ ./access1
Type is "i" with size 4
0x47b008 *val: 0
Type is "i" with size 4
0x47b009 *val: 0
Type is "x" with size 8
0x47b008 *val: 0
Type is "x" with size 8
Bus error (core dumped)

Now we get what we were searching for, a legitimate bus error, in this case when accessing a long long from an unaligned address. Commenting out the offending line and letting the program run further showed the error also when accessing a long double from mem + 1.

Fixing Unaligned Memory Accesses

After proving that this could be a real problem, at least for some architectures, I tried to find a solution that would let me do unaligned memory accesses in the most generic way. I could not find anything safe that was strictly following the C standard. However, all C/C++ compilers have ways to define packed structures, and that came to the rescue.

Packed structures are intended to minimize the padding that is introduced by alignment needed by the structure members. They are used when minimizing storage is a big concern. But what is interesting for us is that its members can be unaligned due to the packing, so dereferencing them must take that into account. Therefore, if we are accessing a type in a CPU that does not support unaligned access for that type the compiler must synthesize code that handles this transparently from the point of view of the C program.

To test that this worked as expected, I wrote access2.cpp, which uses GCC attribute __packed__ to define a packed structure:

#include <iostream>
#include <typeinfo>
#include <cstring>

using namespace std;

template <typename T>
struct __attribute__((__packed__)) struct_safe
    T val;

template <typename T>
void print_unaligned(char *ptr)
    struct_safe<T> *safe = reinterpret_cast<struct_safe<T> *>(ptr);

    cout << "Type is \"" << typeid(T).name()
         << "\" with size " << sizeof(T) << endl;
    cout << safe << " safe->val: " << safe->val << endl;

int main(void)
    char *mem = new char[128];

    memset(mem, 0, 128);

    print_unaligned<int>(mem + 1);
    print_unaligned<long long>(mem);
    print_unaligned<long long>(mem + 1);
    print_unaligned<long double>(mem);
    print_unaligned<long double>(mem + 1);

    delete[] mem;
    return 0;

In this case, instead of directly casting to the type, I cast to a pointer to the packed struct and access the type through it.

Compiling and running for x86-64 got the expected result: no error, all worked as before. Then I compiled and ran it in an ARM device:

$ g++ access2.cpp -o access2
$ file access2
access2: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=9a1ee8c2fcd97393a4b53fe12563676d9f2327a3, not stripped
$ ./access2
Type is "i" with size 4
0x391008 safe->val: 0
Type is "i" with size 4
0x391009 safe->val: 0
Type is "x" with size 8
0x391008 safe->val: 0
Type is "x" with size 8
0x391009 safe->val: 0
Type is "e" with size 8
0x391008 safe->val: 0
Type is "e" with size 8
0x391009 safe->val: 0

No bus errors anymore! It worked as expected. To gain some understanding of what was happening behind the curtains, I disassembled the generated ARM binaries. For both access1 and access2, the same instruction was being used when I was getting a value after casting to int: LDR, which unsurprisingly loads a 32-bit word into a register. But for the long long, I found that access1 was using LDRD, which loads double words (8 bytes) from memory, while access2 was using two LDR instructions instead.

This all made a lot of sense, as ARM states that LDR supports access to unaligned data, while LDRD does not 5. Indeed the later is faster, but has this restriction. It was also good to check that there was no penalty for using the packed structure for integers: GCC does a good job to discriminate when the CPU really needs to handle differently unaligned accesses.

GCC cast-align Warning

GCC has a warning that can help to identify points in the code when we might be accessing unaligned data, which is activated with -Wcast-align. It is not part of the warnings that are activated by options -Wall or -Wextra, so we will have to add it explicitly if we want it. The warning is only triggered when compiling for architectures that do not support unaligned access for all types, so you will not see it if compiling only for x86.

When triggered, you will see something like

file.c:28:23: warning: cast increases required alignment of target type [-Wcast-align]
   int *my_int_ptr = (int *) &buf[i];


The moral of this post is that you need to be very careful when casting pointers to a type different to the original one 6. When you need to do that, think about alignment issues first, and also think on your target architectures. There are programs that we want to run on more than one CPU type and too many times we only test in our reference.

Unfortunately the C standard does not give us a standard way of doing efficient access to unaligned data, but most if not all compilers seem to provide ways to do this. If we are using GCC, __attribute__((__packed__)) can help us when we might be doing unaligned accesses. The ARM compiler has a __packed attribute for pointers 7, and I am sure other compilers provide similar machinery. I also recommend to activate -Wcast-align if using GCC, which makes easier to spot alignment issues.

Finally, a word of caution: in most cases you should not do this type of casts. Some times you can define structures and read directly data onto them, some times you can use unions. Bear in mind always the strict pointer aliasing rules, which can hit back. To summarize, think twice before using the sort of trick showed in the post, and use them only when really needed.

Read more
Dustin Kirkland

I hope you'll enjoy a shiny new 6-part blog series I recently published at
  1. The first article is a bit of back story, perhaps a behind-the-scenes look at the motivations, timelines, and some of the work performed between Microsoft and Canonical to bring Ubuntu to Windows.
  2. The second article is an updated getting-started guide, with screenshots, showing a Windows 10 user exactly how to enable and run Ubuntu on Windows.
  3. The third article walks through a dozen or so examples of the most essential command line utilities a Windows user, new to Ubuntu (and Bash), should absolutely learn.
  4. The fourth article shows how to write and execute your first script, "Howdy, Windows!", in 6 different dynamic scripting languages (Bash, Python, Perl, Ruby, PHP, and NodeJS).
  5. The fifth article demonstrates how to write, compile, and execute your first program in 7 different compiled programming languages (C, C++, Fortran, Golang).
  6. The sixth and final article conducts some performance benchmarks of the CPU, Memory, Disk, and Network, in both native Ubuntu on a physical machine, and Ubuntu on Windows running on the same system.
I really enjoyed writing these.  Hopefully you'll try some of the examples, and share your experiences using Ubuntu native utilities on a Windows desktop.  You can find the source code of the programming examples in Github and Launchpad:

Read more

Sensors are an important part of IoT. Phones, robots and drones all have a slurry of sensors. Sensor chips are everywhere, doing all kinds of jobs to help and entertain us. Modern games and game consoles can thank sensors for some wonderfully active games.

Since I became involved with sensors and wrote QtSensorGestures as part of the QtSensors team at Nokia, sensors have only gotten cheaper and more prolific.

I used Ubuntu Server, snappy, a raspberry pi 3, and the senseHAT sensor board to create a senseHAT sensors snap. Of course, this currently only runs in devmode on raspberry pi3 (and pi2 as well) .

To future proof this, I wanted to get sensor data all the way up to QtSensors, for future QML access.

I now work at Canonical. Snappy is new and still in heavy development so I did run into a few issues. First up was QFactoryLoader which finds and loads plugins, was not looking in the correct spot. For some reason, it uses $SNAP/usr/bin as it's QT_PLUGIN_PATH. I got around this for now by using a wrapper script and setting QT_PLUGIN_PATH to $SNAP/usr/lib/arm-linux-gnueabihf/qt5/plugins

Second issue was that QSensorManager could not see it's configuration file in /etc/xdg/QtProject which is not accessible to a snap. So I used the wrapper script to set up  XDG_CONFIG_DIRS as $SNAP/etc/xdg

[NOTE] I just discovered there is a part named "qt5conf" that can be used to setup Qt's env vars by using the included command qt5-launch  to run your snap's commands.

Since there is no libhybris in Ubuntu Core, I had to decide what QtSensor backend to use. I could have used sensorfw, or maybe iio-sensor-proxy but RTIMULib already worked for senseHAT. It was easier to write a QtSensors plugin that used RTIMULib, as opposed to adding it into sensorfw. iio-sensor-proxy is more for laptop like machines and lacks many sensors.
RTIMULib uses a configuration file that needs to be in a writable area, to hold additional device specific calibration data. Luckily, one of it's functions takes a directory path to look in. Since I was creating the plugin, I made it use a new variable SENSEHAT_CONFIG_DIR so I could then set that up in the wrapper script.

This also runs in confinement without devmode, but involves a simple sensors snapd interface.
One of the issues I can already see with this is that there are a myriad ways of accessing the sensors. Different kernel interfaces - iio,  sysfs, evdev, different middleware - android SensorManager/hybris, libhardware/hybris, sensorfw and others either I cannot speak of or do not know about.

Once the snap goes through a review, it will live here, but for now, there is working code is at my sensehat repo.

Next up to snapify, the Matrix Creator sensor array! Perhaps I can use my sensorfw snap or iio-sensor-proxy snap for that.

Read more

So there I was. I did have to use a proprietary library, for which I had no sources and no real hope of support from the creators. I built my program against it, I ran it, and I got a segmentation fault. An exception that seemed to happen inside that insidious library, which was of course stripped of all debugging information. I scratched my head, changed my code, checked traces, tried valgrind, strace, and other debugging tools, but found no obvious error. Finally, I assumed that I had to dig deeper and do some serious debugging of the library’s assembly code with gdb. The rest of the post is dedicated to the steps I followed to find out what was happening inside the wily proprietary library that we will call libProprietary. Prerequisites for this article are some knowledge of gdb and ARM architecture.

Some background on the task I was doing: I am a Canonical employee that works as developer for Ubuntu for Phones. In most, if not all, phones, the BSP code is not 100% open and we have to use proprietary libraries built for Android. Therefore, these libraries use bionic, Android’s libc implementation. As we want to call them inside binaries compiled with glibc, we resort to libhybris, an ingenious library that is able to load and call libraries compiled against bionic while the rest of the process uses glibc. This will turn out to be critical in this debugging. Note also that we are debugging ARM 32-bits binaries here.

The Debugging Session

To start, I made sure I had installed glibc and other libraries symbols and started to debug by using gdb in the usual way:

$ gdb myprogram
GNU gdb (Ubuntu 7.9-1ubuntu1) 7.9
Starting program: myprogram
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/".
[New Thread 0xf49de460 (LWP 7101)]
[New Thread 0xf31de460 (LWP 7104)]
[New Thread 0xf39de460 (LWP 7103)]
[New Thread 0xf41de460 (LWP 7102)]
[New Thread 0xf51de460 (LWP 7100)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xf49de460 (LWP 7101)]
0x00000000 in ?? ()
(gdb) bt
#0  0x00000000 in ?? ()
#1  0xf520bd06 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) info proc mappings
process 7097
Mapped address spaces:

	Start Addr   End Addr       Size     Offset objfile
	   0x10000    0x17000     0x7000        0x0 /usr/bin/myprogram
	0xf41e0000 0xf49df000   0x7ff000        0x0 [stack:7101]
	0xf51f6000 0xf5221000    0x2b000        0x0 /android/system/lib/
	0xf5221000 0xf5222000     0x1000        0x0 
	0xf5222000 0xf5224000     0x2000    0x2b000 /android/system/lib/
	0xf5224000 0xf5225000     0x1000    0x2d000 /android/system/lib/

We can see here that we get the promised crash. I execute a couple of gdb commands after that to see the backtrace and part of the process address space that will be of interest in the following discussion. The backtrace shows that a segment violation happened when the CPU tried to execute instructions in address zero, and we can see by checking the process mappings that the previous frame lives inside the text segment of There is no backtrace beyond that point, but that should come as no surprise as there is no DWARF information in libProprietary, and also noting that usage of frame pointer is optimized away quite commonly these days.

After this I tried to get a bit more information on the CPU state when the crash happened:

(gdb) info reg
r0             0x0	0
r1             0x0	0
r2             0x0	0
r3             0x9	9
r4             0x0	0
r5             0x0	0
r6             0x0	0
r7             0x0	0
r8             0x0	0
r9             0x0	0
r10            0x0	0
r11            0x0	0
r12            0xffffffff	4294967295
sp             0xf49dde70	0xf49dde70
lr             0xf520bd07	-182403833
pc             0x0	0x0
cpsr           0x60000010	1610612752
(gdb) disassemble 0xf520bd02,+10
Dump of assembler code from 0xf520bd02 to 0xf520bd0c:
   0xf520bd02:	b	0xf49c9cd6
   0xf520bd06:	movwpl	pc, #18628	; 0x48c4	<UNPREDICTABLE>
   0xf520bd0a:	andlt	r4, r11, r8, lsr #12
End of assembler dump.

Hmm, we are starting to see weird things here. First, in 0xf520bd02 (which probably has been executed little before the crash) we get an unconditional branch to some point in the thread stack (see mappings in previous figure). Second, the instruction in 0xf520bd06 (which should be executed after returning from the procedure that provokes the crash) would load into the pc (program counter) an address that is not mapped: we saw that the first mapped address is 0x10000 in the previous figure. The movw instruction has also a “pl” suffix that makes the instruction execute only when the operand is positive or zero… which is obviously unnecessary as 0x48c4 is encoded in the instruction.

I resorted to doing objdump -d to disassemble the library and compare with gdb output. objdump shows, in that part of the file (subtracting the library load address gives us the offset inside the file: 0xf520bd02-0xf51f6000=0x15d02):

   15d02:	f7f3 eade 	blx	92c0 <__android_log_print@plt>;
   15d06:	f8c4 5304 	str.w	r5, [r4, #772]	; 0x304
   15d0a:	4628      	mov	r0, r5
   15d0c:	b00b      	add	sp, #44	; 0x2c
   15d0e:	e8bd 8ff0 	ldmia.w	sp!, {r4, r5, r6, r7, r8, r9, sl, fp, pc}

which is completely different from what gdb shows! What is happening here? Taking a look at addresses for both code chunks, we see that instructions are always 4 bytes in gdb output, while they are 2 or 4 in objdump‘s. Well, you have guessed, don’t you? We are seeing “normal” ARM instructions in gdb, while objdump is decoding THUMB-2 instructions. Certainly objdump seems to be right here as the output is more sensible: we have a call to an executable part of the process space in 0x15d02 (it is resolved to a known function, __android_log_print), and the following instructions seems like a normal function epilogue in ARM: a return value is stored in r0, the sp (stack pointer) is incremented (we are freeing space in the stack), and we restore registers.

If we get back to the register values, we see that cpsr (current program status register [1]) does not have the T bit set, so gdb thinks we are using ARM instructions. We can change this by doing

(gdb) set $cpsr=0x60000030
(gdb) disass 0xf520bd02,+15
Dump of assembler code from 0xf520bd02 to 0xf520bd11:
   0xf520bd02:	blx	0xf51ff2c0
   0xf520bd06:	str.w	r5, [r4, #772]	; 0x304
   0xf520bd0a:	mov	r0, r5
   0xf520bd0c:	add	sp, #44	; 0x2c
   0xf520bd0e:	ldmia.w	sp!, {r4, r5, r6, r7, r8, r9, r10, r11, pc}
End of assembler dump.

Ok, much better now [2]. The thumb bit in cpsr is determined by last bx/blx call: if the address is odd, the procedure to which we are calling contains THUMB instructions, otherwise they are ARM (a good reference for these instructions is [3]). In this case, after an exception the CPU moves to arm mode, and gdb is unable to know which is the right mode when disassembling. We can search for hints on which parts of the code are arm/thumb by looking at the values in registers used by bx/blx, or by looking at the lr (link register): we can see above that the value after the crash was 0xf520bd07, which is odd and indicates that 0xf520bd06 contains a thumb instruction. However, for some reason gdb is not able to take advantage of this information.

Of course this problem does not happen if we have debugging information: in that case we have special symbols that let gdb know if the section where the code is contains thumb instructions or not [4]. As those are not found, gdb uses the cpsr value. Here objdump seems to have better heuristics though.

After solving this issue with instruction decoding, I started to debug __android_log_print to check what was happening there, as it looked like the crash was happening in that call. I spent quite a lot of time there, but found nothing. All looked fine, and I started to despair. Until I inserted a breakpoint in address 0xf520bd06, right after the call to __android_log_print, run the program… and it stopped at that address, no crash happened. I started to execute the program instruction by instruction after that:

(gdb) b *0xf520bd06
(gdb) run
Breakpoint 1, 0xf520bd06 in ?? ()
(gdb) si
0xf520bd0a in ?? ()
(gdb) si
0xf520bd0c in ?? ()
(gdb) si
0xf520bd0e in ?? ()
Cannot insert breakpoint 0.
Cannot access memory at address 0x0

Something was apparently wrong with instruction ldmia, which restores registers, including the pc, from the stack. I took a look at the stack in that moment (taking into account that ldmia had already modified the sp after restoring 9 registers == 36 bytes):

(gdb) x/16xw $sp-36
0xf49dde4c:	0x00000000	0x00000000	0x00000000	0x00000000
0xf49dde5c:	0x00000000	0x00000000	0x00000000	0x00000000
0xf49dde6c:	0x00000000	0x00000000	0x00000000	0x00000000
0xf49dde7c:	0x00000000	0x00000000	0x00000000	0x00000000

All zeros! At this point it is clear that this is the real point where the crash is happening, as we are loading 0 into the pc. This looked clearly like a stack corruption issue.

But, before moving forward, why are we getting a wrong backtrace from gdb? Well, gdb is seeing a corrupted stack, so it is not able to unwind it. It would not be able to unwind it even if having full debug information. The only hint it has is the lr. This register contains the return address after execution of a bl/blx instruction [3]. If the called procedure is non-leaf, it is saved in the prologue, and restored in the epilogue, because it gets overwritten when branching to other procedures. In this case, it is restored on the pc and sometimes it is also saved back in the lr, depending on whether we have arm-thumb interworking built in the procedure or not [5]. It is not overwritten if we have a leaf procedure (as there are no procedure calls inside these).

As gdb has no additional information, it uses the lr to build the backtrace, assuming we are in a leaf procedure. However this is not true and the backtrace turns out to be wrong. Nonetheless, this information was not completely useless: lr was pointing to the instruction right after the last bl/blx instruction that was executed, which was not that far away from the real point where the program was crashing. This happened because fortunately __android_log_print has interworking code and restores the lr, otherwise the value of lr could have been from a point much far away from the point where the real crash happens. Believe or not, but it could have been even worse!

Having now a clear idea of where and why the crash was happening, things accelerated. The procedure where the crash happened, as disassembled by objdump, was (I include here only the more relevant parts of the code)

00015b1c <ProprietaryProcedure@@Base>:
   15b1c:	e92d 4ff0 	stmdb	sp!, {r4, r5, r6, r7, r8, r9, sl, fp, lr}
   15b20:	b08b      	sub	sp, #44	; 0x2c
   15b22:	497c      	ldr	r1, [pc, #496]	; (15d14 <ProprietaryProcedure@@Base+0x1f8>)
   15b24:	2500      	movs	r5, #0
   15b26:	9500      	str	r5, [sp, #0]
   15b28:	4604      	mov	r4, r0
   15b2a:	4479      	add	r1, pc
   15b2c:	462b      	mov	r3, r5
   15b2e:	f8df 81e8 	ldr.w	r8, [pc, #488]	; 15d18 <ProprietaryProcedure@@Base+0x1fc>
   15b32:	462a      	mov	r2, r5
   15b34:	f8df 91e4 	ldr.w	r9, [pc, #484]	; 15d1c <ProprietaryProcedure@@Base+0x200>
   15b38:	ae06      	add	r6, sp, #24
   15b3a:	f8df a1e4 	ldr.w	sl, [pc, #484]	; 15d20 <ProprietaryProcedure@@Base+0x204>
   15b3e:	200f      	movs	r0, #15
   15b40:	f8df b1e0 	ldr.w	fp, [pc, #480]	; 15d24 <ProprietaryProcedure@@Base+0x208>
   15b44:	f7f3 ef76 	blx	9a34 <prctl@plt>
   15b48:	44f8      	add	r8, pc
   15b4a:	4629      	mov	r1, r5
   15b4c:	44f9      	add	r9, pc
   15b4e:	2210      	movs	r2, #16
   15b50:	44fa      	add	sl, pc
   15b52:	4630      	mov	r0, r6
   15b54:	44fb      	add	fp, pc
   15b56:	f7f3 ea40 	blx	8fd8 <memset@plt>
   15b5a:	a807      	add	r0, sp, #28
   15b5c:	f7f3 ef70 	blx	9a40 <sigemptyset@plt>
   15b60:	4b71      	ldr	r3, [pc, #452]	; (15d28 <ProprietaryProcedure@@Base+0x20c>)
   15b62:	462a      	mov	r2, r5
   15b64:	9508      	str	r5, [sp, #32]
   15b66:	4631      	mov	r1, r6
   15b68:	447b      	add	r3, pc
   15b6a:	681b      	ldr	r3, [r3, #0]
   15b6c:	200a      	movs	r0, #10
   15b6e:	9306      	str	r3, [sp, #24]
   15b70:	f7f3 ef6c 	blx	9a4c <sigaction@plt>
   15d02:	f7f3 eade 	blx	92c0 <__android_log_print@plt>
   15d06:	f8c4 5304 	str.w	r5, [r4, #772]	; 0x304
   15d0a:	4628      	mov	r0, r5
   15d0c:	b00b      	add	sp, #44	; 0x2c
   15d0e:	e8bd 8ff0 	ldmia.w	sp!, {r4, r5, r6, r7, r8, r9, sl, fp, pc}

The addresses where this code is loaded can be easily computed by adding 0xf51f6000 to the file offsets shown in the first column. We see that a few calls to different external functions [6] are performed by ProprietaryProcedure, which is itself an exported symbol.

I restarted the debug session, added a breakpoint at the start of ProprietaryProcedure, right after stmdb saves the state, and checked the stack values:

(gdb) b *0xf520bb20
Breakpoint 1 at 0xf520bb20
(gdb) cont
Breakpoint 1, 0xf520bb20 in ?? ()
(gdb) p $sp
$1 = (void *) 0xf49dde4c
(gdb) x/16xw $sp
0xf49dde4c:	0xf49de460	0x0007df00	0x00000000	0xf49dde70
0xf49dde5c:	0xf49de694	0x00000000	0xf77e9000	0x00000000
0xf49dde6c:	0xf75b4491	0x00000000	0xf49de460	0x00000000
0xf49dde7c:	0x00000000	0xfd5b4eba	0xfe9dd4a3	0xf49de460

We can see that the stack contains something, including a return address that looks valid (0xf75b4491). Note also that the procedure must never touch this part of the stack, as it belongs to the caller of ProprietaryProcedure.

Now it is a simply a matter of bisecting the code between the beginning and the end of ProprietaryProcedure to find out where we are clobbering the stack. I will save you of developing here this tedious process. Instead, I will just show, that, in the end, it turned out that the call to sigemptyset() is the culprit [7]:

(gdb) b *0xf520bb5c
Breakpoint 1 at 0xf520bb5c
(gdb) b *0xf520bb60
Breakpoint 2 at 0xf520bb60
(gdb) run
Breakpoint 1, 0xf520bb5c in ?? ()
(gdb) x/16xw 0xf49dde4c
0xf49dde4c:	0xf49de460	0x0007df00	0x00000000	0xf49dde70
0xf49dde5c:	0xf49de694	0x00000000	0xf77e9000	0x00000000
0xf49dde6c:	0xf75b4491	0x00000000	0xf49de460	0x00000000
0xf49dde7c:	0x00000000	0xfd5b4eba	0xfe9dd4a3	0xf49de460
(gdb) cont
Breakpoint 2, 0xf520bb60 in ?? ()
(gdb) x/16xw 0xf49dde4c
0xf49dde4c:	0x00000000	0x00000000	0x00000000	0x00000000
0xf49dde5c:	0x00000000	0x00000000	0x00000000	0x00000000
0xf49dde6c:	0x00000000	0x00000000	0x00000000	0x00000000
0xf49dde7c:	0x00000000	0x00000000	0x00000000	0x00000000

Note here that I am printing the part of the stack not reserved by the function (0xf49dde4c is the value of the sp before execution of the line at offset 0x15b20, see the code).

What is going wrong here? Now, remember that at the beginning of the article I mentioned that we were using libhybris. libProprietary assumes a bionic environment, and the libc functions it calls are from bionic’s libc. However, libhybris has hooks for some bionic functions: for them bionic is not called, instead the hook is invoked. libhybris does this to avoid conflicts between bionic and glibc: for instance having two allocators fighting for process address space is a recipe for disaster, so malloc() and related functions are hooked and the hooks call in the end the glibc implementation. Signals related functions were hooked too, including sigemptyset(), and in this case the hook simply called glibc implementation.

I looked at glibc and bionic implementations, in both cases sigemptyset() is a very simple utility function that clears with memset() a sigset_t variable. All pointed to different definitions of sigset_t depending on the library. Definition turned out to be a bit messy when looking at the code as it depended on build time definitions, so I resorted to gdb to print the type. For a executable compiled for glibc, I saw

(gdb) ptype sigset_t
type = struct {
    unsigned long __val[32];

and for one using bionic

(gdb) ptype sigset_t
type = unsigned long

This finally confirms where the bug is, and explains it: we are overwriting the stack because libProprietary reserves in the stack memory for bionic’s sigset_t, while we are using glibc’s sigemptyset(), which uses a different definition for it. As this definition is much bigger, the stack gets overwritten after the call to memset(). And we get the crash later when trying to restore registers when the function returns.

After knowing this, the solution was simple: I removed the libhybris hooks for signal functions, recompiled it, and… all worked just fine, no crashes anymore!

However, this is not the final solution: as signals are shared resources, it makes sense to hook them in libhybris. But, to do it properly, the hooks have to translate types between bionic in glibc, thing that we were not doing (we were simply calling glibc implementation). That, however, is “just work”.

Of course I wondered why the heck a library that is kind of generic needs to mess around with signals, but hey, that is not my fault ;-).


I can say I learned several things while debugging this:

  1. Not having the sources is terrible for debugging (well, I already knew this). Unfortunately not open sourcing the code is still a standard practice in part of the industry.
  2. The most interesting technical bit here is IMHO that we need to be very cautious with the backtrace that debuggers shows after a crash. If you start to see things that do not make sense it is possible that registers or stack have been messed up and the real crash happens elsewhere. Bear in mind that the very first thing to do when a program crashes is to make sure that we know the exact point where that happens.
  3. We have to be careful in ARM when disassembling, because if there is no debug information we could be seeing the wrong instruction set. We can check evenness of addresses used by bx/blx and of the lr to make sure we are in the right mode.
  4. Some times taking a look at assembly code can help us when debugging, even when we have the sources. Note that if I had had the C sources I would have seen the crash happening right when returning from a function, and it might not have been that immediate to find out that the stack was messed up. The assembly clearly pointed to an overwritten stack.
  5. Finally, I personally learned some bits of ARM architecture that I did not know, which was great.

Well, this is it. I hope you enjoyed the (lengthy, I know) article. Thanks for your reading!

[2] We can get the same result by executing in gdb set arm fallback-mode thumb, but changing the register seemed more pedagogical here.
[6] In fact the calls are to the PLT section, which is inside the library. The PLT calls in turn, by using addresses in the GOT data section, either directly the function or the dynamic loader, as we are doing lazy loading. See, for instance.
[7] I had to use two breakpoints between consecutive instructions because the “ni” gdb command was not working well here.

Read more
Kyle Nitzsche

Running X Apps on Ubuntu Devices You can install, launch, and use traditional debian-packaged X apps on Ubuntu devices. This may be unexpected given that Ubuntu devices do not seem to support user-installed debian packages, nor do they run the X Display Server. But it does work, courtesy of Mir/XMir and Libertine.

So here’s a bit of background to get started.

But first, please note that at this time, display and use of X apps on an external monitor is only available on the Pro5/M10 and on future devices. (BQ 4.5/E5 and Meizu MX4 do not support this feature.)

Hello Mir (Goodbye X)

Traditionally, and still on the Ubuntu Classic desktop with Unity 7, Ubuntu runs an X Display Server. Apps are debian packaged. And, they are written for X:

Due in part to X’s inherent security shortcomings, the Mir display server is now used on Ubuntu Devices under Unity 8 (although not yet by default on the desktop). XMir bridges traditional X apps to Mir. That is, apps written for X can run fine in a Mir/XMir environment:

Packages and the root file system

Ubuntu Classic has a root file system (rootfs) that is populated through installation of a carefully curated set of debian packages. At run time, users can install debian packages to add apps or modify their system.

This approach raises security concerns because debian packages execute installation scripts with root level privileges and because debian packages can alter what the rootfs provides by modifying or replacing core system components.

Ubuntu devices are designed for security and reliability. Ubuntu devices have a read-only rootfs that is small and tight, providing just what is needed and simplifying system updates. The rootfs is not modifiable by the user. Indeed it is mounted as a read-only partition. Users install apps through click packages that do not modify the rootfs.

Given all of this: how do users install debian packaged apps that use X on Ubuntu Devices? The answer is LIbertine with XMir.

Hello Libertine

Libertine is a system to manage app containers. It is specifically designed to support the many traditional X apps that are debian packaged. Each container is a separate Ubuntu rootfs populated through debian package installations. (Currently these containers are chroots: later, LXD contains will be supported. Also, currently the containers must be of the same Ubuntu series as the device: Vivid.)

So, you can install or create a libertine container, install debian packaged X apps into it, and launch them using the XApps scope. The apps access to the user’s key directories: Documents, Downloads, Music, Pictures, and Videos. So data files created and saved by an app in one container are available to apps in any other container, and indeed outside of the containers.

Let’s take a quick look at the XApps scope.

XApps Scope

This scope simply lists the containers and, for each container, it displays its apps. Here’s a look at a device with two containers. This system has two containers (Puritine and My Container). And each has a few apps:

  • Tap an app to launch it.
  • long press an app to hide it.
  • If you have any hidden apps, see them from the search icon (magnifying glass) and tap Hidden X Apps. Long press a hidden app to unhide it.
  • Note that a container with no apps does not display in the scope.
So how does one create and delete containers, and add or remove apps from them?

Libertine Container Manager

libertine-container-manager is a command line tool you use on the device to create and manage containers. This includes installing debian packaged apps into them. (These containers are created in the phablet user’s home directory and are not a part of the read-only rootfs.)

Note: libertine-container-manager currently cannot be run in the Terminal App. Instead please connect to your device from an Ubuntu system using phablet-shell.
Listing Containers
phablet@ubuntu-phablet:~$ libertine-container-manager list

The “puritine” container is pre-installed on many devices through the com.ubuntu.puritine click package (“Desktop Applications”):

phablet@ubuntu-phablet:~$ click list | grep puritine
com.ubuntu.puritine 0.11

The second container (“my-container”) was created on the device with libertine-container-manager.

Note: It is possible to pre-install customized containers through bespoke channels.
Creating a Libertine Container
You can create a new container on a device. The container needs a unique ID and (optionally) a name.

Note: The container must be the same Ubuntu series as the device, currently: vivid.

phablet@ubuntu-phablet:~$ libertine-container-manager create --id my-container --name "My Container" --distro vivid --type chroot

I: Retrieving Release
I: Retrieving Release.gpg
I: Checking Release signature
I: Valid Release signature (key id 790BC7277767219C42C86F933B4FE6ACC0B21F32)
I: Retrieving Packages
I: Validating Packages
I: Resolving dependencies of required packages...
Listing Apps in a Container
It’s easy to list the apps in a container. You just use the container’s id, as follows:

Note: We add the optional --json argument here and show only lines with “name” for display convenience.

phablet@ubuntu-phablet:~$ libertine-container-manager list-apps --id my-container --json | grep "\"name\""
"name": "Panel Manager",
"name": "Python (v3.4)",
"name": "Python (v2.7)",
"name": "gedit",
"name": "Help",
"name": "Notification Daemon",
"name": "Terminal",
Also note that all apps that install a .desktop file are listed by this command, although many of them are not displayed in the XApps scope since they are not appropriate.
Installing an app in a container
To install a debian package in a container, you just use install-package with the container id and the debian binary package name, as follows:

phablet@ubuntu-phablet:~$ libertine-container-manager install-package --id my-container --package terminator

The package and all of its dependencies are installed in the container. After this, assuming the package installs a .desktop file, it displays in the XApps scope and is launchable with a tap as expected.
Installing an app from a specific Launchpad PPA
By default, available debian packages are installed from the standard Ubuntu archive the chroot’s apt configuration points to. You can add a launchpad PPA, as follows:

phablet@ubuntu-phablet:~$ libertine-container-manager configure --id my-container --archive ppa:USER/PPA-NAME

(Currently, private PPAs are scheduled for an upcoming release.)

After this, you can install packages into the container as usual, including from the PPA.
Removing apps from a container
Remove a debian package from a container with:

phablet@ubuntu-phablet:~$ libertine-container-manager remove-package --id my-container --package PACKAGE_NAME
Libertine-container-manager help
Use the --help for top level help.

You can see details on each subcommand, for example remove-package, as follows:

phablet@ubuntu-phablet:~$ libertine-container-manager remove-package --help
usage: libertine-container-manager remove-package [-h] -p PACKAGE [-i ID] [-r]

optional arguments:
-h, --help show this help message and exit
-p PACKAGE, --package PACKAGE
Name of package to remove. Required. -i ID, --id ID Container identifier. Default container is used if
-r, --readline Readline mode. Use text-based frontend during debconf
Updating a container
Want the debian packages in a container updated? Easy:

phablet@ubuntu-phablet:~/.cache/libertine-container/my-container$ libertine-container-manager update --id my-container
Executing a Command in a Container
phablet@ubuntu-phablet:~/.cache/libertine-container/my-container$ libertine-container-manager exec --command "apt-get update" --id my-container
Atteint vivid InRelease
Atteint vivid InRelease
Atteint vivid-updates InRelease
Atteint vivid/main armhf Packages
Atteint vivid/main Translation-en
Atteint vivid/main armhf Packages [...]

Note: Running the apt-get update command in a container may be useful to update the container’s knowledge of newly available packages without installing/updating them all. You can then see whether a package is available, with:

phablet@ubuntu-phablet:~/.cache/libertine-container/my-container$ libertine-container-manager exec --command "apt-cache policy firefox" --id my-container
Installé : (aucun)
Candidat : 44.0+build3-0ubuntu0.15.04.1
Table de version :
44.0+build3-0ubuntu0.15.04.1 0
500 vivid-updates/main armhf Packages
37.0+build2-0ubuntu1 0
500 vivid/main armhf Packages

More about the Libertine Containers

As noted, the container is a directory containing an Ubuntu rootfs. Container directories are here:

phablet@ubuntu-phablet:~/.cache/libertine-container$ pwd
phablet@ubuntu-phablet:~/.cache/libertine-container$ ls
my-container puritine
phablet@ubuntu-phablet:~/.cache/libertine-container$ cd my-container/
phablet@ubuntu-phablet:~/.cache/libertine-container/my-container$ ls

You can get a bash shell into the container as follows:

phablet@ubuntu-phablet:~/.cache/libertine-container/my-container$ libertine-container-manager exec --command "/bin/bash" --id my-container
groups: cannot find name for group ID 1001

Read more
Michael Hall

I’ve had a Nexus 4 since 2013, and I’ve been using it to test out desktop convergence (where you run a desktop environment from the phone) ever since that feature landed just over a year ago. Usually that meant plugging it into my TV via HDMI to make sure it automatically switched to the larger screen, and playing a bit with the traditional windowed-mode of Unity 8, or checking on adaptive layouts in some of the apps. I’ve also run it for hours on end as a demo at conferences such as SCaLE, FOSSETCON, OSCON and SELF. But through all that, I’ve never used it as an actual replacement for my laptop. Until now.

Thanks Frontier

A bit of back-story first. I had been a Verizon FiOS customer for years, and recently they sold all of their FiOS business to Frontier. The transition has been…..less than ideal. A couple of weeks ago I lost all services (phone, TV and internet) and was eventually told that nobody would be out to fix it until the following day. I still had my laptop, but without internet access I couldn’t really do my job on it. And while Ubuntu on phones can offer up a Hotspot, that particular feature doesn’t work on the Nexus 4 (something something, driver, something). Which meant that the only device that I had which could get online was my phone.

No Minecraft for you

13528720_10154238389913419_2608531900571217522_nFortunately, the fact that I’ve been demoing convergence at conferences meant I had all of the equipment I needed to turn my phone into a desktop and keep right on working. I have a bluetooth mouse and keyboard, and a Slimport adapter that let’s me plug it into a bigger screen. But while a TV works for testing, it’s not really great for long-term work. Don’t get me wrong, working from the couch is nice, but the screen is just too far away for reading and writing. Fortunately for me, and unfortunately for my children, their computer is at a desk and is plugged into a monitor with HDMI ports. So I took it over for the day. They didn’t have internet either that day, so they didn’t miss out on much right?

A day of observations

Throughout the day I posted a series of comments on Google+ about my experience. You could go through my post history looking for them, but I’m not going to make you do that. So here’s a quick summary of what I learned:

  • 3G is not nearly fast enough for my daily work. It’s good when using my phone as a phone, doing one thing at a time. But it falls short of broadband when I’ve got a lot of things using it. Still, on that day it was better than my fiber optic service, so there’s that.
  • I had more apps installed on my phone than I thought I did. I was actually taken aback when I opened the Dash in desktop mode and I saw so many icons. It’s far more than I had on Android, though not quite as many as on my laptop.
  • Having a fully-functional Terminal is a lifesaver. I do a lot of my work from the terminal, including IRC, and having one with tabs and keyboard shortcuts for them is a must for me to work.
  • I missed having physical buttons on my keyboard for home/end and page up/down. Thankfully a couple of people came to my rescue in the comments and taught me other combinations to get those.
  • Unity 8 is Unity. Almost all of the keyboard shortcuts that have become second nature to me (an there are a lot of them) were there. There was no learning curve, I didn’t have to change how I did anything or teach myself something new.
  • The phone is still a phone. I got a call (from Frontier, reminding me about an appointment that never happened) while using the device as a desktop. It was a bit disorienting at first, I had forgotten that I was running the desktop the Nexus 4, so when a notification of an incoming call popped up on the screen I didn’t know what was happening. That only lasted a second though, and after clicking answer and picking up the device, I just used it as a phone. Pretty cool


Must go faster

While I was able to do pretty much all of my work that day thanks to my phone, it wasn’t always easy or fun, and I’m not ready to give up my laptop just yet. The Nexus 4 is simply not powerful enough for the kind of workload I was putting on it. But then again, it’s a nearly 4 year old phone, and wasn’t considered a powerhouse even when it was released. The newest Ubuntu phone on the market, the Meizu Pro 5, packs a whole lot more power, and I think it would be able to give a really nice desktop experience.

Read more
Dustin Kirkland

I had the honor and privilege a couple of weeks ago, to participate in a recording of The Changelog, a podcast dedicated to Open Source technology.

You can listen to it here.

These guys -- Jerod and Adam -- produce a fantastic show, and we covered a lot of ground!

Give it a listen, and follow the links at the bottom of their page (their site is hosted on Ubuntu, of course!) to learn more.


Read more
Carla Berkers

OpenStack is the leading open cloud platform, and Ubuntu is the world’s most popular operating system for OpenStack. Over the past two years we have created a tool that allows users to build an Ubuntu OpenStack cloud on their own hardware in a few simple steps: Autopilot.

This post covers the design process we followed on our journey from alpha to beta to release.

Alpha release: getting the basics right

We started by mapping out a basic Autopilot journey based on stakeholder requirements and designed a first cut of all the necessary steps to build a cloud:

  1. Choose the cloud configuration from a range of OpenStack optionsChoose cloud configuration
  1. Select the hardware the cloud should be built on
    Select the hardware
  1. View deployment status while the cloud is being built
    View deployment status
  1. Monitor the status and usage of the cloud
    Monitor Cloud

After the initial design phase Autopilot was developed and released as an alpha and a beta. This means that for over a year, there was a product to play around with, test and improve before it was made generally available.

Beta release: feedback and improvements

Providing a better overview: increased clarity in the dashboard

Almost immediately after the engineering team started building our new designs, we discovered that we needed to display an additional set of data on the storage graphs. On top of that, some guerilla testing sessions with Canonical engineers brought to light that the CPU and the storage graphs were easily misinterpreted.


After some more competitive research and exploratory sketching, we decided to merge the graphs for each section by putting the utilisation on a vertical axis and the time on the horizontal axis. This seemed to improve the experience for our engineers, but we also wanted to validate with users in usability testing, so we tested the designs with eight participants that were potential Autopilot users. From this testing we learned to include more information on the axes and to include detailed information on hover.

The current graphs are quite an evolution compared to what we started with:
Improved dashboard graphs

Setting users up for success: information and help before the process begins

Before a user gets to the Autopilot wizard, they have to configure their hardware, install an application called MAAS to register machines and install Landscape to get access to Autopilot. A third tool called Juju is installed to help Autopilot behind the scenes.

All these bits of software work together to allow users to build their clouds; however, they are all developed as stand-alone products by different teams. This means that during the initial design phase, it was a challenge to map out the entire journey and get a good idea of how the different components work together.

Only when the Autopilot beta was released, was it finally possible for us to find some hardware and go through the entire journey ourselves, step by step. This really helped us to identify common roadblocks and points in the journey where more documentation or in-app explanation was required.

Increasing transparency of the process: helping users anticipate what they need and when configuration is complete

Following our walk-through, we identified a number of points in the Autopilot journey where contextual help was required. In collaboration with the engineering team we gathered definitions of technical concepts, technical requirement, and system restrictions.

Autopilot walk-through

Based on this info, we made adjustments to the UI. We designed a landing page  with a checklist and introduction copy, and we added headings, help text, and tooltips to the installation and dashboard page. We also included a summary panel on the configuration page, to guide users through the journey and provide instant feedback.


GA release: getting Autopilot ready for the general public

Perhaps the most rewarding type of feedback we gathered from the beta release — our early customers liked Autopilot but wanted more features. From the first designs Autopilot has aimed to help users quickly set up a test cloud. But to use Autopilot to build a production cloud, additional features were required.

Testing without the hardware: try Autopilot on VMware

One of the biggest improvements for GA release was making it easy to try Autopilot, even for people that don’t have enough spare hardware to build a cloud. Our solution: try Autopilot using VMware!

Supporting customisation:  user-defined roles for selected hardware

In the alpha version a user could already select nodes, but in most enterprises users want more flexibility. Often there are different types of hardware for different roles in the cloud, so users don’t always want to automatically distribute all the OpenStack services over all the machines. We designed the ability to choose specific roles like storage or compute for machines, to allow users to make the most of their hardware.

Machine roles

Allowing users more control: a scalable cloud on monitored hardware

The first feature we added was the ability to add hardware to the cloud. This makes it possible to grow a small test cloud into a production sized solution. We also added the ability to integrate the cloud with Nagios, a common monitoring tool. This means if something happens on any of the cloud hardware, users would receive a notification through their existing monitoring system.


The benefits of early release

This month we are celebrating another  release of OpenStack Autopilot. In the two years since we started designing Autopilot, we have been able to add many improvements and it has been a great experience for us as designers to contribute to a maturing product.

We will continue to iterate and refine the features that are launched and we’re currently mapping the roadmap for the months ahead. Our goal remains for Autopilot to be a tool for users to maintain and upgrade an enterprise grade cloud that can be at the core of their operations.


Read more
Dustin Kirkland

A few years ago, I wrote and released a fun little script that would carve up an Ubuntu Byobu terminal into a bunch of splits, running various random command line status utilities.

100% complete technical mumbo jumbo.  The goal was to turn your terminal into something that belongs in a Hollywood hacker film.

I am proud to see it included in this NBCNews piece about "Ransomware".  All of the screenshots, demonstrating what a "hacker" is doing with a system are straight from Ubuntu, Byobu, and Hollywood!

Here are a few screenshots, and the video is embedded below...


Read more
Michael Hall

screenshot20160506_103257823During the Ubuntu Online Summit last week, my colleague Daniel Holbach came up with what he called a “10 day challenge” to some of the engineering manager directing the convergence work in Ubuntu. The idea is simple, try and use only the Unity 8 desktop for 10 working days (two weeks). I thought this was a great way to really identify how close it is to being usable by most Ubuntu users, as well as finding the bugs that cause the most pain in making the switch. So on Friday of last week, with UOS over, I took up the challenge.

Below I will discuss all of the steps that I went through to get it working to my needs. They are not the “official” way of doing it (there isn’t an official way to do all this yet) and they won’t cover every usage scenario, just the ones I faced. If you want to try this challenge yourself they will help you get started. If at any time you get stuck, you can find help in the #ubuntu-unity channel on Freenode, where the developers behind all of these components are very friendly and helpful.

Getting Unity 8

To get started you first need to be on the latest release of Ubuntu. I am using Ubuntu 16.04 (Xenial Xerus), which is the best release for testing Unity 8. You will also need the stable-phone-overlay PPA. Don’t let the name fool you, it’s not just for phones, but it is where you will find the very latest packages for Mir, Unity 8, Libertine and other components you will need. You can install is with this command:

sudo add-apt-repository ppa:ci-train-ppa-service/stable-phone-overlay

Then you will need to install the Unity 8 session package, so that you can select it from the login screen:

sudo apt install unity8-desktop-session

Note: The package above used to be unity8-desktop-session-mir but was renamed to just unity-desktop-session.

When I did this there was a bug in the libhybris package that was causing Mir to try and use some Android stuff, which clearly isn’t available on my laptop. The fix wasn’t yet in the PPA, so I had to take the additional step of installing a fix from our continuous integration system (Note: originally the command below used silo 53, but I’ve been told it is now in silo 31). If you get a black screen when trying to start your Unity 8 session, you probably need this too.

sudo apt-get install phablet-tools phablet-tools-citrain
citrain host-upgrade 031

Note: None of the above paragraph is necessary anymore.

This was enough to get Unity 8 to load for me, but all my apps would crash within a half second of being launched. It turned out to be a problem with the cgroups manager, specifically the cgmanager service was disabled for me (I suspect this was leftover configurations from previous attempts at using Unity 8). After re-enabling it, I was able to log back into Unity 8 and start using apps!

sudo systemctl enable cgmanager

Essential Core Apps

The first thing you’ll notice is that you don’t have many apps available in Unity 8. I had probably more than most, having installed some Ubuntu SDK apps natively on my laptop already. If you haven’t installed the webbrowser-app already, you should. It’s in the Xenial archive and the PPA you added above, so just

sudo apt install webbrowser-app

But that will only get you so far. What you really need are a terminal and file manager. Fortunately those have been created as part of the Core Apps project, you just need to install them. Because the Ubuntu Store wasn’t working for me (see bottom of this post) I had to manually download and install them:

sudo click install --user mhall
sudo click install --user mhall

If you want to use these apps in Unity 7 as well, you have to modify their .desktop files located in ~/.local/share/applications/ and add the -x flag after aa-exec-click, this is because by default it prevents running these apps under X11 where they won’t have the safety of confinement that they get under Mir.

The file manager needed a bit of extra effort to get working. It contains many Samba libraries that allow it to access windows network shares, but for some reason the app was looking for them in the wrong place. As a quick and dirty hack, I ended up copying whatever libraries it needed from /opt/ to /usr/lib/i386-linux-gnu/samba/. It’s worth the effort, though, because you need the file manager if you want do things like upload files through the webbrowser.

Using SSH

IRC is a vital communication tool for my job, we all use it every day. In fact, I find it so important that I have a remote client that stays connected 24/7, which I connect to via ssh. Thanks to the Terminal core app, I have quick and easy access to that. But when I first tried to connect to my server, which uses public-key authentication (as they all should), my connection was refused. That is because the Unity 8 session doesn’t run the ssh-agent service on startup. You can start it manually from the terminal:


This will output some shell commands to setup environment variables, copy those and paste them right back into your terminal to set them. Then you should be able to ssh like normal, and if your key needs a passphrase you will be prompted for it in the terminal rather than in a dialog like you get in Unity 7.

Getting traditional apps

Now that you’ve got some apps running natively on Mir, you probably want to try out support for all of your traditional desktop apps, as you’ve heard advertised. This is done by a project called Libertine, which creates an LXC container and XMir to keep those unconfined apps safely away from your new properly confined setup. The first thing you will need to do is install the libertine packages:

apt-get install libertine libertine-scope

screenshot20160506_105035760Once you have those, you will see a Libertine app in your Apps scope. This is the app that lets you manage your Libertine containers (yes, you can have more than one), and install apps into them. Creating a new container is simply a matter of pressing the “Install” button. You can give it a name of leave it blank to get the default “Xenial”.

screenshot20160506_105618896Once your container is setup, you can install as many apps into it as you want, again using the Libertine container manager. You can even use it to search the archives if you don’t know the exact package name. It will also install any dependencies that package needs into your Libertine container.

screenshot20160506_105942480Now that you have your container setup and apps installed into it, you are ready to start trying them out. For now you have to access them from a separate scope, since the default Apps scope doesn’t look into Libertine containers. That is why you had to install the libertine-scope package above. You can find this scope by clicking on the Dash’s bottom edge indicator to open the Scopes manger, and selecting the Legacy Applications Scope. There you will see launchers for the apps you have installed.

Libertine uses a special container manager to launch apps. If it isn’t running, as was the case for me, your legacy app windows will remain black. To fix that, open up the terminal and manually start the manager:

initctl --session start libertine-lxc-manager

Theming traditional apps

screenshot20160506_122713187By default the legacy apps don’t look very nice. They default to the most basic of themes that look like you’ve time-traveled back to the mid-1990s, and nobody wants to do that. The reason for this is because these apps (or rather, the toolkit they use) expect certain system settings to tell them what theme to use, but those settings aren’t actually a dependency of the application’s package. They are part of a default desktop install, but not part of the default Libertine image.

screenshot20160506_112259969I found a way to fix this, at least for some apps, by installing the light-themes and ubuntu-settings packages into the Libertine container. Specifically it should work for any Gtk3 based application, such as GEdit. It does not, however, work for apps that still use the Gtk2 toolkit, such as Geany. I have not dug deeper to try and figure out how to fix Gtk2 themes, if anybody has a suggestion please leave it in the comments.

What works

It has been a couple of months since I last tried the Unity 8 session, back before I upgraded to Xenial, and at that time there wasn’t much working. I went into this challenge expecting it to be better, but not by much. I honestly didn’t expect to spend even a full day using it. So I was really quite surprised to find that, once I found the workarounds above, I was not only able to spend the full day in it, but I was able to do so quite easily.

screenshot20160509_121832656Whenever you have a new DE (which Unity 8 effectively is) and the latest UI toolkit (Qt 5) you have to be concerned about performance and resource use, and given the bleeding-edge nature of Unity 8 on the desktop, I was expecting to sacrifice some CPU cycles, battery life and RAM. If anything, the opposite was the case. I get at least as many hours on my battery as I do with Unity 7, and I was using less than half the RAM I typically do.

screenshot20160509_103139434Moreover, things that I was expecting to cause me problems surprisingly didn’t. I was able to use Google Hangouts for my video conferences, which I knew had just been enabled in the browser. But I fully expected suspend/resume to have trouble with Mir, given the years I spent fighting it in X11 in the past, but it worked nearly flawlessly (see below). The network indicator had all of my VPN configurations waiting to be used, and they worked perfectly. Even pulse audio was working as well as it did in Unity 7, though this did introduce some problems (again, see below). It even has settings to adjust the mouse speed and disable the trackpad when I’m typing. Most imporantly, nearly all of the keyboard shortcuts that have become subconcious to me in Unity 7 are working in Unity 8.

Most importantly, I was able to write this blog post from Unity 8. That includes taking all of the screenshots and uploading them to WordPress. Switching back and forth between my browser and my notes document to see what I had done over the last few days, or going to the terminal to verify the commands I mentioned above.

What doesn’t

Of course, it wasn’t all unicorns and rainbows, Unity 8 is still very bleeding edge as a desktop shell, and if you want to use it you need to be prepared for some pain. None of it has so far been bad enough to stop me, but your mileage may vary.

One of the first minor pain-points is the fact that middle-click doesn’t paste the active text highlight. I hadn’t realized how much I have become dependent on that until I didn’t have it. You also can’t copy/paste between a Mir and an XMir window, which makes legacy apps somewhat less useful, but that’s on the roadmap to be fixed.

Speaking of windows, Unity 8 is still limited to one per app. This is going to change, but it is the current state of things. This doesn’t matter so much for native apps, which were build under this restriciton, and the terminal app having tabs was a saving grace here. But for legacy apps it presents a bigger issue, especially apps like GTG (Getting Things Gnome) where multi-window is a requirement.

Some power-management is missing too, such as dimming the screen after some amount of inactivity, or turning it off altogether. The session also will not lock when you suspend it, so don’t depend on this in a security-critical way (but really, if you’re running bleeding-edge desktops in security-critical environments, you have bigger problems).

I also had a minor problem with my USB headset. It’s actually a problem I have in Unity 7 too, since upgrading to Xenial the volume and mute controls don’t automatically switch to the headset, even though the audio output and input do. I had a workaround for that in Unity 7, I could open the sound settings and manually change it to the headset, at which point the controls work on it. But in Unity 8’s sound settings there is no such option, so my workaround isn’t available.

The biggest hurdle, from my perspective, was not being able to install apps from the store. This is due to something in the store scope, online accounts, or Ubuntu One, I haven’t figured out which yet. So to install anything, I had to get the .click package and do it manually. But asking around I seem to be the only one having this problem, so those of you who want to try this yourself may not have to worry about that.

The end?

No, not for me. I’m on day 3 of this 10 day challenge, and so far things are going well enough for me to continue. I have been posting regular small updates on Google+, and will keep doing so. If I have enough for a new blog post, I may write another one here, but for the most part keep an eye on my G+ feed. Add your own experiences there, and again join #ubuntu-unity if you get stuck or need help.

Read more
Dustin Kirkland

Below you can find the audio/video recording of my OpenStack Austin presentation, where I demonstrated Ubuntu OpenStack Mitaka, running on top of Ubuntu 16.04 LTS, entirely within LXD machine containers.  You can also download the PDF of the slides here.  And there are a number of other excellent talks here!


Read more
Dustin Kirkland

I'm delighted to share the slides from our joint IBM and Canonical webinar about Ubuntu on IBM POWER8 and LinuxOne servers.  You can download the PDF here, watch the recording here, or tab through the slides or watch the video embedded below.  Enjoy!


Read more
Dustin Kirkland

I'm thrilled to introduce Docker 1.10.3, available on every Ubuntu architecture, for Ubuntu 16.04 LTS, and announce the General Availability of Ubuntu Fan Networking!

That's Ubuntu Docker binaries and Ubuntu Docker images for:
  • armhf (rpi2, et al. IoT devices)
  • arm64 (Cavium, et al. servers)
  • i686 (does anyone seriously still run 32-bit intel servers?)
  • amd64 (most servers and clouds under the sun)
  • ppc64el (OpenPower and IBM POWER8 machine learning super servers)
  • s390x (IBM System Z LinuxOne super uptime mainframes)
That's Docker-Docker-Docker-Docker-Docker-Docker, from the smallest Raspberry Pi's to the biggest IBM mainframes in the world today!  Never more than one 'sudo apt install' command away.

Moreover, we now have Docker running inside of LXD!  Containers all the way down.  Application containers (e.g. Docker), inside of Machine containers (e.g. LXD), inside of Virtual Machines (e.g. KVM), inside of a public or private cloud (e.g. Azure, OpenStack), running on bare metal (take your pick).

Let's have a look at launching a Docker application container inside of a LXD machine container:

kirkland@x250:~⟫ lxc launch ubuntu-daily:x -p default -p docker
Creating magical-damion
Starting magical-damion
kirkland@x250:~⟫ lxc list | grep RUNNING
| magical-damion | RUNNING | (eth0) | | PERSISTENT | 0 |
kirkland@x250:~⟫ lxc exec magical-damion bash
root@magical-damion:~# apt update >/dev/null 2>&1 ; apt install -y >/dev/null 2>&1
root@magical-damion:~# docker run -it ubuntu bash
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
759d6771041e: Pull complete
8836b825667b: Pull complete
c2f5e51744e6: Pull complete
a3ed95caeb02: Pull complete
Digest: sha256:b4dbab2d8029edddfe494f42183de20b7e2e871a424ff16ffe7b15a31f102536
Status: Downloaded newer image for ubuntu:latest
root@0577bd7d5db1:/# ifconfig eth0
eth0 Link encap:Ethernet HWaddr 02:42:ac:11:00:02
inet addr: Bcast: Mask:
inet6 addr: fe80::42:acff:fe11:2/64 Scope:Link
RX packets:16 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1296 (1.2 KB) TX bytes:648 (648.0 B)

Oh, and let's talk about networking...  We're also pleased to announce the general availability of Ubuntu Fan networking -- specially designed to connect all of your Docker containers spread across your network.  Ubuntu's Fan networking feature is an easy way to make every Docker container on your local network easily addressable by every other Docker host and container on the same network.  It's high performance, super simple, utterly deterministic, and we've tested it on every major public cloud as well as OpenStack and our private networks.

Simply installing Ubuntu's Docker package will also install the ubuntu-fan package, which provides an interactive setup script, fanatic, should you choose to join the Fan.  Simply run 'sudo fanatic' and answer the questions.  You can trivially revert your Fan networking setup easily with 'sudo fanatic deconfigure'.

kirkland@x250:~$ sudo fanatic 
Welcome to the fanatic fan networking wizard. This will help you set
up an example fan network and optionally configure docker and/or LXD to
use this network. See fanatic(1) for more details.
Configure fan underlay (hit return to accept, or specify alternative) []:
Configure fan overlay (hit return to accept, or specify alternative) []:
Create LXD networking for underlay: overlay: [Yn]: n
Create docker networking for underlay: overlay: [Yn]: Y
Test docker networking for underlay: overlay:
(NOTE: potentially triggers large image downloads) [Yn]: Y
local docker test: creating test container ...
test master: ping test ( ...
test slave: ping test ( ...
test master: ping test ... PASS
test master: short data test ( -> ...
test slave: ping test ... PASS
test slave: short data test ( -> ...
test master: short data ... PASS
test slave: short data ... PASS
test slave: long data test ( -> ...
test master: long data test ( -> ...
test master: long data ... PASS
test slave: long data ... PASS
local docker test: destroying test container ...
local docker test: test complete PASS (master=0 slave=0)
This host IP address:

I've run 'sudo fanatic' here on a couple of machines on my network -- x250 ( and masterbr (, and now I'm going to launch a Docker container on each of those two machines, obtain each IP address on the Fan (250.x.y.z), install iperf, and test the connectivity and bandwidth between each of them (on my gigabit home network).  You'll see that we'll get 900mbps+ of throughput:

kirkland@x250:~⟫ sudo docker run -it ubuntu bash
root@c22cf0d8e1f7:/# apt update >/dev/null 2>&1 ; apt install -y iperf >/dev/null 2>&1
root@c22cf0d8e1f7:/# ifconfig eth0
eth0 Link encap:Ethernet HWaddr 02:42:fa:00:2d:00
inet addr: Bcast: Mask:
inet6 addr: fe80::42:faff:fe00:2d00/64 Scope:Link
RX packets:6423 errors:0 dropped:0 overruns:0 frame:0
TX packets:4120 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:22065202 (22.0 MB) TX bytes:227225 (227.2 KB)

root@c22cf0d8e1f7:/# iperf -c
multicast ttl failed: Invalid argument
Client connecting to, TCP port 5001
TCP window size: 45.0 KByte (default)
[ 3] local port 54274 connected with port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 1.05 GBytes 902 Mbits/sec

And the second machine:
kirkland@masterbr:~⟫ sudo docker run -it ubuntu bash
root@effc8fe2513d:/# apt update >/dev/null 2>&1 ; apt install -y iperf >/dev/null 2>&1
root@effc8fe2513d:/# ifconfig eth0
eth0 Link encap:Ethernet HWaddr 02:42:fa:00:08:00
inet addr: Bcast: Mask:
inet6 addr: fe80::42:faff:fe00:800/64 Scope:Link
RX packets:7659 errors:0 dropped:0 overruns:0 frame:0
TX packets:3433 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:22131852 (22.1 MB) TX bytes:189875 (189.8 KB)

root@effc8fe2513d:/# iperf -s
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
[ 4] local port 5001 connected with port 54274
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 1.05 GBytes 899 Mbits/sec

Finally, let's have another long hard look at the image from the top of this post.  Download it in full resolution to study very carefully what's happening here, because it's pretty [redacted] amazing!

Here, we have a Byobu session, split into 6 panes (Shift-F2 5x Times, Shift-F8 6x times).  In each pane, we have an SSH session to Ubuntu 16.04 LTS servers spread across 6 different architectures -- armhf, arm64, i686, amd64, ppc64el, and s390x.  I used the Shift-F9 key to simultaneously run the same commands in each and every window.  Here are the commands I ran:

lxc launch ubuntu-daily:x -p default -p docker
lxc list | grep RUNNING
uname -a
dpkg -l | grep
sudo docker images | grep -m1 ubuntu
sudo docker run -it ubuntu bash
apt update >/dev/null 2>&1 ; apt install -y net-tools >/dev/null 2>&1
ifconfig eth0

That's right.  We just launched Ubuntu LXD containers, as well as Docker containers against every Ubuntu 16.04 LTS architecture.  How's that for Ubuntu everywhere!?!

Ubuntu 16.04 LTS will be one hell of a release!


Read more
Dustin Kirkland

I happen to have a full mirror of the entire Ubuntu Xenial archive here on a local SSD, and I took the opportunity to run a few numbers...
  • 6: This is our 6th Ubuntu LTS
    • 6.06, 8.04, 10.04, 12.04, 14.04, 16.04
  • 7: With Ubuntu 16.04 LTS, we're supporting 7 CPU architectures
    • armhf, arm64, i386, amd64, powerpc, ppc64el, s390x
  • 25,671: Ubuntu 16.04 LTS is comprised of 25,671 source packages
    • main, universe, restricted, multiverse
  • 150,562+: Over 150,562 (and counting!) cloud instances of Xenial have launched to date
    • and we haven't even officially released yet!
  • 216,475: A complete archive of all binary .deb packages in Ubuntu 16.04 LTS consists of 216,475 debs.
    • 24,803 arch independent
    • 27,159 armhf
    • 26,845 arm64
    • 28,730 i386
    • 28,902 amd64
    • 27,061 powerpc
    • 26,837 ppc64el
    • 26,138 s390x
  • 1,426,792,926: A total line count of all source packages in Ubuntu 16.04 LTS using cloc yields 1,426,792,926 total lines of source code
  • 250,478,341,568: A complete archive all debs, all architectures in Ubuntu 16.04 LTS requires 250GB of disk space
Yes, that's 1.4 billion lines of source code comprising the entire Ubuntu 16.04 LTS archive.  What an amazing achievement of open source development!

Perhaps my fellow nerds here might be interested in a breakdown of all 1.4 billion lines across 25K source packages, and throughout 176 different programming languages, as measured by Al Danial's cloc utility.  Interesting data!

You can see the full list here.  What further insight can you glean?


Read more
Dustin Kirkland

On July 7, 2010, I received the above email.  In hindsight, this note effectively changed the landscape of cloud computing forever.  I was one of 3 Canonical employees in attendance (Nick Barcet, Neil Levine) and among a number former colleagues (Theirry Carrez, Soren Hansen, Rick Clark) at the first OpenStack Design Summit at the Omni hotel in Austin, Texas, in July of 2010.

These are the only pictures I snapped with my phone (metadata says it was an HTC Hero) of the event, which, almost unbelievably fit entirely within a single conference room :-)

The "fishbowl" round table discussion format was modeled after Ubuntu Developer Summits.

It was so much fun to see so many unfamiliar, non-Ubuntu people using the fishbowl discussion format.

Also borrowed from Ubuntu Developer Summits was the collaborative, community-sourced note taking in Etherpad-Lite.

Breakfast, in the beautiful Omni lobby.

Lots of natural light, but thankfully, air conditioned.  By the way, does anyone have pictures from the 120oF Whole Foods roof top event?

My, my, my, how far we've come in 6 short years!

This month's OpenStack Summit returns to Austin, Texas, and fills the entire Austin Convention Center, and overflows into at least two nearby hotels, with 5,000+ OpenStack developers, users, and enthusiasts!

In fact, if you're reading this post on, you're being served by Wordpress and MySQL hosted on a production Ubuntu OpenStack at Canonical.

Welcome back home, OpenStack!


Read more
Dustin Kirkland

As announced last week, Microsoft and Canonical have worked together to bring Ubuntu's userspace natively into Windows 10.

As of today, Windows 10 Insiders can now take Ubuntu on Windows for a test drive!  Here's how...

1) You need to have a system running today's 64-bit build of Windows 10 (Build 14316).

2) To do so, you may need to enroll into the Windows Insider program here,

3) You need to notify your Windows desktop that you're a Windows Insider, under "System Settings --> Advanced Windows Update options"

4) You need to set your update ambition to the far right, also known as "the fast ring".

5) You need to enable "developer mode", as this new feature is very pointedly directed specifically at developers.

6) You need to check for updates, apply all updates, and restart.

7) You need to turn on the new Windows feature, "Windows Subsystem for Linux (Beta)".  Note (again) that you need a 64-bit version of Windows!  Without that, you won't see the new option.

8) You need to reboot again.  (Windows sure has a fetish for rebooting!)

9) You press the start button and type "bash".

10) The first time you run "bash.exe", you'll accept the terms of service, download Ubuntu, and then you're off and running!

If you screw something up, and you want to start over, simply open a Windows command shell, and run: lxrun /uninstall /full and then just run bash again.

For bonus points, you might also like to enable the Ubuntu monospace font in your console.  Here's how!

a) Download the Ubuntu monospace font, from

b) Install the Ubuntu monospace font, by opening the zip file you downloaded, finding UbuntuMono-R.ttf, double clicking on it, and then clicking Install.

c) Enable the Ubuntu monospace font for the command console in the Windows registry.  Open regedit and find this key: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Console\TrueTypeFont and add a new string value name "000" with value data "Ubuntu Mono"

d) Edit your command console preferences to enable the Ubuntu monospace font.


Read more
Dustin Kirkland

This makes me so incredibly happy!

Here's how...

First, start with a fully up-to-date Ubuntu 16.04 LTS desktop.

sudo apt update
sudo apt dist-upgrade -y

Then, install dconf-editor.

sudo apt install -y dconf-editor

Launch dconf-editor and find the "launcher" key and change it to "bottom".


For good measure, I triggered a reboot, to make sure my changes stuck.  And voila!  Beauty!


Read more
Michael Hall

Somehow I missed the fact that I never wrote Community Donations report for Q3 2015. I only realized it because it’s time for me to start working on Q4. Sorry for the oversight, but that report is now published.

The next report should be out soon, in the mean time you can look at all of the past reports so see the great things we’ve been able to do with and for the Ubuntu community through this program. Everybody who has recieved these funds have used them to contribute to the project in one way or another, and we appreciate all of their work.

As you may notice, we’ve been regularly paying out more than we’ve been getting in donations. While we’ve had a carry-over balance ever since we started this program, that balance is running down. If you like the things we’ve been able to support with this program, please consider sending it a contribution and helping us spread the word about it.


Read more
Dustin Kirkland

Update: Here's how to get started using Ubuntu on Windows

See also Scott Hanselman's blog here
I'm in San Francisco this week, attending Microsoft's Build developer conference, as a sponsored guest of Microsoft.

That's perhaps a bit odd for me, as I hadn't used Windows in nearly 16 years.  But that changed a few months ago, as I embarked on a super secret (and totally mind boggling!) project between Microsoft and Canonical, as unveiled today in a demo during Kevin Gallo's opening keynote of the Build conference....

An Ubuntu user space and bash shell, running natively in a Windows 10 cmd.exe console!

Did you get that?!?  Don't worry, it took me a few laps around that track, before I fully comprehended it when I first heard such crazy talk a few months ago :-)

Here's let's break it down slowly...
  1. Windows 10 users
  2. Can open the Windows Start menu
  3. And type "bash" [enter]
  4. Which opens a cmd.exe console
  5. Running Ubuntu's /bin/bash
  6. With full access to all of Ubuntu user space
  7. Yes, that means apt, ssh, rsync, find, grep, awk, sed, sortxargs, md5sum, gpg, curl, wget, apache, mysql, python, perl, ruby, php, gcc, tar, vim, emacs, diff, patch...
  8. And most of the tens of thousands binary packages available in the Ubuntu archives!
"Right, so just Ubuntu running in a virtual machine?"  Nope!  This isn't a virtual machine at all.  There's no Linux kernel booting in a VM under a hypervisor.  It's just the Ubuntu user space.

"Ah, okay, so this is Ubuntu in a container then?"  Nope!  This isn't a container either.  It's native Ubuntu binaries running directly in Windows.

"Hum, well it's like cygwin perhaps?"  Nope!  Cygwin includes open source utilities are recompiled from source to run natively in Windows.  Here, we're talking about bit-for-bit, checksum-for-checksum Ubuntu ELF binaries running directly in Windows.

[long pause]

"So maybe something like a Linux emulator?"  Now you're getting warmer!  A team of sharp developers at Microsoft has been hard at work adapting some Microsoft research technology to basically perform real time translation of Linux syscalls into Windows OS syscalls.  Linux geeks can think of it sort of the inverse of "wine" -- Ubuntu binaries running natively in Windows.  Microsoft calls it their "Windows Subsystem for Linux".  (No, it's not open source at this time.)

Oh, and it's totally shit hot!  The sysbench utility is showing nearly equivalent cpu, memory, and io performance.

So as part of the engineering work, I needed to wrap the stock Ubuntu root filesystem into a Windows application package (.appx) file for suitable upload to the Windows Store.  That required me to use Microsoft Visual Studio to clone a sample application, edit a few dozen XML files, create a bunch of icon .png's of various sizes, and so on.

Not being Windows developer, I struggled and fought with Visual Studio on this Windows desktop for a few hours, until I was about ready to smash my coffee mug through the damn screen!

Instead, I pressed the Windows key, typed "bash", hit enter.  Then I found the sample application directory in /mnt/c/Users/Kirkland/Downloads, and copied it using "cp -a".  I used find | xargs | rename to update a bunch of filenames.  And a quick grep | xargs | sed to comprehensively search and replace s/SampleApp/UbuntuOnWindows/. And Ubuntu's convert utility quickly resized a bunch of icons.   Then I let Visual Studio do its thing, compiling the package and uploading to the Windows Store.  Voila!

Did you catch that bit about /mnt/c...  That's pretty cool...  All of your Windows drives, like C: are mounted read/write directly under /mnt.  And, vice versa, you can see all of your Ubuntu filesystem from Windows Explorer in C:\Users\Kirkland\AppData\Local\Lxss\rootfs\

Meanwhile, I also needed to ssh over to some of my other Ubuntu systems to get some work done.  No need for Putty!  Just ssh directly from within the Ubuntu shell.

Of course apt install and upgrade as expected.

Is everything working exactly as expected?  No, not quite.  Not yet, at least.  The vast majority of the LTP passes and works well.  But there are some imperfections still, especially around tty's an the vt100.  My beloved byobu, screen, and tmux don't quite work yet, but they're getting close!

And while the current image is Ubuntu 14.04 LTS, we're expecting to see Ubuntu 16.04 LTS replacing Ubuntu 14.04 in the Windows Store very, very soon.

Finally, I imagine some of you -- long time Windows and Ubuntu users alike -- are still wondering, perhaps, "Why?!?"  Having dedicated most of the past two decades of my career to free and open source software, this is an almost surreal endorsement by Microsoft on the importance of open source to developers.  Indeed, what a fantastic opportunity to bridge the world of free and open source technology directly into any Windows 10 desktop on the planet.  And what a wonderful vector into learning and using more Ubuntu and Linux in public clouds like Azure.  From Microsoft's perspective, a variety of surveys and user studies have pointed to bash and Linux tools -- very specifically, Ubuntu -- be available in Windows, and without resource-heavy full virtualization.

So if you're a Windows Insider and have access to the early beta of this technology, we certainly hope you'll try it out!  Let us know what you think!

If you want to hear more, hopefully you'll tune into the Channel 9 Panel discussion at 16:30 PDT on March 30, 2016.


Read more