Canonical Voices

Posts tagged with 'canonical'

Dustin Kirkland

Yesterday, I delivered a talk to a lively audience at ContainerWorld in Santa Clara, California.

If I measured "the most interesting slides" by counting "the number of people who took a picture of the slide", then by far "the most interesting slides" are slides 8-11, which pose an answer the question:
"Should I run my PaaS on top of my IaaS, or my IaaS on top of my PaaS"?
In the Ubuntu world, that answer is super easy -- however you like!  At Canonical, we're happy to support:
  1. Kubernetes running on top of Ubuntu OpenStack
  2. OpenStack running on top of Canonical Kubernetes
  3. Kubernetes running along side OpenStack
In all cases, the underlying substrate is perfectly consistent:
  • you've got 1 to N physical or virtual machines
  • which are dynamically provisioned by MAAS or your cloud provider
  • running stable, minimal, secure Ubuntu server image
  • carved up into fast, efficient, independently addressable LXD machine containers
With that as your base, we'll easily to conjure-up a Kubernetes, an OpenStack, or both.  And once you have a Kubernetes or OpenStack, we'll gladly conjure-up one inside the other.

As always, I'm happy to share my slides with you here.  You're welcome to download the PDF, or flip through the embedded slides below.


Read more
Dustin Kirkland

Introducting the Canonical Livepatch Service

Ubuntu 16.04 LTS’s 4.4 Linux kernel includes an important new security capability in Ubuntu -- the ability to modify the running Linux kernel code, without rebooting, through a mechanism called kernel livepatch.

Today, Canonical has publicly launched the Canonical Livepatch Service -- an authenticated, encrypted, signed stream of Linux livepatches that apply to the 64-bit Intel/AMD architecture of the Ubuntu 16.04 LTS (Xenial) Linux 4.4 kernel, addressing the highest and most critical security vulnerabilities, without requiring a reboot in order to take effect.  This is particularly amazing for Container hosts -- Docker, LXD, etc. -- as all of the containers share the same kernel, and thus all instances benefit.

I’ve tried to answer below some questions that you might have. As you have others, you’re welcome
to add them to the comments below or on Twitter with hastag #Livepatch.

Retrieve your token from

Q: How do I enable the Canonical Livepatch Service?

A: Three easy steps, on a fully up-to-date 64-bit Ubuntu 16.04 LTS system.
  1. Go to and retrieve your livepatch token
    1. Install the canonical-livepatch snap
      $ sudo snap install canonical-livepatch 
    2. Enable the service with your token
      $ sudo canonical-livepatch enable [TOKEN] 
    And you’re done! You can check the status at any time using:

    $ canonical-livepatch status --verbose

      Q: What are the system requirements?

      A: The Canonical Livepatch Service is available for the generic and low latency flavors of the 64-bit Intel/AMD (aka, x86_64, amd64) builds of the Ubuntu 16.04 LTS (Xenial) kernel, which is a Linux 4.4 kernel. Canonical livepatches work on Ubuntu 16.04 LTS Servers and Desktops, on physical machines, virtual machines, and in the cloud. The safety, security, and stability firmly depends on unmodified Ubuntu kernels and network access to the Canonical Livepatch Service (  You also will need to apt update/upgrade to the latest version of snapd (at least 2.15).

      Q: What about other architectures?

      A: The upstream Linux livepatch functionality is currently limited to the 64-bit x86 architecture, at this time. IBM is working on support for POWER8 and s390x (LinuxOne mainframe), and there’s also active upstream development on ARM64, so we do plan to support these eventually. The livepatch plumbing for 32-bit ARM and 32-bit x86 are not under upstream development at this time.

      Q: What about other flavors?

      A: We are providing the Canonical Livepatch Service for the generic and low latency (telco) flavors of the the Linux kernel at this time.

      Q: What about other releases of Ubuntu?

      A: The Canonical Livepatch Service is provided for Ubuntu 16.04 LTS’s Linux 4.4 kernel. Older releases of Ubuntu will not work, because they’re missing the Linux kernel support. Interim releases of Ubuntu (e.g. Ubuntu 16.10) are targeted at developers and early adopters, rather than Long Term Support users or systems that require maximum uptime.  We will consider providing livepatches for the HWE kernels in 2017.

      Q: What about derivatives of Ubuntu?

      A: Canonical livepatches are fully supported on the 64-bit Ubuntu 16.04 LTS Desktop, Cloud, and Server operating systems. On other Ubuntu derivatives, your mileage may vary! These are not part of our automated continuous integration quality assurance testing framework for Canonical Livepatches. Canonical Livepatch safety, security, and stability will firmly depend on unmodified Ubuntu generic kernels and network access to the Canonical Livepatch Service.

      Q: How does Canonical test livepatches?

      A: Every livepatch is rigorously tested in Canonical's in-house CI/CD (Continuous Integration / Continuous Delivery) quality assurance system, which tests hundreds of combinations of livepatches, kernels, hardware, physical machines, and virtual machines.  Once a livepatch passes CI/CD and regression tests, it's rolled out on a canary testing basis, first to a tiny percentage of the Ubuntu Community users of the Canonical Livepatch Service. Based on the success of that microscopic rollout, a moderate rollout follows.  And assuming those also succeed, the livepatch is delivered to all free Ubuntu Community and paid Ubuntu Advantage users of the service.  Systemic failures are automatically detected and raised for inspection by Canonical engineers.  Ubuntu Community users of the Canonical Livepatch Service who want to eliminate the small chance of being randomly chosen as a canary should enroll in the Ubuntu Advantage program (starting at $12/month).

      Q: What kinds of updates will be provided by the Canonical Livepatch Service?

      A: The Canonical Livepatch Service is intended to address high and critical severity Linux kernel security vulnerabilities, as identified by Ubuntu Security Notices and the CVE database. Note that there are some limitations to the kernel livepatch technology -- some Linux kernel code paths cannot be safely patched while running. We will do our best to supply Canonical Livepatches for high and critical vulnerabilities in a timely fashion whenever possible. There may be occasions when the traditional kernel upgrade and reboot might still be necessary. We’ll communicate that clearly through the usual mechanisms -- USNs, Landscape, Desktop Notifications, Byobu, /etc/motd, etc.

      Q: What about non-security bug fixes, stability, performance, or hardware enablement updates?

      A: Canonical will continue to provide Linux kernel updates addressing bugs, stability issues, performance problems, and hardware compatibility on our usual cadence -- about every 3 weeks. These updates can be easily applied using ‘sudo apt update; sudo apt upgrade -y’, using the Desktop “Software Updates” application, or Landscape systems management. These standard (non-security) updates will still require a reboot, as they always have.

      Q: Can I rollback a Canonical Livepatch?

      A: Currently rolling-back/removing an already inserted livepatch module is disabled in Linux 4.4. This is because we need a way to determine if we are currently executing inside a patched function before safely removing it. We can, however, safely apply new livepatches on top of each other and even repatch functions over and over.

      Q: What about low and medium severity CVEs?

      A: We’re currently focusing our Canonical Livepatch development and testing resources on high and critical security vulnerabilities, as determined by the Ubuntu Security Team.  We'll livepatch other CVEs opportunistically.

      Q: Why are Canonical Livepatches provided as a subscription service?

      A: The Canonical Livepatch Service provides a secure, encrypted, authenticated connection, to ensure that only properly signed livepatch kernel modules -- and most importantly, the right modules -- are delivered directly to your system, with extremely high quality testing wrapped around it.

      Q: But I don’t want to buy UA support!

      A: You don’t have to! Canonical is providing the Canonical Livepatch Service to community users of Ubuntu, at no charge for up to 3 machines (desktop, server, virtual machines, or cloud instances). A randomly chosen subset of the free users of Canonical Livepatches will receive their Canonical Livepatches slightly earlier than the rest of the free users or UA users, as a lightweight canary testing mechanism, benefiting all Canonical Livepatch users (free and UA). Once those canary livepatches apply safely, all Canonical Livepatch users will receive their live updates.

      Q: But I don’t have an Ubuntu SSO account!

      A: An Ubuntu SSO account is free, and provides services similar to Google, Microsoft, and Apple for Android/Windows/Mac devices, respectively. You can create your Ubuntu SSO account here.

      Q: But I don’t want login to!

      A: You don’t have to! Canonical Livepatch is absolutely not required maintain the security of any Ubuntu desktop or server! You may continue to freely and anonymously ‘sudo apt update; sudo apt upgrade; sudo reboot’ as often as you like, and receive all of the same updates, and simply reboot after kernel updates, as you always have with Ubuntu.

      Q: But I don't have Internet access to!

      A: You should think of the Canonical Livepatch Service much like you think of Netflix, Pandora, or Dropbox.  It's an Internet streaming service for security hotfixes for your kernel.  You have access to the stream of bits when you can connect to the service over the Internet.  On the flip side, your machines are already thoroughly secured, since they're so heavily firewalled off from the rest of the world!

      Q: Where’s the source code?

      A: The source code of livepatch modules can be found here.  The source code of the canonical-livepatch client is part of Canonical's Landscape system management product and is commercial software.

      Q: What about Ubuntu Core?

      A: Canonical Livepatches for Ubuntu Core are on the roadmap, and may be available in late 2016, for 64-bit Intel/AMD architectures. Canonical Livepatches for ARM-based IoT devices depend on upstream support for livepatches.

      Q: How does this compare to Oracle Ksplice, RHEL Live Patching and SUSE Live Patching?

      A: While the concepts are largely the same, the technical implementations and the commercial terms are very different:

      • Oracle Ksplice uses it’s own technology which is not in upstream Linux.
      • RHEL and SUSE currently use their own homegrown kpatch/kgraft implementations, respectively.
      • Canonical Livepatching uses the upstream Linux Kernel Live Patching technology.
      • Ksplice is free, but unsupported, for Ubuntu Desktops, and only available for Oracle Linux and RHEL servers with an Oracle Linux Premier Support license ($2299/node/year).
      • It’s a little unclear how to subscribe to RHEL Kernel Live Patching, but it appears that you need to first be a RHEL customer, and then enroll in the SIG (Special Interests Group) through your TAM (Technical Account Manager), which requires Red Hat Enterprise Linux Server Premium Subscription at $1299/node/year.  (I'm happy to be corrected and update this post)
      • SUSE Live Patching is available as an add-on to SUSE Linux Enterprise Server 12 Priority Support subscription at $1,499/node/year, but does come with a free music video.
      • Canonical Livepatching is available for every Ubuntu Advantage customer, starting at our entry level UA Essential for $150/node/year, and available for free to community users of Ubuntu.

      Q: What happens if I run into problems/bugs with Canonical Livepatches?

      A: Ubuntu Advantage customers will file a support request at where it will be serviced according to their UA service level agreement (Essential, Standard, or Advanced). Ubuntu community users will file a bug report on Launchpad and we'll service it on a best effort basis.

      Q: Why does canonical-livepatch client/server have a proprietary license?

      A: The canonical-livepatch client is part of the Landscape family of tools available to Canonical support customers. We are enabling free access to the Canonical Livepatch Service for Ubuntu community users as a mark of our appreciation for the broader Ubuntu community, and in exchange for occasional, automatic canary testing.

      Q: How do I build my own livepatches?

      A: It’s certainly possible for you to build your own Linux kernel live patches, but it requires considerable skill, time, computing power to produce, and even more effort to comprehensively test. Rest assured that this is the real value of using the Canonical Livepatch Service! That said, Chris Arges has blogged a howto for the curious a while back:

      Q: How do I get notifications of which CVEs are livepatched and which are not?

      A: You can, at any time, query the status of the canonical-livepatch daemon using: ‘canonical-livepatch status --verbose’. This command will show any livepatches successfully applied, any outstanding/unapplied livepatches, and any error conditions. Moreover, you can monitor the Ubuntu Security Notices RSS feed and the ubuntu-security-announce mailing list.

      Q: Isn't livepatching just a big ole rootkit?

      A: Canonical Livepatches inject kernel modules to replace sections of binary code in the running kernel. This requires the CAP_SYS_MODULE capability. This is required to modprobe any module into the Linux kernel. If you already have that capability (root does, by default, on Ubuntu), then you already have the ability to arbitrarily modify the kernel, with or without Canonical Livepatches. If you’re an Ubuntu sysadmin and you want to disable module loading (and thereby also disable Canonical Livepatches), simply ‘echo 1 | sudo tee /proc/sys/kernel/modules_disabled’.

      Keep the uptime!

      Read more

      Movin' on...

      A year has gone by since I started work with Canonical. As it turns out, I must be on my way. Where to? Not real sure at this moment, there seems plenty of companies using Qt & QML these days. \0/

      But saying that, I am open to suggestions. LinkedIn
      Plenty of IoT and devices using sensors around. Heck, even Moto Z phone has some great uses for sensor gestures similar to what I wrote for QtSensors while I was at Nokia.

      But a lack of companies that allow freelance or remote work. The last few years I have worked remotely doing work for Jolla and Canonical. Both fantastic companies to work for, which really have it together for working remotely.

      I am still surprised that only a handful of companies regularly allow remote work. I do not miss the stuffy non window opening offices and the long daily commute, which sometimes means riding a motorcycle through hail! (I do not suggest this for anyone)

      Of course, I am still maintainer for QtSensors, QtSystemInfo for the Qt Project, and Sensor Framework for Mer, and always dreaming up new ways to use sensors. Still keeping tabs on QtNetwork bearer classes.

      Although I had to send back the Canonical devices, I still have Ubuntu on my Nexus 4. I still have my Jolla phones and tablet.

      That said, I still have this blog here, and besides spending my time looking for a new programming gig, I am (always) preparing to release a new album.
      and always willing to work with anyone needing music/audio/soundtrack work.

      Read more
      Dustin Kirkland

      What's yours?

      Happy 2017!

      Read more
      Daniel Holbach

      Taking a break

      It’s a bit strange to write this blog post in the same week as Martin Pitt is announcing moving on from Canonical. I remember many moments of Martin’s post very vividly and he was one of the first I ran into on my flight to Sydney for Ubuntu Down Under in 2005.

      Fast forward to today: 2016 was a year full of change – my personal life was no exception there. In the last weeks I had to realise more and more that I need a long break from everything. I therefore decided to move on from Canonical, take some time off, wander the world, recharge my batteries, come back and surprise you all with what’s next.

      I’m very much leaving on good terms and I could imagine I won’t be too far away (I’d miss all you great people who became good friends way too much). Having been with Canonical for 11 years and 12 years in the Ubuntu community, it has been an incredibly hard decision to take. Still it’s necessary now and it’ll be good open myself up again to new challenges, new ways of working and new sets of problems.

      It was a great privilege to work with you all and be able to add my humble contribution to this crazy undertaking called Ubuntu. I’m extremely grateful for the great moments with you all, the opportunities to learn, your guidance, the friends I made around the world, the laughs, the discussions, the excellent work we did together. This was a very important time of my life.

      In the coming weeks I will be without internet, I haven’t quite decided yet, which part of the world I’m going to go to, but maybe I’ll post a picture or two somewhere. </p>
            <a href=Read more


      I’ve had the pleasure of working on Ubuntu for 12½ years now, and during that time used up an entire Latin alphabet of release names! (Well, A and C are still free, but we used H and W twice, so on average.. ☺ ) This has for sure been the most exciting time in my life with tons of good memories! Very few highlights:

      • Getting some spam mail from a South African multi-millionaire about a GREAT OPPORTUNITY
      • Joining #warthogs (my first IRC experience) and collecting my first bounties for “derooting” Debian (i. e. drop privileges from root daemons and suid binaries)
      • Getting invited to Oxford to meet a bunch of people for which I had absolutely zero proof of existence, and tossing myself into debts for buying a laptop for that occasion
      • Once being there, looking into my fellows’ stern and serious faces and being amazed by their professionalism:
      • The excitement and hype around going public with Warty Warthogs Beta
      • Meeting lots of good folks at many UDSes, with great ideas and lots of enthusiasm, and sometimes “Bags of Death”. Group photo from Ubuntu Down Under:
      • Organizing UDSes without Launchpad or other electronic help:
      • Playing “Wish you were Here” with Bill, Tony, Jono, and the other All Stars
      • Seeing bug #1 getting closed, and watching the transformation of Microsoft from being TEH EVIL of the FOSS world to our business partner
      • Getting to know lots of great places around the world. My favourite: luring a few colleagues for a “short walk through San Francisco” but ruining their feet with a 9 hour hike throughout the city, Golden Gate Park and dipping toes into the Pacific.
      • Seeing Ubuntu grow from that crazy idea into one of the main pillars of the free software world
      • ITZ GTK BUG!
      • Getting really excited when Milbank and the Canonical office appeared in the Harry Potter movie
      • Moving between and getting to know many different teams from the inside (security, desktop, OEM, QA, CI, Foundations, Release, SRU, Tech Board, …) to appreciate and understand the value of different perspectives
      • Breaking burning wood boards, making great and silly videos, and team games in the forest (that was La Mola) at various All Hands

      But all good things must come to an end — after tossing and turning this idea for a long time, I will leave Canonical at the end of the year. One major reason for me leaving is that after that long time I am simply in need for a “reboot”: I’ve piled up so many little and large things that I can hardly spend one day on developing something new without hopelessly falling behind in responding to pings about fixing low-level stuff, debugging weird things, handholding infrastructure, explaining how things (should) work, do urgent archive/SRU/maintenance tasks, and whatnot (“it’s related to boot, it probably has systemd in the name, let’s hand it to pitti”). I’ve repeatedly tried to rid myself of some of those or at least find someone else to share the load with, but it’s too sticky :-/ So I spent the last few weeks with finishing some lose ends and handing over some of my main responsibilities.

      Today is my last day at work, which I spend mostly on unsubscribing from package bugs, leaving Launchpad teams, and catching up with emails and bugs, i. e. “clean up my office desk”. From tomorrow on I’ll enjoy some longer EOY holidays, before starting my new job in January.

      I got offered to work on Cockpit, on the product itself and its ties into the Linux plumbing stack (storaged/udisks, systemd, and the like). So from next year on I’ll change my Hat to become Red instead of orange. I’m curious to seeing for myself how that other side of the fence looks like!

      This won’t be a personal good-bye. I will continue to see a lot of you Ubuntu folks on FOSDEMs, debconfs, Plumber’s, or on IRC. But certainly much less often, and that’s the part that I regret most — many of you have become close friends, and Canonical feels much more like a family than a company. So, thanks to all lof you for being on that journey with me, and of course a special and big Thank You to Mark Shuttleworth for coming up with this great Ubuntu vision and making all of this possible!

      Read more

      Hacked By TeaM_CC :: sec_d@rK WAS HERE

      Your Security breached ….
      No security is perfect


      Read more

      Like it says on the Intel IoT developer site, "Without sensors, there's no IoT".

      Because I am the maintainer of QtSensors, I like to inquire about  people's use of sensors and if they use QtSensors. Over the years, I have heard quite often something like, 'Qt is thought of as a UI framework'. *sigh*
      But Qt is more than just a UI framework and it's use is not dependent on widgets or declarative wizardry. It is used in quite a few middleware components without UI elements. One of those middleware frameworks is Sensor Framework.

      Sensor framework is a daemon that uses a plugin system written using Qt for reading various sensors such as accelerometer or light sensors. It was originally developed by Nokia for Harmattan and ran on the N9. It was also used in MeeGo and later included in the Mer Project and on Jolla phones and the ill fated tablet. So it has been released onto a few commercial products.

      We looked at it when I was working at Nokia on the project that I still cannot name, but we had decided we would come up with our own solution. Looking back, this was the wrong decision, we should have taken the already proven sensor framework and ran with that. Why? Because it existed and works.

      I started maintaining it when I was a privateer (contractor) developer for Jolla. No one else had touched it for some time so I grabbed the few not yet merged bug fixes and added support for libhybris/android libhardware adaptors.

      Sensor Framework has support for multiple clients with down sampling for different data rates. It uses dbus for control lines (to start and stop, etc) but sends data through a socket. It also has a working backend in QtSensors.

      I noticed that Ubuntu's Unity does nothing to respond when I put this into "tablet mode". I have to manually open the virtual keyboard among other things.

      So I thought I could use sensorfw on my Dell 2 in 1. It's one of those converged laptop/tablet devices. It has a few sensors - accelerometer, gyroscope, magnetometer, and lid sensors. One problem... sensorfw does not support lid sensors, or a few other sensors that are around today in IoT (which I will add a bit later). Lid "sensor" might be a bit of a misnomer, as they could be switches but I'd like to think it is more like a hal effect sensor that uses magnets. In any case there are event nodes to use.

      First one I chose is to add the lid sensor - to detect when this machine is put into tablet mode, so the UI can better deal with it.

      I also noticed that this kernel has support for iio sensor interface for the accel and gyro. Sensorfw only supports sysfs, evdev and hybris interfaces, so I also wanted to add support for that.

      I worked on adding iio support first. Well... really just wrote a sensor adaptor plugin. My plugin supports accelerometer, gyroscope and magnetometer, which this device seems to have. I will expand this to support other sensors later, as well as clean it up a bit.

      Thanks to QtSensors sensor framework backend, I can make a UI app change with the orientation and lid changes. Better yet, I can create a game that uses accelerometer data like a marble maze game. Or I can upload the data to one of those Node.js data visualization web apps.

      And since sensor framework is opensource, others can as well.

      Read more

      My team at work has been focused on snaps this year and one thing we’ve tried to do internally is establish a set of best practices for snap packaging software. Toward that end I’ve been working on a little tool I’m calling snaplint to encode those practices and verify that we’re following them.

      Right now you can run snaplint against your snapcraft project directory
      and it will scan the prime subdirectory for the following things:

      • copyright (basically that you included usr/share/doc/*copyright*) for
        any stage-packages
      • developer cruft (things like header and object files or static libs
        that might have made their way into your snap)
      • libraries (examine the ELF files in your snap and look for libraries
        which aren’t used)

      The next things I’m planning on adding are:

      • checking for copyright info from apps/parts themselves.
      • checking for mixing of incompatible licenses

      I would love to hear suggestions on further improvements.

      You can find the source at

      And, of course if you’re running Ubuntu 16.04 or later you can try it on your own machine with:
      $ snap install snaplint
      $ snaplint path/to/your/project

      Read more
      Dustin Kirkland

      From Linux kernel livepatches to encryption to ASLR to compiler optimizations and configuration hardening, we strive to ensure that Ubuntu 16.04 LTS is the most secure Linux distribution out of the box.

      These slides try to briefly explain:

      • what we do to secure Ubuntu
      • how the underlying technology works
      • when the features took effect in Ubuntu

      I hope you find this slide deck informative and useful!  The information herein is largely collected from the Ubuntu Security Features wiki page, where you can always find up to date information.


      Read more
      Dustin Kirkland

      If you haven't heard about last week's Dirty COW vulnerability, I hope all of your Linux systems are automatically patching themselves...

      Why?  Because every single Linux-based phone, router, modem, tablet, desktop, PC, server, virtual machine, and absolutely everything in between -- including all versions of Ubuntu since 2007 -- was vulnerable to this face-palming critical security vulnerability.

      Any non-root local user of a vulnerable system can easily exploit the vulnerability and become the root user in a matter of a few seconds.  Watch...

      Coincidentally, just before the vulnerability was published, we released the Canonical Livepatch Service for Ubuntu 16.04 LTS.  The thousands of users who enabled canonical-livepatch on their Ubuntu 16.04 LTS systems with those first few hours received and applied the fix to Dirty COW, automatically, in the background, and without rebooting!

      If you haven't already enabled the Canonical Livepatch Service on your Ubuntu 16.04 LTS systems, you should really consider doing so, with 3 easy steps:
      1. Go to and retrieve your livepatch token
      2. Install the canonical-livepatch snap
        $ sudo snap install canonical-livepatch 
      3. Enable the service with your token
        $ sudo canonical-livepatch enable [TOKEN]
      And you’re done! You can check the status at any time using:

      $ canonical-livepatch status --verbose

      Let's retry that same vulnerability, on the same system, but this time, having been livepatched...

      Aha!  Thwarted!

      So that's the Ubuntu 16.04 LTS kernel space...  What about userspace?  Most of the other recent, branded vulnerabilities (Heartbleed, ShellShock, CRIME, BEAST) have been critical vulnerabilities in userspace packages.

      As of Ubuntu 16.04 LTS, the unattended-upgrades package is now part of the default package set, so you should already have it installed on your Ubuntu desktops and servers.  If you don't already have it installed, you can install it with:

      $ sudo apt install unattended-upgrades

      And moreover, as of Ubuntu 16.04 LTS, the unattended-upgrades package automatically downloads and installs important security updates once per day, automatically patching critical security vulnerabilities and keeping your Ubuntu systems safe by default.  Older versions of Ubuntu (or Ubuntu systems that upgraded to 16.04) might need to enable this behavior using:

      $ sudo dpkg-reconfigure unattended-upgrades

      With that combination enabled -- (1) automatic livepatches to your kernel, plus (2) automatic application of security package updates -- Ubuntu 16.04 LTS is the most secure Linux distribution to date.  Period.


      Read more
      Dustin Kirkland

      My wife, Kimberly, and I watch Saturday Night Live religiously.  As in, we probably haven't missed a single episode since we started dating more than 12 years ago.  And in fact, we both watched our fair share of SNL before we had even met, going back to our teenage years.

      We were catching up on SNL's 42nd season premier late this past Sunday night, after putting the kids to bed, when I was excited to see a hilarious sketch/parody of Mr. Robot.

      If SNL is my oldest TV favorite, Mr. Robot is certainly my newest!  Just wrapping its 2nd season, it's a brilliantly written, flawlessly acted, impeccably set techno drama series on USA.  I'm completely smitten, and the story seems to be just getting started!

      Okay, so Kim and I are watching a hilarious sketch where Leslie Jones asks Elliot to track down the person who recently hacked her social media accounts.  And, as always, I take note of what's going in the background on the computer screen.  It's just something I do.  I love to try and spot the app, the OS, the version, identify the Linux kernel oops, etc., of anything on any computer screen on TV.

      At about the 1:32 mark of the SNL/Mr.Robot skit, there was something unmistakable on the left computer, just over actor Pete Davidson's right shoulder.  Merely a fraction of a second, and I recognized it instantly!  A dark terminal, split into a dozen sections.  A light grey boarder, with a thicker grey highlighting one split.  The green drip of text from The Matrix in one of the splits. A flashing, bouncing yellow audio wave in another.  An instant rearrangement of all of those windows each second.

      It was Byobu and Hollywood!  I knew it.  Kim didn't believe me at first, until I proved it ;-)

      A couple of years ago, after seeing a 007 film in the theater, I created a bit of silliness -- a joke of a program that could turn any Linux terminal into a James Bond caliber hacker screen.  The result is a package called hollywood, which any Ubuntu user can install and run by simply typing:

      $ sudo apt install hollywood
      $ hollywood

      And a few months ago , Hollywood found its way into an NBC News piece that took itself perhaps a little too seriously, as it drummed up a bit of fear around "Ransomware".

      But, far more appropriately, I'm absolutely delighted to see another NBC program -- Saturday Night Live -- using Hollywood exactly as intended -- for parody!

      Enjoy a few screenshots below...


      Read more
      Dustin Kirkland

      On Monday this week, I was afforded the distinct privilege to deliver the opening keynote at the OpenZFS Developer Summit in San Francisco.  It was a beautiful little event, with a full day of informative presentations and lots of networking during lunch and breaks.

      Below, you can view my slides, download the PDF, or watch the talk (starts at 31:10) and demo in its entirety.

      Hopefully you'll enjoy the demo -- especially the most interesting raw tracing system new in the Ubuntu 16.04 LTS Linux 4.4 kernel, something called The Berkeley Packet Filter, or "BPF" for short.  I used a series of open source utilities from Brendan Gregg (from Netflix), called iovisor/bcc.  Quoting the on Github:

      BCC is a toolkit for creating efficient kernel tracing and manipulation programs, and includes several useful tools and examples. It makes use of extended BPF (Berkeley Packet Filters), formally known as eBPF, a new feature that was first added to Linux 3.15. Much of what BCC uses requires Linux 4.1 and above.
      I'll follow up this post with another one, formally introducing BPF and how to install and use bcc in Ubuntu 16.04 LTS, if anyone is interested...


      Read more
      Michael Hall

      KDE Neon developer Harald Sitter was able to package up the KDE calculator, kcalc, in a snap that weighs in at a mere 320KB! How did he do it?

      KCalc and KDE Frameworks snaps

      Like most applications in KDE, kcalc depends on several KDE Frameworks (though not all), sets of libraries and services that provide the common functionality and shared UI/UX found in KDE and it’s suite of applications. This means that, while kcalc is itself a small application, it’s dependency chain is not. In the past, any KDE application snap had to include many megabytes of platforms dependencies, even for the smallest app.

      Recently I introduced the new “content” interface that has been added to snapd. I used this interface to share plugin code with a text editor, but Harald has taken it even further and created a KDE Frameworks snap that can share the entire platform with applications that are built on it!

      While still in the very early stages of development, this approach will allow the KDE project to deliver all of their applications as independent snaps, while still letting them all share the one common set of Frameworks that they depend on. The end result will be that you, the user, will get the very latest stable (or development!) version of the KDE platform and applications, direct from KDE themselves, even if you’re on a stable/LTS release of your distro.

      If you are running a snap-capable distro, you can try these experimental packages yourself by downloading kde-frameworks-5_5.26_amd64.snap and kcalc_0_amd64.snap from Neon’s build servers, and installing them with “snap install –devmode –force-dangerous <snap_file>”. To learn more about how he did this, and to help him build more KDE application snaps, you can find Harald as <sitter> on #kde-neon on Freenode IRC.

      Read more
      Dustin Kirkland

      A couple of weeks ago, I delivered a talk at the Container Camp UK 2016.  It was an brilliant event, on a beautiful stage at Picturehouse Central in Picadilly Circus in London.

      You're welcome to view the slides or download them as a PDF, or watch my talk below.

      And for the techies who want to skip the slide fluff and get their hands dirty, setup your OpenStack and LXD and start streamlining your HPC workloads using this guide.


      Read more
      Michael Hall

      Snaps are a great way to get the most up to date applications on your desktop without putting the security or stability or your system at risk. I’ve been snapping up a bunch of things lately and the potential this new paradigm offers is going to be revolutionary. Unfortunately nothing comes for free, and the security of snaps comes with some necessary tradeoffs like isolation and confinement, which reduces some of the power and flexibility we’ve become used to as Linux users.

      But now the developers of the snappy system (snapd, snap-confine and snapcraft) are giving us back some of that missing flexibility in the form of a new “content” interface which allows you to share files (executables, libraries, or data) between the snap packages that you develop. I decided to take this new interface for a test drive using one of the applications I had recently snapped: Geany, my editor of choice. Geany has the ability to load plugins to extend it’s functionality, and infact has a set of plugins available in a separate Github repository from the application itself.

      I already had a working snap for Geany, so the next thing I had to do was create a snap for the plugins. Like Geany itself, the plugins are hosted on GitHub and have a nice build configuration already, so turning it into a snap was pretty trivial. I used the autotools plugin in Snapcraft to pull the git source and build all of the available plugins. Because my Geany snap was built with Gtk+ 3, I had to build the plugins for the same toolkit, but other than that I didn’t have to do anything special.

       plugin: autotools
       source-type: git
       configflags: [--enable-gtk3=yes --enable-all-plugins]

      Now that I had a geany.snap and geany-plugins.snap, the next step was to get them working together. Specifically I wanted Geany to be able to see and load the plugin files from the plugins snap, so it was really just a one-way sharing. To do this I had to create both a slot and a plug using the content interface. Usually when you’re building snap you only use plugs, such as network or x11, because you are consuming services provided by the core OS. In those cases also you just have to provide the interface name in the list of plugs, because the interface and the plug have the same name.

      But with the content interface you need to do more than that. Because different snaps will provide different content, and a single snap can provide multiple kinds of content, you have to define a new name that is specific to what content you are sharing. So in my geany-plugins snapcraft.yaml I defined a new kind of content that I called geany-plugins-all (because it contains all the geany plugins in the snap), and I put that into a slot called geany-plugins-slot which is how we will refer to it later. I told snapcraft that this new slot was using the content interface, and then finally told it what content to share across that interface, which for geany-plugins was the entire snap’s content.

       content: geany-plugins-all
       interface: content
       - /

      With that I had one half of the content interface defined. I had a geany-plugins.snap that was able to share all of it’s content with another snap. The next step was to implement the plug half of the interface in my existing geany.snap. This time instead of using a slots: section I would define a plugs: section, with a new plug named geany-plugins-plug and again specifying the interface to be content just like in the slot. Here again I had to specify the content by name, which had to match the geany-plugins-all that was used in the slot. The names of the plug and slot are only relevant to the user who needs to connect them, it’s this content name that snapd uses to make sure they can be connected in the first place. Finally I had to give the plug a target directory for where the shared content will be put. I chose a directory called plugins, and when the snaps are connected the geany-plugins.snap content will be bind-mounted into this directory in the geany.snap

       content: geany-plugins-all
       default-provider: geany-plugins
       interface: content
       target: plugins

      Lastly I needed to tell snapcraft which app would use this interface. Since the Geany snap only has one, I added it there.

       command: gtk-launch geany
       plugs: [x11, unity7, home, geany-plugins-plug]

      Once the snaps were built, I could install them and the new plug and slot were automatically connected

      $ snap interfaces
      Slot                             Plug
      geany-plugins:geany-plugins-slot geany:geany-plugins-plug

      Now that put the plugins into the application’s snap space, but it wasn’t enough for Geany to actually find them. To do that I used Geany’s Extra plugin path preferences to point it to the location of the shared plugin files.

      Screenshot from 2016-08-30 16-27-12

      After doing that, I could open the Plugin manager and see all of the newly shared plugins. Not all of them work, and some assume specific install locations or access to other parts of the filesystem that they won’t have being in a snap. The Geany developers warned me about that, but the ones I really wanted appear to work.

      Screenshot from 2016-08-30 16-29-54

      Read more

      Occasionally I find myself processing input data which arrives as a stream, like data from files or from a socket, but that has a known structure that can be modeled with C types. For instance, let’s say we are receiving from a socket a parcel that consists on a header of one byte, and a payload that is an integer. A naive way to handle this is the following (simplified for readability) code snippet:

      int main(void)
          int fd;
          char *buff;
          struct sockaddr_in addr;
          int vint;
          char vchar;
          fd = socket(AF_INET, SOCK_STREAM, 0);
          buff = malloc(BUFF_SIZE);
          /* Init socket address */
          connect(fd, (struct sockaddr *) &addr, sizeof(addr));
          read(fd, buff, BUFF_SIZE);
          vchar = buff[0];
          vint  = *(int *) &buff[1];
          /* Do something with extracted data, free resources */
          return 0;

      Here we get the raw data with a read() call, we read the first byte, then we read an integer by taking a pointer to the second read byte and casting it to a pointer to an integer. (for this example we are assuming that the integer inserted in the stream has the same size and endianness as the CPU ones).

      There is a big issue with this: the cast to int *, which is undefined behavior according to the C standard 1. And it is because things can go wrong in at least two ways, first due to pointer aliasing rules, second due to type alignment.

      Strict pointer aliasing tells the compiler that it can assume that pointers to different types point to different places in memory. This allows some optimizations, like reordering. Therefore, we could be in trouble if, say, we take &buff[1] into a char * and use it to write a byte in that location, as reordering could hit us. So just do not do that. Let’s also hope that we have a compiler that is not completely insane and does not move our reading by int pointer before the read() system call. We could also disable strict aliasing if we are using GCC with option -fno-strict-aliasing, which by the way is something that the Linux kernel does. At any rate, this is a complex subject and I will not dig into it this time.

      We will concentrate in this article on how to solve the other problem, that is, how to access safely types that are not stored in memory in their natural alignment.

      The C Standard-Compliant Solution

      Before moving further, keep in mind that it is always possible to be strictly compliant with the standard and access safely memory without breaking language rules or using compiler or machine specific tricks. In the example, we could retrieve vint by doing

          vint  =   buff[1] + (buff[2] << 8)
                  + (buff[3] << 16) + (buff[4] << 24);

      (supposing stored data is little endian).

      The issue here is performance: we are implicitly transforming four bytes to integers, then we have to bit-shift three of them, and finally we have to add them up. Note however that this is what we want if data and CPU have different endianness.

      Doing Unaligned Memory Accesses

      In all machine architectures there is a natural alignment for the different data types. This alignment is usually the size of the types, for instance in 32 bits architectures the alignment for integers is 4, for doubles it is 8, etc. If instances of these types are not stored in memory positions that are multiple of their alignment, we are talking about unaligned access. If we try to access unaligned data either of these can happen:

      • The hardware let’s us access it – but always at a performance penalty.
      • An exception is triggered by the CPU. This type of exception is called bus error 2.

      We might be willing to accept the performance penalty 3, which is mitigated by CPU caches and not that noticeable in certain architectures like x86-64 , but we certainly do not want our program to crash. How possible is this? To be honest it is not something I have seen that often. Therefore, as a first analysis step, I checked how easy it was to get bus errors. To do so, I created the following C++ program, access1.cpp (I could not resist to use templates here to reduce the code size):

      #include <iostream>
      #include <typeinfo>
      #include <cstring>
      using namespace std;
      template <typename T>
      void print_unaligned(char *ptr)
          T *val = reinterpret_cast<T *>(ptr);
          cout << "Type is \"" << typeid(T).name()
               << "\" with size " << sizeof(T) << endl;
          cout << val << " *val: " << *val << endl;
      int main(void)
          char *mem = new char[128];
          memset(mem, 0, 128);
          print_unaligned<int>(mem + 1);
          print_unaligned<long long>(mem);
          print_unaligned<long long>(mem + 1);
          print_unaligned<long double>(mem);
          print_unaligned<long double>(mem + 1);
          delete[] mem;
          return 0;

      The program allocates memory using new char[], which as malloc() in C is guaranteed to allocate memory with the same alignment as the strictest fundamental type. After zeroing the memory, we access mem and mem + 1 by casting to different pointer types, knowing that the second address is odd, and therefore unaligned except for char * access.

      I compiled the file with g++ on my laptop, ran it, and got

      $ g++ access1.cpp -o access1
      $ file access1
      access1: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/, for GNU/Linux 2.6.32, BuildID[sha1]=09d0fb19340a10941eef4c3dc4d6eb29383e717d, not stripped
      $ ./access1
      Type is "i" with size 4
      0x16c3c20 *val: 0
      Type is "i" with size 4
      0x16c3c21 *val: 0
      Type is "x" with size 8
      0x16c3c20 *val: 0
      Type is "x" with size 8
      0x16c3c21 *val: 0
      Type is "e" with size 16
      0x16c3c20 *val: 0
      Type is "e" with size 16
      0x16c3c21 *val: 0

      No error for x86-64. This was expected as Intel architecture is known to support unaligned access by hardware, at a performance penalty (which is apparently quite small these days, see 4).

      The second try was with an ARM CPU, compiling for arm-32:

      $ g++ access1.cpp -o access1
      $ file access1
      access1: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=8c3c3e7d77fddd5f95d18dbffe37d67edc716a1c, not stripped
      $ ./access1
      Type is "i" with size 4
      0x47b008 *val: 0
      Type is "i" with size 4
      0x47b009 *val: 0
      Type is "x" with size 8
      0x47b008 *val: 0
      Type is "x" with size 8
      Bus error (core dumped)

      Now we get what we were searching for, a legitimate bus error, in this case when accessing a long long from an unaligned address. Commenting out the offending line and letting the program run further showed the error also when accessing a long double from mem + 1.

      Fixing Unaligned Memory Accesses

      After proving that this could be a real problem, at least for some architectures, I tried to find a solution that would let me do unaligned memory accesses in the most generic way. I could not find anything safe that was strictly following the C standard. However, all C/C++ compilers have ways to define packed structures, and that came to the rescue.

      Packed structures are intended to minimize the padding that is introduced by alignment needed by the structure members. They are used when minimizing storage is a big concern. But what is interesting for us is that its members can be unaligned due to the packing, so dereferencing them must take that into account. Therefore, if we are accessing a type in a CPU that does not support unaligned access for that type the compiler must synthesize code that handles this transparently from the point of view of the C program.

      To test that this worked as expected, I wrote access2.cpp, which uses GCC attribute __packed__ to define a packed structure:

      #include <iostream>
      #include <typeinfo>
      #include <cstring>
      using namespace std;
      template <typename T>
      struct __attribute__((__packed__)) struct_safe
          T val;
      template <typename T>
      void print_unaligned(char *ptr)
          struct_safe<T> *safe = reinterpret_cast<struct_safe<T> *>(ptr);
          cout << "Type is \"" << typeid(T).name()
               << "\" with size " << sizeof(T) << endl;
          cout << safe << " safe->val: " << safe->val << endl;
      int main(void)
          char *mem = new char[128];
          memset(mem, 0, 128);
          print_unaligned<int>(mem + 1);
          print_unaligned<long long>(mem);
          print_unaligned<long long>(mem + 1);
          print_unaligned<long double>(mem);
          print_unaligned<long double>(mem + 1);
          delete[] mem;
          return 0;

      In this case, instead of directly casting to the type, I cast to a pointer to the packed struct and access the type through it.

      Compiling and running for x86-64 got the expected result: no error, all worked as before. Then I compiled and ran it in an ARM device:

      $ g++ access2.cpp -o access2
      $ file access2
      access2: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=9a1ee8c2fcd97393a4b53fe12563676d9f2327a3, not stripped
      $ ./access2
      Type is "i" with size 4
      0x391008 safe->val: 0
      Type is "i" with size 4
      0x391009 safe->val: 0
      Type is "x" with size 8
      0x391008 safe->val: 0
      Type is "x" with size 8
      0x391009 safe->val: 0
      Type is "e" with size 8
      0x391008 safe->val: 0
      Type is "e" with size 8
      0x391009 safe->val: 0

      No bus errors anymore! It worked as expected. To gain some understanding of what was happening behind the curtains, I disassembled the generated ARM binaries. For both access1 and access2, the same instruction was being used when I was getting a value after casting to int: LDR, which unsurprisingly loads a 32-bit word into a register. But for the long long, I found that access1 was using LDRD, which loads double words (8 bytes) from memory, while access2 was using two LDR instructions instead.

      This all made a lot of sense, as ARM states that LDR supports access to unaligned data, while LDRD does not 5. Indeed the later is faster, but has this restriction. It was also good to check that there was no penalty for using the packed structure for integers: GCC does a good job to discriminate when the CPU really needs to handle differently unaligned accesses.

      GCC cast-align Warning

      GCC has a warning that can help to identify points in the code when we might be accessing unaligned data, which is activated with -Wcast-align. It is not part of the warnings that are activated by options -Wall or -Wextra, so we will have to add it explicitly if we want it. The warning is only triggered when compiling for architectures that do not support unaligned access for all types, so you will not see it if compiling only for x86.

      When triggered, you will see something like

      file.c:28:23: warning: cast increases required alignment of target type [-Wcast-align]
         int *my_int_ptr = (int *) &buf[i];


      The moral of this post is that you need to be very careful when casting pointers to a type different to the original one 6. When you need to do that, think about alignment issues first, and also think on your target architectures. There are programs that we want to run on more than one CPU type and too many times we only test in our reference.

      Unfortunately the C standard does not give us a standard way of doing efficient access to unaligned data, but most if not all compilers seem to provide ways to do this. If we are using GCC, __attribute__((__packed__)) can help us when we might be doing unaligned accesses. The ARM compiler has a __packed attribute for pointers 7, and I am sure other compilers provide similar machinery. I also recommend to activate -Wcast-align if using GCC, which makes easier to spot alignment issues.

      Finally, a word of caution: in most cases you should not do this type of casts. Some times you can define structures and read directly data onto them, some times you can use unions. Bear in mind always the strict pointer aliasing rules, which can hit back. To summarize, think twice before using the sort of trick showed in the post, and use them only when really needed.

      Read more
      Dustin Kirkland

      I hope you'll enjoy a shiny new 6-part blog series I recently published at
      1. The first article is a bit of back story, perhaps a behind-the-scenes look at the motivations, timelines, and some of the work performed between Microsoft and Canonical to bring Ubuntu to Windows.
      2. The second article is an updated getting-started guide, with screenshots, showing a Windows 10 user exactly how to enable and run Ubuntu on Windows.
      3. The third article walks through a dozen or so examples of the most essential command line utilities a Windows user, new to Ubuntu (and Bash), should absolutely learn.
      4. The fourth article shows how to write and execute your first script, "Howdy, Windows!", in 6 different dynamic scripting languages (Bash, Python, Perl, Ruby, PHP, and NodeJS).
      5. The fifth article demonstrates how to write, compile, and execute your first program in 7 different compiled programming languages (C, C++, Fortran, Golang).
      6. The sixth and final article conducts some performance benchmarks of the CPU, Memory, Disk, and Network, in both native Ubuntu on a physical machine, and Ubuntu on Windows running on the same system.
      I really enjoyed writing these.  Hopefully you'll try some of the examples, and share your experiences using Ubuntu native utilities on a Windows desktop.  You can find the source code of the programming examples in Github and Launchpad:

      Read more

      Sensors are an important part of IoT. Phones, robots and drones all have a slurry of sensors. Sensor chips are everywhere, doing all kinds of jobs to help and entertain us. Modern games and game consoles can thank sensors for some wonderfully active games.

      Since I became involved with sensors and wrote QtSensorGestures as part of the QtSensors team at Nokia, sensors have only gotten cheaper and more prolific.

      I used Ubuntu Server, snappy, a raspberry pi 3, and the senseHAT sensor board to create a senseHAT sensors snap. Of course, this currently only runs in devmode on raspberry pi3 (and pi2 as well) .

      To future proof this, I wanted to get sensor data all the way up to QtSensors, for future QML access.

      I now work at Canonical. Snappy is new and still in heavy development so I did run into a few issues. First up was QFactoryLoader which finds and loads plugins, was not looking in the correct spot. For some reason, it uses $SNAP/usr/bin as it's QT_PLUGIN_PATH. I got around this for now by using a wrapper script and setting QT_PLUGIN_PATH to $SNAP/usr/lib/arm-linux-gnueabihf/qt5/plugins

      Second issue was that QSensorManager could not see it's configuration file in /etc/xdg/QtProject which is not accessible to a snap. So I used the wrapper script to set up  XDG_CONFIG_DIRS as $SNAP/etc/xdg

      [NOTE] I just discovered there is a part named "qt5conf" that can be used to setup Qt's env vars by using the included command qt5-launch  to run your snap's commands.

      Since there is no libhybris in Ubuntu Core, I had to decide what QtSensor backend to use. I could have used sensorfw, or maybe iio-sensor-proxy but RTIMULib already worked for senseHAT. It was easier to write a QtSensors plugin that used RTIMULib, as opposed to adding it into sensorfw. iio-sensor-proxy is more for laptop like machines and lacks many sensors.
      RTIMULib uses a configuration file that needs to be in a writable area, to hold additional device specific calibration data. Luckily, one of it's functions takes a directory path to look in. Since I was creating the plugin, I made it use a new variable SENSEHAT_CONFIG_DIR so I could then set that up in the wrapper script.

      This also runs in confinement without devmode, but involves a simple sensors snapd interface.
      One of the issues I can already see with this is that there are a myriad ways of accessing the sensors. Different kernel interfaces - iio,  sysfs, evdev, different middleware - android SensorManager/hybris, libhardware/hybris, sensorfw and others either I cannot speak of or do not know about.

      Once the snap goes through a review, it will live here, but for now, there is working code is at my sensehat repo.

      Next up to snapify, the Matrix Creator sensor array! Perhaps I can use my sensorfw snap or iio-sensor-proxy snap for that.

      Read more

      So there I was. I did have to use a proprietary library, for which I had no sources and no real hope of support from the creators. I built my program against it, I ran it, and I got a segmentation fault. An exception that seemed to happen inside that insidious library, which was of course stripped of all debugging information. I scratched my head, changed my code, checked traces, tried valgrind, strace, and other debugging tools, but found no obvious error. Finally, I assumed that I had to dig deeper and do some serious debugging of the library’s assembly code with gdb. The rest of the post is dedicated to the steps I followed to find out what was happening inside the wily proprietary library that we will call libProprietary. Prerequisites for this article are some knowledge of gdb and ARM architecture.

      Some background on the task I was doing: I am a Canonical employee that works as developer for Ubuntu for Phones. In most, if not all, phones, the BSP code is not 100% open and we have to use proprietary libraries built for Android. Therefore, these libraries use bionic, Android’s libc implementation. As we want to call them inside binaries compiled with glibc, we resort to libhybris, an ingenious library that is able to load and call libraries compiled against bionic while the rest of the process uses glibc. This will turn out to be critical in this debugging. Note also that we are debugging ARM 32-bits binaries here.

      The Debugging Session

      To start, I made sure I had installed glibc and other libraries symbols and started to debug by using gdb in the usual way:

      $ gdb myprogram
      GNU gdb (Ubuntu 7.9-1ubuntu1) 7.9
      Starting program: myprogram
      [Thread debugging using libthread_db enabled]
      Using host libthread_db library "/lib/arm-linux-gnueabihf/".
      [New Thread 0xf49de460 (LWP 7101)]
      [New Thread 0xf31de460 (LWP 7104)]
      [New Thread 0xf39de460 (LWP 7103)]
      [New Thread 0xf41de460 (LWP 7102)]
      [New Thread 0xf51de460 (LWP 7100)]
      Program received signal SIGSEGV, Segmentation fault.
      [Switching to Thread 0xf49de460 (LWP 7101)]
      0x00000000 in ?? ()
      (gdb) bt
      #0  0x00000000 in ?? ()
      #1  0xf520bd06 in ?? ()
      Backtrace stopped: previous frame identical to this frame (corrupt stack?)
      (gdb) info proc mappings
      process 7097
      Mapped address spaces:
      	Start Addr   End Addr       Size     Offset objfile
      	   0x10000    0x17000     0x7000        0x0 /usr/bin/myprogram
      	0xf41e0000 0xf49df000   0x7ff000        0x0 [stack:7101]
      	0xf51f6000 0xf5221000    0x2b000        0x0 /android/system/lib/
      	0xf5221000 0xf5222000     0x1000        0x0 
      	0xf5222000 0xf5224000     0x2000    0x2b000 /android/system/lib/
      	0xf5224000 0xf5225000     0x1000    0x2d000 /android/system/lib/

      We can see here that we get the promised crash. I execute a couple of gdb commands after that to see the backtrace and part of the process address space that will be of interest in the following discussion. The backtrace shows that a segment violation happened when the CPU tried to execute instructions in address zero, and we can see by checking the process mappings that the previous frame lives inside the text segment of There is no backtrace beyond that point, but that should come as no surprise as there is no DWARF information in libProprietary, and also noting that usage of frame pointer is optimized away quite commonly these days.

      After this I tried to get a bit more information on the CPU state when the crash happened:

      (gdb) info reg
      r0             0x0	0
      r1             0x0	0
      r2             0x0	0
      r3             0x9	9
      r4             0x0	0
      r5             0x0	0
      r6             0x0	0
      r7             0x0	0
      r8             0x0	0
      r9             0x0	0
      r10            0x0	0
      r11            0x0	0
      r12            0xffffffff	4294967295
      sp             0xf49dde70	0xf49dde70
      lr             0xf520bd07	-182403833
      pc             0x0	0x0
      cpsr           0x60000010	1610612752
      (gdb) disassemble 0xf520bd02,+10
      Dump of assembler code from 0xf520bd02 to 0xf520bd0c:
         0xf520bd02:	b	0xf49c9cd6
         0xf520bd06:	movwpl	pc, #18628	; 0x48c4	<UNPREDICTABLE>
         0xf520bd0a:	andlt	r4, r11, r8, lsr #12
      End of assembler dump.

      Hmm, we are starting to see weird things here. First, in 0xf520bd02 (which probably has been executed little before the crash) we get an unconditional branch to some point in the thread stack (see mappings in previous figure). Second, the instruction in 0xf520bd06 (which should be executed after returning from the procedure that provokes the crash) would load into the pc (program counter) an address that is not mapped: we saw that the first mapped address is 0x10000 in the previous figure. The movw instruction has also a “pl” suffix that makes the instruction execute only when the operand is positive or zero… which is obviously unnecessary as 0x48c4 is encoded in the instruction.

      I resorted to doing objdump -d to disassemble the library and compare with gdb output. objdump shows, in that part of the file (subtracting the library load address gives us the offset inside the file: 0xf520bd02-0xf51f6000=0x15d02):

         15d02:	f7f3 eade 	blx	92c0 <__android_log_print@plt>;
         15d06:	f8c4 5304 	str.w	r5, [r4, #772]	; 0x304
         15d0a:	4628      	mov	r0, r5
         15d0c:	b00b      	add	sp, #44	; 0x2c
         15d0e:	e8bd 8ff0 	ldmia.w	sp!, {r4, r5, r6, r7, r8, r9, sl, fp, pc}

      which is completely different from what gdb shows! What is happening here? Taking a look at addresses for both code chunks, we see that instructions are always 4 bytes in gdb output, while they are 2 or 4 in objdump‘s. Well, you have guessed, don’t you? We are seeing “normal” ARM instructions in gdb, while objdump is decoding THUMB-2 instructions. Certainly objdump seems to be right here as the output is more sensible: we have a call to an executable part of the process space in 0x15d02 (it is resolved to a known function, __android_log_print), and the following instructions seems like a normal function epilogue in ARM: a return value is stored in r0, the sp (stack pointer) is incremented (we are freeing space in the stack), and we restore registers.

      If we get back to the register values, we see that cpsr (current program status register [1]) does not have the T bit set, so gdb thinks we are using ARM instructions. We can change this by doing

      (gdb) set $cpsr=0x60000030
      (gdb) disass 0xf520bd02,+15
      Dump of assembler code from 0xf520bd02 to 0xf520bd11:
         0xf520bd02:	blx	0xf51ff2c0
         0xf520bd06:	str.w	r5, [r4, #772]	; 0x304
         0xf520bd0a:	mov	r0, r5
         0xf520bd0c:	add	sp, #44	; 0x2c
         0xf520bd0e:	ldmia.w	sp!, {r4, r5, r6, r7, r8, r9, r10, r11, pc}
      End of assembler dump.

      Ok, much better now [2]. The thumb bit in cpsr is determined by last bx/blx call: if the address is odd, the procedure to which we are calling contains THUMB instructions, otherwise they are ARM (a good reference for these instructions is [3]). In this case, after an exception the CPU moves to arm mode, and gdb is unable to know which is the right mode when disassembling. We can search for hints on which parts of the code are arm/thumb by looking at the values in registers used by bx/blx, or by looking at the lr (link register): we can see above that the value after the crash was 0xf520bd07, which is odd and indicates that 0xf520bd06 contains a thumb instruction. However, for some reason gdb is not able to take advantage of this information.

      Of course this problem does not happen if we have debugging information: in that case we have special symbols that let gdb know if the section where the code is contains thumb instructions or not [4]. As those are not found, gdb uses the cpsr value. Here objdump seems to have better heuristics though.

      After solving this issue with instruction decoding, I started to debug __android_log_print to check what was happening there, as it looked like the crash was happening in that call. I spent quite a lot of time there, but found nothing. All looked fine, and I started to despair. Until I inserted a breakpoint in address 0xf520bd06, right after the call to __android_log_print, run the program… and it stopped at that address, no crash happened. I started to execute the program instruction by instruction after that:

      (gdb) b *0xf520bd06
      (gdb) run
      Breakpoint 1, 0xf520bd06 in ?? ()
      (gdb) si
      0xf520bd0a in ?? ()
      (gdb) si
      0xf520bd0c in ?? ()
      (gdb) si
      0xf520bd0e in ?? ()
      Cannot insert breakpoint 0.
      Cannot access memory at address 0x0

      Something was apparently wrong with instruction ldmia, which restores registers, including the pc, from the stack. I took a look at the stack in that moment (taking into account that ldmia had already modified the sp after restoring 9 registers == 36 bytes):

      (gdb) x/16xw $sp-36
      0xf49dde4c:	0x00000000	0x00000000	0x00000000	0x00000000
      0xf49dde5c:	0x00000000	0x00000000	0x00000000	0x00000000
      0xf49dde6c:	0x00000000	0x00000000	0x00000000	0x00000000
      0xf49dde7c:	0x00000000	0x00000000	0x00000000	0x00000000

      All zeros! At this point it is clear that this is the real point where the crash is happening, as we are loading 0 into the pc. This looked clearly like a stack corruption issue.

      But, before moving forward, why are we getting a wrong backtrace from gdb? Well, gdb is seeing a corrupted stack, so it is not able to unwind it. It would not be able to unwind it even if having full debug information. The only hint it has is the lr. This register contains the return address after execution of a bl/blx instruction [3]. If the called procedure is non-leaf, it is saved in the prologue, and restored in the epilogue, because it gets overwritten when branching to other procedures. In this case, it is restored on the pc and sometimes it is also saved back in the lr, depending on whether we have arm-thumb interworking built in the procedure or not [5]. It is not overwritten if we have a leaf procedure (as there are no procedure calls inside these).

      As gdb has no additional information, it uses the lr to build the backtrace, assuming we are in a leaf procedure. However this is not true and the backtrace turns out to be wrong. Nonetheless, this information was not completely useless: lr was pointing to the instruction right after the last bl/blx instruction that was executed, which was not that far away from the real point where the program was crashing. This happened because fortunately __android_log_print has interworking code and restores the lr, otherwise the value of lr could have been from a point much far away from the point where the real crash happens. Believe or not, but it could have been even worse!

      Having now a clear idea of where and why the crash was happening, things accelerated. The procedure where the crash happened, as disassembled by objdump, was (I include here only the more relevant parts of the code)

      00015b1c <ProprietaryProcedure@@Base>:
         15b1c:	e92d 4ff0 	stmdb	sp!, {r4, r5, r6, r7, r8, r9, sl, fp, lr}
         15b20:	b08b      	sub	sp, #44	; 0x2c
         15b22:	497c      	ldr	r1, [pc, #496]	; (15d14 <ProprietaryProcedure@@Base+0x1f8>)
         15b24:	2500      	movs	r5, #0
         15b26:	9500      	str	r5, [sp, #0]
         15b28:	4604      	mov	r4, r0
         15b2a:	4479      	add	r1, pc
         15b2c:	462b      	mov	r3, r5
         15b2e:	f8df 81e8 	ldr.w	r8, [pc, #488]	; 15d18 <ProprietaryProcedure@@Base+0x1fc>
         15b32:	462a      	mov	r2, r5
         15b34:	f8df 91e4 	ldr.w	r9, [pc, #484]	; 15d1c <ProprietaryProcedure@@Base+0x200>
         15b38:	ae06      	add	r6, sp, #24
         15b3a:	f8df a1e4 	ldr.w	sl, [pc, #484]	; 15d20 <ProprietaryProcedure@@Base+0x204>
         15b3e:	200f      	movs	r0, #15
         15b40:	f8df b1e0 	ldr.w	fp, [pc, #480]	; 15d24 <ProprietaryProcedure@@Base+0x208>
         15b44:	f7f3 ef76 	blx	9a34 <prctl@plt>
         15b48:	44f8      	add	r8, pc
         15b4a:	4629      	mov	r1, r5
         15b4c:	44f9      	add	r9, pc
         15b4e:	2210      	movs	r2, #16
         15b50:	44fa      	add	sl, pc
         15b52:	4630      	mov	r0, r6
         15b54:	44fb      	add	fp, pc
         15b56:	f7f3 ea40 	blx	8fd8 <memset@plt>
         15b5a:	a807      	add	r0, sp, #28
         15b5c:	f7f3 ef70 	blx	9a40 <sigemptyset@plt>
         15b60:	4b71      	ldr	r3, [pc, #452]	; (15d28 <ProprietaryProcedure@@Base+0x20c>)
         15b62:	462a      	mov	r2, r5
         15b64:	9508      	str	r5, [sp, #32]
         15b66:	4631      	mov	r1, r6
         15b68:	447b      	add	r3, pc
         15b6a:	681b      	ldr	r3, [r3, #0]
         15b6c:	200a      	movs	r0, #10
         15b6e:	9306      	str	r3, [sp, #24]
         15b70:	f7f3 ef6c 	blx	9a4c <sigaction@plt>
         15d02:	f7f3 eade 	blx	92c0 <__android_log_print@plt>
         15d06:	f8c4 5304 	str.w	r5, [r4, #772]	; 0x304
         15d0a:	4628      	mov	r0, r5
         15d0c:	b00b      	add	sp, #44	; 0x2c
         15d0e:	e8bd 8ff0 	ldmia.w	sp!, {r4, r5, r6, r7, r8, r9, sl, fp, pc}

      The addresses where this code is loaded can be easily computed by adding 0xf51f6000 to the file offsets shown in the first column. We see that a few calls to different external functions [6] are performed by ProprietaryProcedure, which is itself an exported symbol.

      I restarted the debug session, added a breakpoint at the start of ProprietaryProcedure, right after stmdb saves the state, and checked the stack values:

      (gdb) b *0xf520bb20
      Breakpoint 1 at 0xf520bb20
      (gdb) cont
      Breakpoint 1, 0xf520bb20 in ?? ()
      (gdb) p $sp
      $1 = (void *) 0xf49dde4c
      (gdb) x/16xw $sp
      0xf49dde4c:	0xf49de460	0x0007df00	0x00000000	0xf49dde70
      0xf49dde5c:	0xf49de694	0x00000000	0xf77e9000	0x00000000
      0xf49dde6c:	0xf75b4491	0x00000000	0xf49de460	0x00000000
      0xf49dde7c:	0x00000000	0xfd5b4eba	0xfe9dd4a3	0xf49de460

      We can see that the stack contains something, including a return address that looks valid (0xf75b4491). Note also that the procedure must never touch this part of the stack, as it belongs to the caller of ProprietaryProcedure.

      Now it is a simply a matter of bisecting the code between the beginning and the end of ProprietaryProcedure to find out where we are clobbering the stack. I will save you of developing here this tedious process. Instead, I will just show, that, in the end, it turned out that the call to sigemptyset() is the culprit [7]:

      (gdb) b *0xf520bb5c
      Breakpoint 1 at 0xf520bb5c
      (gdb) b *0xf520bb60
      Breakpoint 2 at 0xf520bb60
      (gdb) run
      Breakpoint 1, 0xf520bb5c in ?? ()
      (gdb) x/16xw 0xf49dde4c
      0xf49dde4c:	0xf49de460	0x0007df00	0x00000000	0xf49dde70
      0xf49dde5c:	0xf49de694	0x00000000	0xf77e9000	0x00000000
      0xf49dde6c:	0xf75b4491	0x00000000	0xf49de460	0x00000000
      0xf49dde7c:	0x00000000	0xfd5b4eba	0xfe9dd4a3	0xf49de460
      (gdb) cont
      Breakpoint 2, 0xf520bb60 in ?? ()
      (gdb) x/16xw 0xf49dde4c
      0xf49dde4c:	0x00000000	0x00000000	0x00000000	0x00000000
      0xf49dde5c:	0x00000000	0x00000000	0x00000000	0x00000000
      0xf49dde6c:	0x00000000	0x00000000	0x00000000	0x00000000
      0xf49dde7c:	0x00000000	0x00000000	0x00000000	0x00000000

      Note here that I am printing the part of the stack not reserved by the function (0xf49dde4c is the value of the sp before execution of the line at offset 0x15b20, see the code).

      What is going wrong here? Now, remember that at the beginning of the article I mentioned that we were using libhybris. libProprietary assumes a bionic environment, and the libc functions it calls are from bionic’s libc. However, libhybris has hooks for some bionic functions: for them bionic is not called, instead the hook is invoked. libhybris does this to avoid conflicts between bionic and glibc: for instance having two allocators fighting for process address space is a recipe for disaster, so malloc() and related functions are hooked and the hooks call in the end the glibc implementation. Signals related functions were hooked too, including sigemptyset(), and in this case the hook simply called glibc implementation.

      I looked at glibc and bionic implementations, in both cases sigemptyset() is a very simple utility function that clears with memset() a sigset_t variable. All pointed to different definitions of sigset_t depending on the library. Definition turned out to be a bit messy when looking at the code as it depended on build time definitions, so I resorted to gdb to print the type. For a executable compiled for glibc, I saw

      (gdb) ptype sigset_t
      type = struct {
          unsigned long __val[32];

      and for one using bionic

      (gdb) ptype sigset_t
      type = unsigned long

      This finally confirms where the bug is, and explains it: we are overwriting the stack because libProprietary reserves in the stack memory for bionic’s sigset_t, while we are using glibc’s sigemptyset(), which uses a different definition for it. As this definition is much bigger, the stack gets overwritten after the call to memset(). And we get the crash later when trying to restore registers when the function returns.

      After knowing this, the solution was simple: I removed the libhybris hooks for signal functions, recompiled it, and… all worked just fine, no crashes anymore!

      However, this is not the final solution: as signals are shared resources, it makes sense to hook them in libhybris. But, to do it properly, the hooks have to translate types between bionic in glibc, thing that we were not doing (we were simply calling glibc implementation). That, however, is “just work”.

      Of course I wondered why the heck a library that is kind of generic needs to mess around with signals, but hey, that is not my fault ;-).


      I can say I learned several things while debugging this:

      1. Not having the sources is terrible for debugging (well, I already knew this). Unfortunately not open sourcing the code is still a standard practice in part of the industry.
      2. The most interesting technical bit here is IMHO that we need to be very cautious with the backtrace that debuggers shows after a crash. If you start to see things that do not make sense it is possible that registers or stack have been messed up and the real crash happens elsewhere. Bear in mind that the very first thing to do when a program crashes is to make sure that we know the exact point where that happens.
      3. We have to be careful in ARM when disassembling, because if there is no debug information we could be seeing the wrong instruction set. We can check evenness of addresses used by bx/blx and of the lr to make sure we are in the right mode.
      4. Some times taking a look at assembly code can help us when debugging, even when we have the sources. Note that if I had had the C sources I would have seen the crash happening right when returning from a function, and it might not have been that immediate to find out that the stack was messed up. The assembly clearly pointed to an overwritten stack.
      5. Finally, I personally learned some bits of ARM architecture that I did not know, which was great.

      Well, this is it. I hope you enjoyed the (lengthy, I know) article. Thanks for your reading!

      [2] We can get the same result by executing in gdb set arm fallback-mode thumb, but changing the register seemed more pedagogical here.
      [6] In fact the calls are to the PLT section, which is inside the library. The PLT calls in turn, by using addresses in the GOT data section, either directly the function or the dynamic loader, as we are doing lazy loading. See, for instance.
      [7] I had to use two breakpoints between consecutive instructions because the “ni” gdb command was not working well here.

      Read more