Canonical Voices

Posts tagged with 'open-source'

The Ubuntu Developer Summit was held in Copenhagen last week, to discuss plans for the next six-month cycle of Ubuntu. This was the most productive UDS that I've been to — maybe it was the shorter four-day schedule, or the overlap with Linaro Connect, but it sure felt like a whirlwind of activity.

I thought I'd share some details about some of the sessions that cover areas I'm working on at the moment. In no particular order:

Improving cross-compilation

Blueprint: foundations-r-improve-cross-compilation

This plan is a part of a mutli-cycle effort to improve cross-compilation support in Ubuntu. Progress is generally going well — the consensus from the session was that the components are fairly close to complete, but we still need some work to pull those parts together into something usable.

So, this cycle we'll be working on getting that done. While we have a few bugfixes and infrastructure updates to do, one significant part of this cycle’s work will be to document the “best-practices” for cross builds in Ubuntu, on wiki.ubuntu.com. This process will be heavily based on existing pages on the Linaro wiki. Because most of the support for cross-building is already done, the actual process for cross-building should be fairly straightforward, but needs to be defined somewhere.

I'll post an update when we have a working draft on the Ubuntu wiki, stay tuned for details.

Rapid archive bringup for new hardware

Blueprint: foundations-r-rapid-archive-bringup

I'd really like for there to be a way to get an Ubuntu archive built “from scratch”, to enable custom toolchain/libc/other system components to be built and tested. This is typically useful when bringing up new hardware, or testing rebuilds with new compiler settings. Because we may be dealing with new hardware, doing this bootstrap through cross-compilation is something we'd like too.

Eventually, it would be great to have something as straightforward as the OpenEmbedded or OpenWRT build process to construct a repository with a core set of Ubuntu packages (say, minbase), for previously-unsupported hardware.

The archive bootstrap process isn't done often, and can require a large amount of manual intervention. At present, there's only a couple of folks who know how to get it working. The plan here is to document the bootstrap process in this cycle, so that others can replicate the process, and possibly improve the bits that are a little too janky for general consumption.

ARM64 / ARMv8 / aarch64 support

Blueprint: foundations-r-aarch64

This session is an update for progress on the support for ARMv8 processors in Ubuntu. While no general-purpose hardware exists at the moment, we want to have all the pieces ready for when we start seeing initial implementations. Because we don't have hardware yet, this work has to be done in a cross-build environment; another reason to keep on with the foundations-r-improve-cross-compilation plan!

So far, toolchain progress is going well, with initial cross toolchains available for quantal.

Although kernel support isn’t urgent at the moment, we’ll be building an initial kernel-headers package for aarch64. There's also a plan to get a page listing the aarch64-cross build status of core packages, so we'll know what is blocked for 64-bit ARM enablement.

We’ve also got a bunch of workitems for volunteers to fix cross-build issues as they arise. If you're interested, add a workitem in the blueprint, and keep an eye on it for updates.

Secure boot support in Ubuntu

Blueprint: foundations-r-secure-boot

This session covered progress of secure boot support as at the 12.10 Quantal Quetzal release, items that are planned for 13.04, and backports for 12.04.2.

As for 12.10, we’ve got the significant components of secure boot support into the release — the signed boot chain. The one part that hasn't hit 12.10 yet is the certificate management & update infrastructure, but that is planned to reach 12.10 by way of a not-too-distant-future update.

The foundations team also mentioned that they were starting the 12.04.2 backport right after UDS, which will bring secure boot support to our current “Long Term Support” (LTS) release. Since the LTS release is often preferred Ubuntu preinstall situations, this may be used as a base for hardware enablement on secure boot machines. Combined with the certificate management tools (described at sbkeysync & maintaining uefi key databases), and the requirement for “custom mode” in general-purpose hardware, this will allow for user-defined trust configuration in an LTS release.

As for 13.04, we're planning to update the shim package to a more recent version, which will have Matthew Garrett's work on the Machine Owner Key plan from SuSE.

We're also planning to figure out support for signed kernel modules, for users who wish to verify all kernel-level code. Of course, this will mean some changes to things like DKMS, which run custom module builds outside of the normal Ubuntu packages.

Netboot with secure boot is still in progress, and will require some fixes to GRUB2.

And finally, the sbsigntools codebase could do with some new testcases, particularly for the PE/COFF parsing code. If you're interested in contributing, please contact me at jeremy.kerr@canonical.com.

Read more

Cross-compile-o-naut wookey has recently posted a set of Debian & Ubuntu packages for an aarch64 (ie, 64-bit ARMv8) cross compiler:

The repositories here contain sources and binaries for the arm64 bootstrap in Debian (unstable) and Ubuntu (quantal). There are both toolchain and tools packages for amd64 build machines and arm64 binaries built with them. And corresponding sources.

For the lazy:

[jk@pecola ~]$ wget -O - http://people.debian.org/~wookey/bootstrap/bootstrap-archive.key |
    sudo apt-key add -
[jk@pecola ~]$ sudo apt-add-repository 'deb http://people.debian.org/~wookey/bootstrap/ubunturepo/ quantal-bootstrap main universe'
[jk@pecola ~]$ sudo apt-get update
[jk@pecola ~]$ sudo apt-get install --install-recommends gcc-4.7-aarch64-linux-gnu

... which gives you a cross compiler for aarch64:

[jk@pecola ~]$ cat helloworld.c 
#include <stdio.h>
int main(void) { printf("hello world\n"); return 0; }
[jk@pecola ~]$ aarch64-linux-gnu-gcc-4.7 -Wall -O2 -o helloworld helloworld.c
[jk@pecola ~]$ aarch64-linux-gnu-objdump -f helloworld 
helloworld:     file format elf64-littleaarch64
architecture: aarch64, flags 0x00000112:
EXEC_P, HAS_SYMS, D_PAGED
start address 0x0000000000400450

Read more

Most of the components of the 64-bit ARM toolchain have been released, so I've put together some details on building a cross compiler for aarch64. At present, this is only binutils & compiler (ie, no libc), so is probably not useful for applications. However, I have a 64-bit ARM kernel building without any trouble.

Update: looking for an easy way to install a cross-compiler on Ubuntu or debian? Check out aarch64 cross compiler packages for Ubuntu & Debian.

pre-built toolchain

If you're simply looking to download a cross compiler, here's one I've built earlier: aarch64-cross.tar.gz (.tar.gz, 85MB). It's built for an x86_64 build machine, using Ubuntu 12.04 LTS, but should work with other distributions too.

The toolchain is configured for paths in /opt/cross/. To install it, do a:

[jk@pecola ~]$ sudo mkdir /opt/cross
[jk@pecola ~]$ sudo chown $USER /opt/cross
[jk@pecola ~]$ tar -C /opt/cross/ -xf aarch64-x86_64-cross.tar.gz

If you'd like to build your own, here's how:

initial setup

We're going to be building in ~/build/aarch64-toolchain/, and installing into /opt/cross/aarch64/. If you'd prefer to use other paths, simply change these in the commands below.

[jk@pecola ~]$ mkdir -p ~/build/arm64-toolchain/
[jk@pecola ~]$ cd ~/build/arm64-toolchain/
[jk@pecola ~]$ prefix=/opt/cross/aarch64/

We'll also need a few packages for the build:

[jk@pecola ~]$ sudo apt-get install bison flex libmpfr-dev libmpc-dev texinfo

binutils

I have a git repository with a recent version of ARM's aarch64 support, plus a few minor updates at git://kernel.ubuntu.com/jk/arm64/binutils.git (or browse the gitweb view). To build:

Update: arm64 support has been merged into upstream binutils, so you can now use the official git repository. The commit 02b16151 builds successfully for me.

[jk@pecola arm64-toolchain]$ git clone git://gcc.gnu.org/git/binutils.git
[jk@pecola arm64-toolchain]$ cd binutils
[jk@pecola binutils]$ ./configure --prefix=$prefix --target=aarch64-none-linux
[jk@pecola binutils]$ make
[jk@pecola binutils]$ make install
[jk@pecola binutils]$ cd ..

kernel headers

Next up, the kernel headers. I'm using Catalin Marinas' kernel tree on kernel.org here. We don't need to do a full build (we don't have a compiler yet..), just the headers_install target.

[jk@pecola arm64-toolchain]$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/cmarinas/linux-aarch64.git
[jk@pecola arm64-toolchain]$ cd linux-aarch64
[jk@pecola linux-aarch64]$ git reset --hard b6fe1645
[jk@pecola linux-aarch64]$ make ARCH=arm64 INSTALL_HDR_PATH=$prefix headers_install
[jk@pecola linux-aarch64]$ cd ..

gcc

And now we should have things ready for the compiler build. I have a git tree up at git://kernel.ubuntu.com/jk/arm64/gcc.git (gitweb), but this is just the aarch64 branch of upstream gcc.

[jk@pecola arm64-toolchain]$ git clone git://kernel.ubuntu.com/jk/arm64/gcc.git
[jk@pecola arm64-toolchain]$ cd gcc/aarch64-branch/
[jk@pecola aarch64-branch]$ git reset --hard d6a1e14b
[jk@pecola aarch64-branch]$ ./configure --prefix=$prefix \
    --target=aarch64-none-linux --enable-languages=c \
    --disable-threads --disable-shared --disable-libmudflap \
    --disable-libssp --disable-libgomp --disable-libquadmath
[jk@pecola aarch64-branch]$ make
[jk@pecola aarch64-branch]$ make install
[jk@pecola aarch64-branch]$ cd ../..

That's it! You should have a working compiler for arm64 kernels in /opt/cross/aarch64.

Read more
Sidnei

Over the past few months, friends and family have been very curious about how my new job is going, and it’s been hard to stop for a moment and go into detail about it. I’ve been simply nodding and saying “It’s fine”.

This is an attempt at summarizing all the activity that happened in the last five months, though it’s far from being a short summary. If I had to pick a two words to describe my first five months at Canonical, it would be “Pure Awesomeness“. For a more detailed view, grab a cup of joe or your favorite other beverage and keep on reading.

Today is a special day. Exactly five months ago, on January 5th, I joined Canonical to work on the Landscape project.

It has been quite a ride so far, with two sprints with the Landscape team, AllHands and UDS in Barcelona just a week ago, and lots of excitement about the future. Saying that I’m completely stunned by the work everyone at Canonical has been putting together and how much the teams have grown in the last few months doesn’t do enough justice to it.

At AllHands and UDS we got a short preview of the things that are coming out in the next cycle and beyond. An example of that is the newly formed Design and User Experience (DUX) team which will not only be focusing on Ubuntu itself, but on many other areas across Canonical and the whole Open Source community in general.

At UDS, the DUX team had a special ‘booth’ where any person from any project could walk to them and get advice about their personal or favorite application. One person which has favorably used such advice already was Seif Lofty, from Gnome Zeitgeist. I met Seif during breakfast at the first day of UDS and I cannot describe how excited he was about simply *being* at UDS. And he was certainly twice as happy when he left.

Another person I had the joy to meet was David Siegel, of Gnome Do fame. We teamed during AllHands on the infamous “Fun In The Woods” activity, which had some people walking like zombies on the day after.

Even more importantly, I was able to meet many many more other colleagues from different teams and wrap up some loose ends from tasks that I got started during the first couple months. I’m definitely impressed by the amount of plain brilliant people that are part of Canonical as of today.

In a sense, being at a Canonical event is very much like being at a Plone conference. Everyone seems to be very receptive about new ideas and very friendly and laid back. And as a bonus, I was able to exercise my (not so secret now) power of throwing some crazy ideas around and see how they influence people. And man, I’m already impressed by the outcomes, just a week after the fact.

To me, this has been the most rewarding thing so far, to be able make big contributions not in lines of code, but in ideas that can make a concrete difference in the hands of the right people. This is something that can only be possible at a company the size Canonical is at the moment, where it’s just big enough that you can grab a mind or two to push an agenda without affecting the rest of the team and still small enough that you can influence decisions.

As my colleague Jamu would best describe, “I’m PUMPED!”. :)

But that’s not all. Software-wise, I was able to make some big contributions too. The Landscape team just finished a 6 month development cycle that brought many cool features to life. I’m really happy with that, and specially with the speed that this team can get features from the black board into reality. It’s also a much different environment than what I was used to, with very well-defined and refined processes for ensuring the overall quality of anything that is produced. One process that I’m specially enjoying is the requirement for two positive reviews before landing a branch. I hope to talk more about that soon (that is, sooner than 5 months from now *wink*).

As for my role in the team, it is quite different than what I’m used to. I’ve been focusing a lot on the UI aspects of Landscape, on ways to make things more obvious and more streamlined. I’ve been also writing a ton of Javascript, and collaborating with other teams to define better policies for Javascript testing in general. And finally, we will now have a person from the DUX team dedicated to working with us, which will push work on the Landscape UI even further.

I also had the chance of interacting with the Launchpad team, which has a much more refined process due to the size of their team. Over at Launchpad, I started a branch back in mid-December, even before starting at Canonical, to allow Launchpad to use the Chameleon Template Engine.

That was another wild ride, and during the course of this project I was able to contribute tons of fixes upstream to Malthe Borch to make Chameleon even more compatible with plain old ZPT. In fact, it is so compatible at the moment that due to the magic of z3c.ptcompat Launchpad will be able to run *both* Chameleon and ZPT with the flip of an environment variable. Even more stunning, the changes required to code were minimal, basically changing imports to use z3c.ptcompat, and in templates we’ve had to fix some non-XHTML compliant ones and remove unused i18n tags. I am happy to announce that this branch will soon be merged (it was submitted to PQM, successfully accepted and is waiting to land the buildbot queue). The bad news is that not all tests pass at the moment with Chameleon enabled, but we will be dogfooding and fixing those tests as we go. It was too much pain already to maintain a nearly 6 month old branch outside the main tree. ;)

I am really interested in many of the things that the Launchpad team is doing, process-wise. The PQM seems like a very nice idea for a bigger team like theirs, though it would probably be useful to our smaller team in Landscape too, and to others in general. Hopefully I will get a chance to explore it more and talk about it during the upcoming FISL 10, in Porto Alegre.

Lastly, but not least important, I’m also working on getting nightly builds of the Bzr Installer for Windows rolling, and a more streamlined process for the official builds. Karl Fogel, of Producing OSS fame, and our Launchpad Ombudsman is making sure I keep my promises about that, which is yet another great incentive.

All in all, there’s of course a ton of things I forgot to talk about and which happened in the last 5 months, but this post is already getting too long so I will stop right here and save some of the meat for a future one. Stay tuned!

Read more
Sidnei

No, this is not a blog post about the kind of landscape you’re accustomed to, though it might trigger a few ‘I want to be a Landscape Architect’ thoughts from a person or two.

The news this time is that I’m going through a landscape change myself, and to me it’s still a bit scary just to think about it.

The last such change in my life happened roughly ten years ago when I left my job as PHP Programmer and Systems Administrator at a small ISP to start my own company with a few colleagues from university. At that time, leaving PHP behind to learn this new (to me) fancy things called Zope and Python felt really weird, not to mention the fact that I was about 20 and knew no-one of my age that had successfully started a web development company (I mean, I live in a very small town, this is not Silicon Valley).

So cutting to the chase, I would like to let everybody know I will be joining the Landscape team at Canonical starting January 5th. I will not be leaving Enfold Systems though. We keep working together part time at least until April, where we expect to make at least one big release of a fully eggified Enfold Server based on Plone 3.2 in that timeframe. After April, I will still be doing work for Enfold Systems, but the time available for that will much more constrained.

Joining the Landscape team is very exciting to me, not only because I will get to work with some really smart people, but also because I will be working from home and being supervised by one of the top minds in the Brazilian Python scene: Gustavo Niemeyer, responsible for pearls like `Mocker`, `Storm` and `Smart`. And on top of that, I will get to learn new ways of managing and collaborating within a distributed software development team, a subject that was theme for my graduation thesis.

Last, but not least important, there’s a long story of back and forth between me and Canonical that dates back as far as 2004. On that year, DebConf 4 was happening in Porto Alegre which is about one hour drive from where I live.

Hanging out on the #zope3-dev irc channel I’ve noticed that one guy, Steve Alexander, connected from an IP range in Brazil. That struck me as odd because I didn’t know Steve very well, but maybe well enough to guess that something hot was going on. So Steve told me he was here for DebConf, and I self-invited me to stop by and say `hi`.

Arriving at DebConf, I met Steve personally for the first time, and again had an odd sensation there: Steve and about 30-40 folks where separated from the rest of the DebConf crowd on their own room, hacking away. I also met Mark Shuttleworth there and saw him talk about his space flight. That was one of the most exciting things and very heart-warming to me since my dream as a kid was to be an astronaut.

Later on that same day I came to know they were creating a new distro: they were taking name suggestions and voting. It was not surprise at least to me when I first saw Ubuntu being mentioned in the news, though I don’t recall clearly if that was one of the suggestions being voted.

Having just finished a big project using Zope 3 and relational databases, Steve quickly asked for my input on the preliminary design of a Zope 3-based web application for managing translations, which I promptly gave. The code I was looking at right there was the beginning of Launchpad. Up until very recently, Launchpad was powered by a creation of mine named `sqlos`, which has now been replaced by `storm`.

Steve asked me if I was interested in joining the team, but I had to decline given that I had just signed up with Enfold Systems, which was only about a month old at that time. I did put them in touch with a good friend I met only a couple years before: Christian Reis (kiko). In retrospect, that might have been one of the best contributions I made to Canonical, and to Open Source in general.

So, all in all, very exciting news for me. Canonical has a special place in my heart since I basically saw when Ubuntu and Launchpad were born and have some really good friends working there. The situation is no different with Enfold Systems. I have been working with Alan Runyan since before Enfold was Enfold, and really since before Plone was Plone: when Plone 1.0 was announced I was with Alan Runyan, Paul Everitt and Alexander Limi in Paris, at SolutionsLinux 2003, participating on a Zope 3 sprint, mind you.

To summarize, I will be splitting my time between Enfold Systems and Canonical between January 5th and April 1st 2009. After that I will be working full time at Canonical but still expect to contribute significantly at Enfold Systems. One of the responsibilities I have right now and which I don’t want to drop is building the Plone installers for Windows. Hopefully I will keep doing that. Unless some Ubuntu dude sneaks in by night and erases my Windows partition, that is. :)

Read more