Recently we had our online Ubuntu Developer Summit where we discussed a range of topics, defined next steps, and documented work items. The very last session at the event was an overall summary of the tracks (you can watch the video here), but I wanted to blog an overall summary too. These notes are quick and to the point, but they should give an overall idea of decisions made.

Client

• Content Handling -

  • Siloing apps.
  • Main applications will define a “main repo” and provide an API to deliver, share and access the data in the main repo.

• X.org

  • Want to update to 1.14 or even 1.15 if the video ABI doesn’t change.

• System Settings

  • Focus on the phone settings defined here.

• Scopes

  • Scopes that didn’t land in 13.04 should land within 2 weeks.
  • Several scopes will be migrated from Python to either C++ or Go for memory purposes.

• Chromium

  • Expressed interest in moving to Chromium as default for a better user experience. Gathered feedback on the possible move. Next steps are to take discussion to the mailing list.

• Unity 8/Mir Preview in 13.10

  • Want to have a preview of Ubuntu 8 (Phablet UI) running on Mir as an optional session (installable from universe or PPA, most likely).

Foundations

• Reviewed the current 13.10 release schedule found several changes made in 13.04 that mistakenly hadn’t been carried over, such as later freeze dates and one fewer alpha; Adam Conrad will be syncing all this up and sending mail to the ubuntu-release list for review.
• We discussed the positioning of the development release in light of some conversations last cycle, and put some more flesh on the design for making it easier for people to follow along with the development release all the time.
• This cycle, we’ll be bringing up a new 64-bit ARM architecture based on cross-building work done last cycle, and we’ll update developers on that once we get closer to the point of starting up builds in Launchpad.
• Moving forward with click packages. Fleshed out ideas on source package provision, integrating with existing client package management stacks, and clarifying some other things like the security model.
• For image based upgrades, the team held a demo and Q&A for the current proposed solution, which is split into client, server, and upgrader; client is going well and expected to land by the end of June, server is currently blocked on infrastructure but should be ready around the same time, and Ondrej Kubik has been making good progress at tweaking the CyanogenMod recovery environment for the upgrader.
• Firmed up the plan for packaging Android components for Ubuntu Touch images.
• Upstart will be used as the standard way of spawning desktop apps for Unity on touch devices and ideally on desktop too (Unity 7 and 8). This will let us make sure we only have one instance per app, and will make it easy to apply AppArmor, seccomp and cgroup confinement consistently to all apps.
• Defined a goal to reduce the amount of time it takes to prepare, test and make a Checkbox release, automating more of the process. This will benefit people who use the Checkbox tool as part of their daily work. It’s possible that Checkbox may move to Universe, although this needs some more discussion.
• The server certification tools are being reengineered to use the new plainbox engine as their core. This will preserve the existing UI, but we’ll have co-installable packages with the new core, and will eventually switch over to the new tools.
• The cert tools and test suite are being upgraded to work well on ARM for our hyperscale and mobile work, fixing any issues so we can get full, clean test runs on ARM servers. MaaS will be used for provisioning, and tested as a part of the ARM server solution.
• We will be basing the primary kernels for 13.10 on Linux 3.10, but will strongly consider 3.11 depending on timing. For Ubuntu Touch devices, we already have kernels available for Nexus 4 and 7, and plan to also bring up kernels for Galaxy Nexus and Nexus 10. We’ll provide a 13.04 hardware enablement kernel in the 12.04.3 point release.
• In terms of Ubuntu Touch power management, we have some preliminary baselines against Android on Nexus 4 and will replicate those on other devices, although they won’t be entirely meaningful until things like Mir land. We’ve written some new utilities such as eventstat to track down problems here. On power management policy, we agreed key requirements for the system power manager and we’ll extend powerd to serve our immediate needs.

Community

• Community Roundtable:

  • Approved LoCo teams are no more, will be verified teams.
  • Bringing back fortnightly leadership meetings.
  • Ubuntu Advocacy Kit is driving to 1.0.
  • Gathered UDS feedback.

• Ubuntu Community Website

  • Great discussion which clarified everybody’s involvement in the project.
  • Clear roadmap for completing the content and design in the next few weeks.
  • Design and web team have the templates we need to finish the work.
  • No discussion with IS yet around deployment – this will be coordinated next week.

• Ubuntu Womens Session

  • Several events planned to get more people involved and the word out (Career Days, UOW, etc.).
  • Discussion about a women in technology themed event at CLS.

• Ubuntu Status Tracker

  • The status tracker is many things to many different teams, but we managed to figure out a number of issues we can tackle, which should make everybody’s lives easier.
  • Removal of kanban view.
  • Add links from team pages to milestones pages.
  • Set up a meeting to discuss setting up an “ongoing” dev series.

• Ubuntu On Air! Discussion

  • Issues with supporting multiple hosts.
  • Discussion about building support into summit and re-using vUDS components to support more shows and multiple hosts.
  • We want to open it up to more contributors, so we get more variety into the shows.

• Development Onramp for Touch / Unity Next

  • Goals to improve docs.
  • We will track contributions to all the projects to see how we improve.
  • Increased focus on testing, coordination with the SDK team.

• Documentation Team

  • Update Getting Started Page, review current docs and previous mailing list feedback.
  • Regular doc review cadence and more health check meetings.
  • Focus on content in the UAK.

• Ubuntu Enterprise Desktop Discussions

  • Another meeting will be planned to get more input from users of enterprise desktops.
  • Some common issues were identified and discussed:
  • outdated cfengine package
  • access to Firefox/Thunderbird packages before publication (resolved)
  • support for livemeeting/linc

• More Ubuntu Touch images

  • We identified the current blockers and will simplify thingsby using an image without firmware blobs, so they can be added by a local tool afterwards.
  • After the rebase to saucy we will also update the docs accordingly.
  • Kernel images for devices will first live in PPA, afterwards probably in universe.

• Regular Ubuntu Development Updates

  • Organise regular Ubuntu on Air Hangouts to which we invite people from news sites as moderators.
  • Briefly summarise work from the last week(s).
  • Ask engineers to demo/showcase interesting developments.
  • Do Q&A sessions.
  • Also invite members of governance teams along.

Cloud and Server

• Openstack Next Steps

  • Looked at some high level areas for this cycle, avoiding digging into areas covered by other sessions. We decided that at current, moving over to Git for our packaging work doesn’t add value. We also agreed to clean up on some cruft within the packaging branches.

• Cloud Archive Status Check

  • Decided we had to support Grizzly for 12 months, which exposes a 3 month support gap from the backing Raring release. Need to discuss with the security team about how to fill this gap. Reviewed proposal for SRU cadence and tentatively rubber stamped.
  • Better co-ordination around trumping by Security dates, specifically if it covers more than one project.
  • Looked at using updates as a reason to increase our messaging.

• 12.04.x images with LTS Enablement Kernel

  • The cloud images currently only contain the Precise (3.2) kernel. Discussed adding the kernel HWE stack to cloud images. We need to document how to enable backports, clearly state the support, and possibly tool cloud-init to handle updating the kernel on boot if folks need a more recent kernel on boot.
  • We will not be creating new images with the HWE kernel for the default images. The HWE kernel will be used for Clouds that have a high velocity of change in the Hypervisor (i.e. Windows Azure). For the regular images, we will investigate tooling in cloud-init and other places to make the ingestion of the HWE kernels easier, such as enhancing the documentation, allowing for easier enablement of backports, and making it easier to enable the HWE kernel at boot time.

• Cloud-Init for Vagrant

  • We will create a good Ubuntu development workflow for Vagrant users (cross platform OSX, Windows). Ben Howard will investigate cloud-init tooling as well as the best method to enable the DKMS modules.

• Cloud Init & Cloud Image Development for Saucy

  • We will define the development work to improve cloud-init and cloud images for the saucy cycle.
  • Discussed work on pre-cloud init phase, vendor hooks, cloud init plugin, and rebuding tools.

• Juju Core Development

  • 1.10 version of juju available in backports for 13.04, and should be available in precise backports soon.
  • Look for releases in juju/dev ppa updating weekly, juju PPA monthly, and have stable release go into backports (couple of times per cycle).

• OpenStack Hypervisors

  • HyperV support is currently untested.
  • VMWare support in charms, but not primary supported charms.
  • We need a matrix to demonstrate interoperability and support of each variation.
  • Need to work out additional hardening support.

• Openstack QA

  • Building on our great history, moving away from per commit hardware testing to a more fluid multi virtualised separated environment, allowing greater interoperability testing. Hardware Cert term showed interest in getting more involved. The scope of this will be ratified when the interop matrix is created.

• Flag Bearer Charms

  • Will improve flag bearer charm integration testing.
  • Implement list of reference charms.
  • Develop Percona backup XtraBackup flag bearer charm.
  • Document flag bearer and reference charm criteria in best practices.
  • Discuss flag bearer charms on mailing list.

• Charm Policy Review

  • Add into policy for a charm to provide a config option to specify the version. The other items such as installation location (ie /srv), implementation of common subordinates, backups are to be added to best practices. The 3 ack on charm reviews is still under discussion.
  • Split Juju docs best practices and policy sections.

• Audit Charms

  • Discussed re-reviewing the current charms in the charm store to ensure accurate readmes, tests, functionality, rating, categories, and icons. The workflow was discussed for queues, and which charms to tackle first.

• Charm Development Tooling

  • Discussed gathering all the different charm development tools into one central package. These charm development tools include charm-tools, charmsupport, juju-gui,openstack-charm-helpers. Folks also discussed how the tools could be improved, and used as a singular set.

• Juju Framework Charm for Server Application Technologies

  • Discussed further building out of the Django, Rails, Node.js, and possibly Java.

• Improve Juju Documentation

  • Make a better user and charm developer experience for juju.ubuntu.com/docs. Discussed getting a permanent beta site going, methods to get documentation contributions. Hopefully a revamped docs will be in production in the next couple of weeks, and if not we’ll have a beta site very shortly (days).

• Juju Charm Testing

  • Currently jenkins.qu.u.c has graph testing showing reliable results. Marco will be landing integration soon (days), with a more formal testing framework to follow (weeks).
  • Some ideas discussed were to gate charm store commits on testing, showing testing status in the GUI, and pre-deployment testing. Test examples will be made available along with a charm testing school.

• Add User Feedback loops and Social Networking to Charm Store Charm Pages

  • Discussed making sure users have a method to give and receive feedback on a per charm basis. We currently have social networking (+1s, Likes, Tweets) in addition to downloads, quality rating, bug links, and testing status. Some ideas were to get clarification from design on showing social networking numbers, as well as a ‘leave feedback’ form.

• Juju GUI Development

  • Discussed development done, and upcoming work. Covered ideas such as design, bundles, diagnostics, user data, juju feature parity, maintenance and support.

• Improving QA for seeded server packages

  • Established three distinctive areas of testing, these are upstream test suites which typically run at build-time, integration tests via dep8 and service level testing which often requires multiple nodes and is conducted using juju.
  • We established that there is the regression test suite that can be included in many of the packages directly, with the requirement that we package some of the common ubuntu testing libraries.
  • Discussed some areas of standardisation for dep8 testing.

• Fastpath installer work for 13.10

  • Established what FPI is, and the processes which are part of it.
  • Fast Path installer will be delivered as a installable package in Ubuntu, most likely in python. The interface to it will we yaml formatted configuration.

• OpenStack Charm work for Saucy/Havana

  • Migrate all charms to be python based.
  • Consolidation into new charm-helpers nextgen library.
  • Complete SSL/HTTPS support into charms.
  • Integration of wiki and help documentation, upstream series aligned with upgrading notes.
  • Design around how proprietary+1 plugins will be integrated into core charms for Cinder and Quantum.

• Investigate alternatives to mysql

  • Agreed that the best course of action was to maintain mysql for this cycle, and try and support other flavours of mysql getting into Ubuntu via Debian.

• Ceph activities for Saucy

  • Dumpling release will be out in August (co-incides with FF for Saucy) so will be target for this release.
  • Lots of incremental improvements in efficiency and performance, full RESTful API for RADOS Gateway admin features, block device encryption for data at rest.
  • ceph-deploy (upstream cross platform deploy tooling) will be packaged.
  • Implementation of more automated testing during Saucy.

• HA Openstack charms V2

  • Reviewed the current state of HA support in Openstack charms. Percona has volunteered to charm their offering, allowing great coverage by their mysql HA variant for active/active clustering.
  • Work also on active/active and brokerless messaging options (ZeroMQ) and incremental improvements for service check monitoring in load balancers.
  • Cluster stack (Corosync/Pacemaker) to be reviewed and upgraded for Saucy in preparation for 14.04.

• MongoDB activities for Saucy

  • File Main inclusion report for Mongo to support Ceilometer and Juju use cases. Raise a Micro Release Exemption (MRE) to the techboard, as point releases are bug fix only.
  • Upstream armhf enablement patches. Re-sync with Debian. OpenSSL license exception.

• Virtualization Stack Work for Saucy

  • If debian libcgroup maintainer finds time, we’d like to merge cgroup-lite into libcgroup. For per-user configuration, first make it default-off optional, conditional on userns sysctl being enabled.
  • LXC work is going well on track to 14.04 (and lxc 1.0) roadmaps. For this cycle, we’d like to get user namespaces enabled in the saucy kernel with a new off-by-default sysctl controling unprivileged use, and complete the ability to create and start basic containers without privilege; add console, attach and snapshot to the API, complete the create API function, and convert all of the lxc-* programs to use the API; write a libvirt driver based upon the API, and a patch to enable testing it with openstack; write loopback and qcow2 block device drivers; Get the subuid (user namespace enablement) patches into the shadow package; discuss with the community the maintenance of stable trees; improve the API thread safety; and work our distro lxc tests into the upstream package (akin to how it is done in netcf).
  • In edk2, we want to contribute to the implementation of the ability to save and restore nvvars from the ovmf bios from qemu. We’ll fix the apparmor bug preventing the block device mounting in libvirt-lxc, which is blocking use of that feature by openstack.
  • We intend to merge libvirt at least at version 1.0.6, qemu at 1.5, and hopefully xen 4.3. We’ll follow up on citrix’ plans for xcp. The blueprint lists additional xen work planned. We’ll also look into default use of openvswitch bridges in libvirt.

Quality Assurance

• Core Apps

  • Autopilot testcases written for ubuntu core applications will be checked to ensure they pass before auto-landing updates in the ubuntu touch images.
  • The quality community team will help core application developers develop a suite of manual testcases for each ubuntu core application. These will be run as part of the verification process for the 1.0 stable release of each application.

• Testcases

  • Add testcases so all default desktop applications for each flavor are covered.
  • Expand and improve server testcases to allow more participation by those who might lack domain specific knowledge and/or hardware.

• Growth/Experience

  • Make available documentation more accessible by linking to it from the tools we use for testing, like the qatracker.
  • Continue holding testing ‘how-to’ and knowledge sharing sessions during UDW, UOW, as part of UGJ, and on ubuntu on air.
  • Add testing achievements to the ubuntu accomplishments project.

• Ubuntu Touch

  • Ubuntu Touch images will be smoke tested using the pending/current model already in use for other images. This ensures no image is published for general consumption that doesn’t pass a set of tests ensuring basic functionality of the image.
  • Current Ubuntu Touch autopilot tests for the core applications will be investigated for use as part of these smoke tests.
  • The concept of smoke test is going to be expanded to cover a no regression build.

• Autopilot

  • Autopilot 1.3 is now released and will be available in raring and saucy. No quantal support is planned. Precise support is being examined, but requires further investigation and backporting work.
  • Autopilot developers will now be available on #ubuntu-autopilot — no need to always ping thomi!

• Mir

  • Planned tests for stressing mir to ensure good behavior during stressed conditions for things like OOM, memory leaks and race conditions.
  • Stress tests targeted to be run as often as possible, but might be limited due to time constraints of wanting to run the tests over a longer period of time.

• UTAH

  • UTAH will be expanded to include automated upgrade testing capabilities. UTAH jobs will be created for bootstrapping base images, for performing upgrades, and running post-upgrade tests. The old auto-upgrade-testing tool can still be used by flavors if desired.

• Dashboard

  • Create high-level views of the state of quality in ubuntu by aggregating results of test runs. This will allow for ‘problem’ areas within ubuntu to be more easily identified and targetted for further testing or investigation by interested parties. You can follow this work on the QA dashboard here.

• Upstream

  • autopilot-gtk will now be maintained by the upstream QA team. Bug fixes and outstanding issues will be solved in order to allow for the autopilot desktop tests to run
  • Once running properly, the autopilot desktop tests will become a part of daily image testing
  • Continue development on umockdev to add support for more exotic networking tests (eg, 3G) and research sound testing

As ever, you can track progress on work items on status.ubuntu.com and we hope to see you at the next UDS in three months.

Read more

For a while, I have been wanting to write about MAAS and how it can easily deploy workloads (specially OpenStack) with Juju, and the time has finally come. This will be the first of a series of posts where I’ll provide an Overview of how to quickly get started with MAAS and Juju.

What is MAAS?

I think that MAAS does not require introduction, but if people really need to know, this awesome video will provide a far better explanation than the one I can give in this blog post.

http://youtu.be/J1XH0SQARgo

 

Components and Architecture

MAAS have been designed in such a way that it can be deployed in different architectures and network environments. MAAS can be deployed as both, a Single-Node or Multi-Node Architecture. This allows MAAS to be a scalable deployment system to meet your needs. It has two basic components, the MAAS Region Controller and the MAAS Cluster Controller.

Region Controller

The MAAS Region Controller is the component the users interface with, and is the one that controls the Cluster Controllers. It is the place of the WebUI and API. The Region Controller is also the place for the MAAS meta-data server for cloud-init, as well as the place where the DNS server runs. The region controller also configures a rsyslogd server to log the installation process, as well as a proxy (squid-deb-proxy) that is used to cache the debian packages. The preseeds used for the different stages of the process are also being stored here.

Cluster Controller

The MAAS Cluster Controller only interfaces with the Region controller and is the one in charge of provisioning in general. The Cluster Controller is the place the TFTP and DHCP server(s) are located. This is the place where both the PXE files and ephemeral images are being stored. It is also the Cluster Controller’s job to power on/off the managed nodes (if configured).

The Architecture

As you can see in the image above, MAAS can be deployed in both a single node or multi-node. The way MAAS has being designed makes MAAS highly scalable allowing to add more Cluster Controllers that will manage a different pool of machines. A single-node scenario can become in a multi-node scenario by simply adding more Cluster Controllers. Each Cluster Controller has to register with the Region Controller, and each can be configured to manage a different Network. The way has this is intended to work is that each Cluster Controller will manage a different pool of machines in different networks (for provisioning), allowing MAAS to manage hundreds of machines. This is completely transparent to users because MAAS makes the machines available to them as a single pool of machines, which can all be used for deploying/orchestrating your services with juju.

How Does It Work?

MAAS has 3 basic stages. These are Enlistment, Commissioning and Deployment which are explained below:

Enlistment

The enlistment process is the process on which a new machine is registered to MAAS. When a new machine is started, it will obtain an IP address and PXE boot from the MAAS Cluster Controller. The PXE boot process will instruct the machine to load an ephemeral image that will run and perform an initial discovery process (via a preseed fed to cloud-init). This discovery process will obtain basic information such as network interfaces, MAC addresses and the machine’s architecture. Once this information is gathered, a request to register the machine is made to the MAAS Region Controller. Once this happens, the machine will appear in MAAS with a Declared state.

Commissioning

The commissioning process is the process where MAAS collects hardware information, such as the number of CPU cores, RAM memory, disk size, etc, which can be later used as constraints. Once the machine has been enlisted (Declared State), the machine must be accepted into the MAAS in order for the commissioning processes to begin and for it to be ready for deployment. For example, in the WebUI, an “Accept & Commission” button will be present. Once the machine gets accepted into MAAS, the machine will PXE boot from the MAAS Cluster Controller and will be instructed to run the same ephemeral image (again). This time, however, the commissioning process will be instructed to gather more information about the machine, which will be sent back to the MAAS region controller (via cloud-init from MAAS meta-data server). Once this process has finished, the machine information will be updated it will change to Ready state. This status means that the machine is ready for deployment.

Deployment

Once the machines are in Ready state, they can be used for deployment. Deployment can happen with both juju or the maas-cli (or even the WebUI). The maas-cli will only allow you to install Ubuntu on the machine, while juju will not only allow you to deploy Ubuntu on them, but will allow you to orchestrate services. When a machine has been deployed, its state will change to Allocated to <user>. This state means that the machine is in use by the user who requested its deployment.

Releasing Machines

Once a user doesn’t need the machine anymore, it can be released and its status will change from Allocated to <user> back to Ready. This means that the machine will be turned off and will be made available for later use.

But… How do Machines Turn On/Off?

Now, you might be wondering how are the machines being turned on/off or who is the one in charge of that. MAAS can manage power devices, such as IPMI/iLO, Sentry Switch CDU’s, or even virsh. By default, we expect that all the machines being controlled by MAAS have IPMI/iLO cards. So if your machines do, MAAS will attempt to auto-detect and auto-configure your IPMI/iLO cards during the Enlistment and Commissioning processes. Once the machines are Accepted into MAAS (after enlistment) they will be turned on automatically and they will be Commissioned (that is if IPMI was discovered and configured correctly).. This also means that every time a machine is being deployed, they will be turned on automatically.

Note that MAAS not only handles physical machines, it can also handle Virtual Machines, hence the virsh power management type. However, you will have to manually configure the details in order for MAAS to manage these virtual machines and turn them on/off automatically.

Read more

If you're running an Android device with GNU userland Linux in a chroot and need a full network access over USB cable (so that you can use your laptop/desktop machine's network connection from the device), here's a quick primer on how it can be set up.

When doing Openmoko hacking, one always first plugged in the USB cable and forwarded network, or like I did later forwarded network over Bluetooth. It was mostly because the WiFi was quite unstable with many of the kernels.

I recently found out myself using a chroot on a Nexus 4 without working WiFi, so instead of my usual WiFi usage I needed network over USB... trivial, of course, except that there's Android on the way and I'm a Android newbie. Thanks to ZDmitry on Freenode, I got the bits for the Android part so I got it working.

On device, have eg. data/usb.sh with the following contents.

#!/system/xbin/sh
CHROOT="/data/chroot"

ip addr add 192.168.137.2/30 dev usb0
ip link set usb0 up
ip route delete default
ip route add default via 192.168.137.1;
setprop net.dns1 8.8.8.8
echo 'nameserver 8.8.8.8' >> $CHROOT/run/resolvconf/resolv.conf

On the host, execute the following:

adb shell setprop sys.usb.config rndis,adb
adb shell data/usb.sh
sudo ifconfig usb0 192.168.137.1
sudo iptables -A POSTROUTING -t nat -j MASQUERADE -s 192.168.137.0/24
echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 
sudo iptables -P FORWARD ACCEPT

This works at least with Ubuntu saucy chroot. The main difference in some other distro might be whether the resolv.conf has moved to /run or not. You should be now all set up to browse / apt-get stuff from the device again.

Update: Clarified that this is to forward the desktop/laptop's network connection to the device so that network is accessible from the device over USB.

Read more

Recently there was yet another storm in a teacup that distracted us from creating and sharing Ubuntu and our flavors with others. I am not going to dive into the details of this particular incident…it has been exhaustively documented elsewhere…but at the heart of this case was a concern around the conduct in which some folks engaged around something they disagreed with. This is not the first time we have seen disappointing conduct in a debate, and I wanted to share some thoughts on this too.

In every community I have worked in I have tried to build an environment in which all view points that challenge decisions or decision makers are welcome with the requirement that they are built on a platform of respectful discourse; this is the essence of our Code Of Conduct. Within the context of an Open Source community we also encourage this engagement around differences to be expressed as solutions with a focus on solving problems; this helps us to be productive and move the project forward. This is why we have such a strong emphasis on blueprints, specs, bugs, and other ways of expressing issues and exploring solutions.

Within the context of this most recent issue I saw three problems (problems I have seen present in other similar arguments too):

1. Irrespective of the voracity or content of an opinion we must never forget to be respectful and polite in the way we express and engage with others, irrespective of whether you are a volunteer, Canonical employee, or otherwise. Respect must always be present in our discourse, irrespective of the content of our opinions; without it we become a barbaric people and lose the magic that brought this wonderful set of minds together in the first place. There is simply no excuse for rudeness, and inflammatory FUD that has no evidence to back it up other than presumed ill-intent serves nothing but to demotive folks and ratchet up the flames, as opposed to resolve the issue and make things better.
2. Trust needs to be earned, but trust should always be built within the wider context of a set of contributions and conduct. Unfortunately some folks consider decisions they disagree with to be a basis for (a) entering into a paranoid debate about the “real reason” the individual or company made that decision (and typically not believing the rationale provided by said decision-maker) and (b) seemingly forgetting about all the other positive contributions that the person or company has contributed. I can assure you there is no nefarious scheme at place at Canonical; our goals are well known in the community. If I felt Canonical was fundamentally trying to demote and shut the community out, I wouldn’t work here; I have no interest in working for a company that doesn’t understand the value of community, and I am not worried about finding suitable employment elsewhere. I work at Canonical because I believe our goals with Ubuntu are just and the company’s commitment to our community is sincere.
3. Ubuntu is not a consensus-based community. Consensus communities rarely work, and I am not aware of any Open Source project that bases their work on wider consensus in the community. It would be impossible and impractical to notify our community of every decision we make, let alone try to base a decision on a majority view, but we do try to ensure that major changes are communicated to our leaders first (this is something we have been driving improvements in recently). We always need to find the right balance between transparency and JFDI, and sometimes the balance isnt’t quite there, but that does not mean there is some kind of illuminati-ish scheme going on behind the scenes.

Ubuntu is a community filled with passionate people, and I love that we have folks who are critical of our direction and decisions. If everyone agreed with what we are doing, we would not always make the right decisions, and our diversity is what makes Ubuntu and our flavors such a great place to participate.

As I said at the beginning of this post, it is important that all viewpoints are welcome, but we have to get the tone and conduct of some of these debates under control. The sheer level of sensationalist and confrontational language that is often in place in these disagreements doesn’t serve anyone but hungry journalists looking for page hits.

Now, I am not suggesting here that anyone should change any of their viewpoints. If you vehemently disagree with an aspect of what we are doing in Ubuntu or at Canonical, that is fine and of course, welcome. What I am appealing to everyone though is to treat others like you wish to be treated, with respect and dignity, and lets keep the sensationalism out of our community and focus on what we do best…building a world-class Free Software platform and its rich ecosystem of flavors.

Read more

As many of you will know, our goal is to get the Ubuntu phone in a state where it can be used on a daily basis for testing, and importantly, finding bugs, UI issues, and other details that help us to refine the overall Ubuntu Touch experience. Progress is on-track for the end of May.

I decided to start dogfooding a little early (please remember, we are shooting for the beginning of July to be broadly in shape for dogfooding, so if you try, don’t expect things to be ready right now), so today I put my SIM card in my Galaxy Nexus with Ubuntu Touch and things are working pretty well so far. It seems that my data is no longer getting wiped on image updates, which helps testing significantly, so I am regularly upgrading with the daily images.

As ever, if you decide to test, you are doing so at your own risk…don’t be surprised to see bugs, crashes, and potential data loss (although I have not seen any data loss so far).

Some notes about my experience dogfooding:

• Making and recieving phone calls works well. I am using T-Mobile as my network.
• Sending and recieving texts works well too. Messages appear chronologically.
• Contact syncing is not in place but Sergio blogged about how to sync your contacts from Google. This has made my phone infinitely more useful and rather nicely, it pulls in the avatars too so I can see who is calling me.
• Browsing and connecting to wireless networks works well.
• The browser works well overall, although currently requires wifi (3G browsing coming soon).
• Camera works well (for still photos, video not implemented yet) and I can browse my pictures in the gallery.
• Many of the community-written core apps are present and working. Calendar lets me save and browse calendar events (although syncing with a calendar service is not there yet). Weather shows me the weather for my area right now and a week long forcast. Calculator is working and largely feature-complete. Other core apps are on their way to the daily image soon.
• Overall the core Unity UI is working well. I can search for apps, load them, quit them, multi-tasting works well, and the indicators work (for adjusting volume etc).

The primary blockers in my way right now for normal use out and about are:

• The screen does not auto shut-off. This means if the screen gets turned on in my pocket it never turns off and the battery dies.
• Speakerphone not wired into the UI yet.
• Can’t set the time on the phone yet. Also, the alarm feature in the clock doesn’t work; I need this to get me up in the morning.
• Not so much a blocker, but the phone is still filled with example material and contacts. They need to be removed.

All of these are on the TODO list for completion by the end of the month.

I have been filing bugs for a bunch of the issues I am seeing on a day to day basis and the team are working hard to hit the end of May goal. Overall progress is looking good.

Although I have been using the daily images for quite some time on a phone without a SIM card, using as an actual phone is even more motivating than before. I can feel the phone coming together and when we get many of these issues fixed, it is going to deliver a far superior experience than the Android phone I was using before.

Read more

Here is an inexpensive Ubuntu notebook, the ASUS X201E-DH01.

• Intel Celeron 847 (1.1GHz) Sandy Bridge
• 4 GB DDR3
• 320 GB 5400 rpm Hard Drive
• 11.6-Inch Screen, Intel GMA HD Graphic card
• 1 USB 3.0 and 2 USB 2.0
• SD MMC Card Reader
• WiFi, Ethernet and Bluetooth 4.0
• 1 HDMI and 1 VGA
• 5 Hours claimed battery life.
• Light Weight: 1.3 Kgs (2.9 Pounds)
• Ubuntu 12.04 preinstalled!

It is not the fastest PC around, but enough for day to day tasks. Runs faster on Ubuntu than Windows. Is light weight for people on the move, inexpensive and has enough of ports.

Read more

A while back I started a project called the Ubuntu Advocacy Kit. The goal is simple: create a single downloadable kit that provides all the information and materials you need to go out and help advocate Ubuntu and our flavors to others. The project lives here on Launchpad and is available in this daily PPA. If you want to see the kit in action just run:

sudo add-apt-repository ppa:uak-admins/uak
sudo apt-get update
sudo apt-get install uak-en

Now open the dash and search for “advocacy”. Click the icon to see the kit load in your browser.

We discussed the UAK this week at UDS and I want to get the kit to 1.0 level of completeness. This doesn’t require a huge amount of work, just getting a core set of content written up in a concise, simple, but detailed fashion. I want to complete this work and then get the kit up on loco.ubuntu.com as something people can download to get started advocating Ubuntu and our flavors.

I have created a blueprint to track this work and I am stubbing out a bunch of pages in the kit for pages that I think we will need as part of a 1.0 release.

And why are you telling me this?

Well, I am looking for help.

If you enjoy writing and have a knowledge of good quality advocacy, I would like to invite you to write some content. If you can just reply to this post in the comments (or anywhere else I tend to look, such as email or IRC), we coordinate who works on what and I will update the blueprint where appropriate.

Thanks for reading!

Read more

Recently the Technical Board made a decision to sunset Brainstorm, the site we have been using for some time to capture a list of what folks would like to see fixed and improved in Ubuntu. Although the site has been in operation for quite some time, it had fallen into something of a state of disrepair. Not only was it looking rather decrepit and old, but the ideas highlighted there were not curated and rendered into the Ubuntu development process. Some time ago the Technical Board took a work item to try to solve this problem by regularly curating the most popular items in brainstorm with a commentary around technical feasibility, but the members of the TB unfortunately didn’t have time to fulfill this. As such, brainstorm turned into a big list of random ideas, ranging from the sublime to the ridiculous, and largely ignored by the Ubuntu development process.

Now, some folks have mused on the decision to sunset brainstorm and wondered if this is somehow a reflection on our community and our openness to ideas. I don’t think this is the case. While it is always important to build an environment where ideas are openly discussed and debated, ideas are free and relatively simply to come by, and the real challenge is converting that awesome vision in your head into something we can see and touch and deliver to others; this is not quite so free and simple. While Brainstorm provided a great place to capture the ideas, and we had no shortage of them, the challenge was connecting brainstorm to the people who were happy and willing to perform the work, and it didn’t really serve this purpose very well.

There were two problems with this. Firstly, picking up other people’s popular ideas is not how Open Source traditionally works. Open Source is built on a philosophy of scratching your own itch, traditionally fueled by programmers fixing their annoyances and building features and applications they want. Now, this is not to say a non-programmer can’t rally the community around their idea and build momentum around an implementation, but doing this requires significantly more effort than a fire and forget submission into brainstorm. In other words, just because an idea is popular doesn’t necessarily mean it is interesting enough for a developer to want to implement it. Secondly, brainstorm started to garner an unrealistic social expectation that popular ideas would be automatically added to the TODO list of prominent Ubuntu developers, which was never the case.

Today at UDS we had a discussion about these deficiencies in brainstorm in traversing the chasm between idea and implementation and Randall Ross had an interesting idea. With brainstorm retired we should re-focus the brainstorm URL and provide some guidance for tips and tricks for how to take an idea and rally support around it to develop an implementation. As an example, over the years I have discovered that taking an idea and building a well formed spec with detailed UI mock-ups and architectural diagrams, a detailed blueprint, regular meetings, and burndown charts, all significantly help to taking ideas from fiction to fandom. Equipping our community with the skills and tools to bring these ideas to fruition is a better use of our time.

So, the TL;DR of all of this is…brainstorm was a great idea at the time, but it didn’t effectively drive the most popular ideas in our community to fruition and delivery in Ubuntu. We want to help provide guidance and best practice to help our community be more successful in converting their ideas into development plans and getting people interested in participating.

Read more

Hot on the heels of my last post showing Unity 8 running on Mir on a Macbook Pro Retina, there were some folks who were curious about how well Unity and Mir work on a phone.

Well, thanks to your friend and mine, Kevin Gunn, you can see a video of Unity 8 on Mir running on a Galaxy Nexus (which is by no means a super-powerful smartphone these days):

Can’t see the video? See it here!

Again, just to emphasize, this has not been through a round of performance optimizations, so you can expect additional performance improvements in the future, but I think this demonstrates that we are heading in the right direction.

If you are interested in participating in Mir development, click here and if you are interested in participating in Unity 8, click here.

Read more

Recently the Mir and Unity Next teams got Unity 8 up and running on Mir. Now, this work is still very early in development and neither Mir nor Unity Next are finished yet, but I reached out to Michael Zanetti, who is on the team, and asked him to put together a short video demo to show the progress of this work. This demo shows the phone/tablet part of the Unity 8 codebase; the final desktop version will come later.

Here is is:

Can’t see the video? Click here!

As you can see, impressive progress is being made; this demo is running on a MacBook Pro Retina utilizing the full resolution of 2880×1800 pixels and using Intel HD 4400 graphics. The performance is already looking great, and the team haven’t done a deep dive into performance optimization yet.

If you are interested in participating in Mir development, click here and if you are interested in participating in Unity 8, click here.

Read more

As a pretty simple-minded person, I am a big fan of simplicity. The world is filled with too much complexity and too much detail. Many often feel the detail is necessary for particular outcomes or to solve particular problems. The lesson I have learned as I have gotten older though is that while the skill is in matching the level of detail to the mind of the observer, the real elegance is in delivering the same level of detail but in a way that feels simpler than expected to the observer. This results in delightful experiences.

Ross Gardler recently quoted Einstein who said “everything should be made as simple as possible, but not simpler“. This so beautifully summarizes my view of the world; life should be as simple as we can make it, but we should not compromise in our goals merely to make things simple. In other words, if we can boil our projects, processes, interfaces, and ideas down into simpler parts that still let us be productive, they become more enjoyable to engage with and thus more successful. Of course, making complex things simple is…complex. It is though, worthwhile, and for many (myself included), a fun challenge. I am sure I am not alone.

As we step into our Ubuntu Developer Summit this week I would like to encourage everyone to think about ways in which we can simplify all aspects of how create and deliver Ubuntu to others as a means to further the project and experience. This doesn’t just apply to user interface design though. How do we make our teams easier to navigate and participate in? How do we make it easier to create your first app, charm, bug fix, translation, document, mailing list post, question, answer, or otherwise? If we can make in-roads this week in simplicity, I am confident it will continue the bold stride Ubuntu is making into the future of devices and the cloud.

Read more

Just a quick note to remind everyone that our next Ubuntu Developer Summit is taking place this week on Tuesday, Wednesday and Thursday, and is open and available to everyone to participate. This is the event where we get together to discuss, debate, and plan the next three months of work.

The event takes place online from 2pm – 8pm UTC. All sessions will run using a combination of Google+ streaming video hangouts and IRC, and you can see the full schedule on summit.ubuntu.com. Consequently, for those who cannot attend or might miss certain sessions, all sessions will be available pre-recorded from the session pages when the session is complete.

The event kicks off on Tuesday at 2pm with our keynote. We hope to see you there!

Read more

Last cycle we saw quite a number of changes to the way that planning works for Ubuntu. Some of these changes are causing issues that the current implementation of the Ubuntu Status Tracker is not easily able to handle. The main issue that I have noticed from helping people setup their work items and blueprints for tracking is that tracking needs to not be so closely dependent on the Ubuntu release cycles. This is causing issues in two ways. The first that I have seen is that a feature is planned to be released in stages essentially X number of cycles. It currently isn’t possible to track a single blueprint across different cycles, let alone multiple cycles. If you try to do this anyway by changing the cycle every 6 months, then the Status Tracker sends out what are essentially validation errors because as far as it is aware, any milestone that isn’t in the cycle that it is looking at is in valid (ubuntu-13.04 is a raring milestone and isn’t valid on a saucy blueprint).

In order to discuss these issues and hopefully come up with a solution, I have created a meeting for the virtual Ubuntu Developer Summit which starts tomorrow.

Read more

I blogged about the progress on our community website a while ago and we’re getting closer. A few community members helped on getting the content for the site ready. Here I’d like to take the time and thank all of them – they are all not the kind of person who end up in long arguments, but those who see that a task is important, ask what needs to get done and get right to it. A big kudos to all of you!

The first stage of the work is largely done. Michael Hall set up a wordpress test instance here where we put all the updated content, which is a great achievement already. It’s not only up to date, but also much more welcoming and friendly. The Canonical Web team should help us update the style to match the new ubuntu.com site.

What we need now is to get a few eyes over the test instance, so we can make sure all the content is accurate and makes sense. Any help is appreciated. Please just leave a comment on the blog post and we’ll take care of it.

Once we’re happy with the content, we will ask for the site to be put up in a more official place and then ask for redirects and links to be placed into all the right spots.

Thanks everyone. Let’s make the new community website happen together!

(There’s also a session at the next UDS about this. Make sure you attend if you want to get involved.)

Read more

When I bought Samsung ARM Chromebook few months ago I had no idea about UCM profiles and burnt speakers (left is dead, right is resting)…

This was good lesson. I learnt more about how UseCase Manager works, took profiles from ChromeOS and added them into Ubuntu so other users will be a bit more safe (due to lack of testers it took months to merge it into “precise” and “quantal” releases).

During last months I had discussions with some Debian, Ubuntu, Fedora developers about how to solve such problems and how to keep UCM profiles shared between distributions.

In meantime Liam Girdwood pointed me to (not used) UCM git tree at ALSA Project server. So finally I spent some time and sent Ubuntu ones for merging.

I also got newer profiles for OMAP4 devices and some updates for Chromebook ones.

The idea is to collect UCM profiles, keep them in one place and share in each distribution packages. So if your hardware has profiles created then join us to make users life easier.

---

All rights reserved © Marcin Juszkiewicz
Call for ALSA UCM profiles was originally posted on Marcin Juszkiewicz website

Read more

I'm writing a bunch of new code these days for Ubuntu Touch's Image Based Upgrade system.  Think of it essentially as Ubuntu Touch's version of upgrading the phone/tablet (affectionately called phablet) operating system in a bulk way rather than piecemeal apt-gets the way you do it on a traditional Ubuntu desktop or server.  One of the key differences is that a phone has to detour through a reboot in order to apply an upgrade since its Ubuntu root file system is mounted read-only during the user session.

Anyway, those details aren't the focus of this article.  Instead, just realize that because it's a pile of new code, and because we want to rid ourselves of Python 2, at least on the phablet image if not everywhere else in Ubuntu, I am prototyping all this in Python 3, and specifically 3.3.  This means that I can use all the latest and greatest cool stuff in the most recent stable Python release.  And man, is there a lot of cool stuff!

One module in particular that I'm especially fond of is contextlib.  Context managers are objects implementing the protocol behind the with statement, and they are typically used to guarantee that some resource is cleaned up properly, even in the event of error conditions.  When you see code like this:

with open(somefile) as fp:
    data = fp.read()

you are invoking a context manager.  Python was clever enough to make file objects support the context manager protocol so that you never have to explicitly close the file; that happens automatically when the with statement completes, regardless of whether the code inside the with statement succeeds or raises an exception.

It's also very easy to define your own context managers to properly handle other kinds of resources.  I won't go into too much detail here, because this is all well-established; the with statement has been, er, with us since Python 2.5.

You may be familiar with the contextlib module because of the @contextmanager decorator it provides.  This makes it trivial to define a new context manager without having to deal with all the intricacies of the protocol.  For example, here's how you would implement a context manager that temporarily changes the current working directory:

import os
from contextlib import contextmanager

@contextmanager
def chdir(dir):
    cwd = os.getcwd()
    try:
        os.chdir(dir)
        yield
    finally:
        os.chdir(cwd)

In this example, the yield cedes control back to the body of the with statement, and when that completes, the code after the yield is executed.  Because the yield is wrapped inside a try/finally, it is guaranteed that the original working directory is restored.  You would use this code like so:

with chdir('/tmp'):
    print(os.getcwd())

So far, so good, but this is nothing revolutionary.  Python 3.3 brings additional awesomeness to contextlib by way of the new ExitStack class.

The documentation for ExitStack is a bit dense, and even the examples didn't originally make it clear to me how amazing this new API is.  In my opinion, this is so powerful, it changes completely the way you think about deploying safe code.

So what is an ExitStack?  One way to think about it is as an extensible context manager.  It's used in with statements just like any other context manager:

from contextlib import ExitStack
with ExitStack() as stack:
    # do some magical stuff

Just like any other context manager, the ExitStack's "exit" code is guaranteed to be run at the end of the with statement.  It's the programmable extensibility of the ExitStack where the cool stuff happens.

The first interesting method of an ExitStack you might use is the callback() method.  Let's say for example that in your with statement, you are creating a temporary directory and you want to make sure that temporary directory gets deleted when the with statement exits.  You could do something like this:

import shutil, tempfile
with ExitStack() as stack:
    tempdir = tempfile.mkdtemp()
    stack.callback(shutil.rmtree, tempdir)

Now, when the with statement completes, it calls all of its callbacks, which includes removing the temporary directory.

So, what's the big deal?  Let's say you're actually creating three temporary directories and any of those calls could fail.  To guarantee that all successfully created directories are deleted at the end of the with statement, regardless of whether an exception occurred in the middle, you could do this:

with ExitStack() as stack:
    tempdirs = []
    for i in range(3):
        tempdir = tempfile.mkdtemp()
        stack.callback(shutil.rmtree, tempdir)
        tempdirs.append(tempdir)
    # Do something with the tempdirs

If you knew statically that you wanted three temporary directories, you could set this up with nested with statements, or a single with statement containing multiple backslash-separated targets, but that gets unwieldy very quickly.  And besides, that's impossible if you only know the number of directories you need dynamically at run time.  On the other hand, the ExitStack makes it easy to guarantee everything gets cleaned up and there are no leaks.

That's powerful enough, but it's not all you can do!  Another very useful method is enter_context().

Let's say that you are opening a bunch of files and you want the following behavior: if all of the files open successfully, you want to do something with them, but if any of them fail to open, you want to make sure that the ones that did get open are guaranteed to get closed.  Using ExitStack.enter_context() you can write code like this:

files = []
with ExitStack() as stack:
    for filename in filenames:
        # Open the file and automatically add its context manager to the stack.
        # enter_context() returns the passed in context manager, i.e. the 
        # file object.
        fp = stack.enter_context(open(filename))
        files.append(fp)
    # Capture the close method, but do not call it yet.
    close_all_files = stack.pop_all().close

(Note that the contextlib documentation contains a more efficient, but denser way of writing the same thing.)

So what's going on here?  First, the open(filename) does what it always does of course, it opens the file and returns a file object, which is also a context manager.  However, instead of using that file object in a with statement, we add it to the ExitStack by passing it to the enter_context() method.  For convenience, this method returns the passed in object.

So what happens if one of the open() calls fail before the loop completes?  The with statement will exit as normal and the ExitStack will exit all the context managers it knows about.  In other words, all the files that were successfully opened will get closed.  Thus, in an error condition, you will be left with no open files and no leaked file descriptors, etc.

What happens if the loop completes and all files got opened successfully?  Ah, that's where the next bit of goodness comes into play: the ExitStack's pop_all() method.

pop_all() creates a new ExitStack, and populates it from the original ExitStack, removing all the context managers from the original ExitStack.  So, after stack.pop_all() completes, the original ExitStack, i.e. the one used in the with statement, is now empty.  When the with statement exits, the original ExitStack contains no context managers so none of the files are closed.

Well, then, how do you close all the files once you're done with them?  That's the last bit of magic.  ExitStacks have a .close() method which unwinds all the registered context managers and callbacks and invokes their exit functionality.  So, after you're finally done with all the files and you want to clean everything up, you would just do:

close_all_files()

And that's it.

Hopefully that all makes sense.  I know it took a while to sink in for me, but now that it has, it's clear the enormous power this gives you.  You can write much safer code, in the sense that it's easier to ensure much better guarantees that your resources are cleaned up at the right time.

The real power comes when you have many different disparate resources to clean up for a particular operation.  For example, in the test suite for the Image Based Upgrader, I have a test where I need to create a temporary directory and start an HTTP server in a thread.  Roughly, my code looks like this:

@classmethod
def setUpClass(cls):
    cls._cleaner = ExitStack()
    try:
        cls._serverdir = tempfile.mkdtemp()
        cls._cleaner.callback(shutil.rmtree, cls._serverdir)
        # ...
        cls._stop = make_http_server(cls._serverdir)
        cls._cleaner.callback(cls._stop)
    except:
        cls._cleaner.pop_all().close()
        raise

@classmethod
def tearDownClass(cls):
    cls._cleaner.close()

Notice there's no with statement there at all. :)   This is because the resources must remain open until tearDownClass() is called, unless some exception occurs during the setUpClass().  If that happens, the bare except will ensure that all the context managers are properly closed, leaving the original ExitStack empty.  (The bare except is acceptable here because the exception is re-raised after the resources are cleaned up.)  Even though the exception will prevent the tearDownClass() from being called, it's still safe to do so in case it is called for some odd reason, because the original ExitStack is empty.

But if no exception occurs, the original ExitStack will contain all the context managers that need to be closed, and calling .close() on it in the tearDownClass() does exactly that.

I have one more example from my recent code.  Here, I need to create a GPG context (the details are unimportant), and then use that context to verify the detached signature of a file.  If the signature matches, then everything's good, but if not, then I want to raise an exception and throw away both the data file and the signature (i.e. .asc) file.  Here's the code:

with ExitStack() as stack:
    ctx = stack.enter_context(Context(pubkey_path))
    if not ctx.verify(asc_path, channels_path):
        # The signature did not verify, so arrange for the .json and .asc
        # files to be removed before we raise the exception.
        stack.callback(os.remove, channels_path)
        stack.callback(os.remove, asc_path)
        raise FileNotFoundError

Here we create the GPG context, which itself is a context manager, but instead of using it in a with statement, we add it to the ExitStack.  Then we verify the detached signature (asc_path) of a data file (channels_path), and only arrange to remove those files if the verification fails.  When the FileNotFoundError is raised, the ExitStack in the with statement unwinds, removing both files and closing the GPG context.  Of course, if the signature matches, only the GPG context is closed -- the channels_path and asc_path files are not removed.

You can see how an ExitStack actually functions as a fairly generic resource manager!

To me, this revolutionizes the management of external resources.  The new ExitStack object, and the methods and semantics it exposes, make it so much easier to manage those resources, guaranteeing that they get cleaned up at the right time, once and only once, regardless of whether errors occur or not.

ExitStack takes the already powerful concept of context managers and turns it up to 11.  There's more you can do, and it's worth spending some time reading the contextlib documentation in Python 3.3, especially the examples and recipes.

As I mentioned on Twitter, it's features like this that make using Python 2 seem downright barbaric.

Read more

Congrats to the Debian release team on the new release of Debian 7.0 (wheezy)!

Leading up to the release, a meme making the rounds on Planet Debian has been to play a #newinwheezy game, calling out some of the many new packages in 7.0 that may be interesting to users. While upstart as a package is nothing new in wheezy, the jump to upstart 1.6.1 from 0.6.6 is quite a substantial change. It does bring with it a new package, mountall, which by itself isn't terribly interesting because it just provides an upstart-ish replacement for some core scripts from the initscripts package (essentially, /etc/rcS.d/*mount*). Where things get interesting (and, typically, controversial) is the way in which mountall leverages plymouth to achieve this.

What is plymouth?

There is a great deal of misunderstanding around plymouth, a fact I was reminded of again while working to get a modern version of upstart into wheezy. When Ubuntu first started requiring plymouth as an essential component of the boot infrastructure, there was a lot of outrage from users, particularly from Ubuntu Server users, who believed this was an attempt to force pretty splash screen graphics down their throats. Nothing could be further from the truth.

Plymouth provides a splash screen, but that's not what plymouth is. What plymouth is, is a boot-time I/O multiplexer. And why, you ask, would upstart - or mountall, whose job is just to get the filesystem mounted at boot - need a boot-time I/O multiplexer?

Why use plymouth?

The simple answer is that, like everything else in a truly event-driven boot system, filesystem mounting is handled in parallel - with no defined order. If a filesystem is missing or fails an fsck, mountall may need to interact with the user to decide how to handle it. And if there's more than one missing or broken filesystem, and these are all being found in parallel, there needs to be a way to associate each answer from the user to the corresponding question from mountall, to avoid crossed signals... and lost data.

One possible way to handle this would be for mountall to serialize the fsck's / mounts. But this is a pretty unsatisfactory answer; all other things (that is, boot reliability) being equal, admins would prefer their systems to boot as fast as possible, so that they can get back to being useful to users. So we reject the idea of solving the problem of serializing prompts by making mountall serialize all its filesystem checks.

Another option would be to have mountall prompt directly on the console, doing its own serialization of the prompts (even though successful mounts / fscks continue to be run in parallel). This, too, is not desirable in the general case, both because some users actually would like to have pretty splash screens at boot time, and this would be incompatible with direct console prompting; and because mountall is not the only piece of software that need to prompt at boot time (see also: cryptsetup).

Plymouth: not just a pretty face

Enter plymouth, which provides the framework for serializing requests to the user while booting. It can provide a graphical boot splash, yes; ironically, even its own homepage suggests that this is its purpose. But it can also provide a text-only console interface, which is what you get automatically when booting without a splash boot argument, or even handle I/O over a serial console.

Which is why, contrary to the initial intuitions of the s390 porters upon seeing this package, plymouth is available for all of Debian's Linux architectures in wheezy, s390 and s390x included, providing a consistent architecture for boot-time I/O for systems that need it - which is any machine using a modern boot system, such as upstart or systemd.

Room for improvement

Now, having a coherent architecture for your boot I/O is one thing; having a bug-free splash screen is another. The experience of plymouth in Ubuntu has certainly not been bug-free, with plymouth making significant demands of the kernel video layer. Recently, the binary video driver packages in Ubuntu have started to blacklist the framebuffer kernel driver entirely due to stability concerns, making plymouth splash screens a non-starter for users of these drivers and regressing the boot experience.

One solution for this would be to have plymouth offload the video handling complexity to something more reliable and better tested. Plymouth does already have an X backend, but we don't use that in Ubuntu because even if we do have an X server, it normally starts much later than when we would want to display the splash screen. With Mir on the horizon for Ubuntu, however, and its clean separation between system and session compositors, it's possible that using a Mir backend - that can continue running even after the greeter has started, unlike the current situation where plymouth has to cede the console to the display manager when it starts - will become an appealing option.

This, too, is not without its downsides. Needing to load plymouth when using crypted root filesystems already makes for a bloated initramfs; adding a system compositor to the initramfs won't make it any better, and introduces further questions about how to hand off between initramfs and root fs. Keeping your system compositor running from the initramfs post-boot isn't really ideal, particularly for low-memory systems; whereas killing the system compositor and restarting it will make it harder to provide a flicker-free experience. But for all that, it does have its architectural appeal, as it lets us use plymouth as long as we need to after boot. As the concept of static runlevels becomes increasingly obsolete in the face of dynamic systems, we need to design for the world where the distinction between "booting" and "booted" doesn't mean what it once did.

Read more

Packages

I quite like the current status of Qt 5 in Debian and Ubuntu (the links are to the qtbase packages, there are ca. 15 other modules as well). Despite Qt 5 being bleeding edge and Ubuntu having had the need to use it before even the first stable release came out in December, the co-operation with Debian has gone well. Debian is now having the first Qt 5 uploads done to experimental and later on to unstable. My work contributed to pkg-kde git on the modules has been welcomed, and even though more work has been done there by others, there haven't been drastic changes that would cause too big transition problems on the Ubuntu side. It has of course helped to ask others what they want, like the whole usage of qtchooser. Now with Qt 5.0.2 I've been able to mostly re-sync all newer changes / fixes to my packaging from Debian to Ubuntu and vice versa.

There will remain some delta, as pkg-kde plans to ask for a complete transition to qtchooser so that all Qt using packages would declare the Qt version either by QT_SELECT environment variable (preferable) or a package dependency (qt5-default or qt4-default). As a temporary change related to that, Debian will have a debhelper modification that defaults QT_SELECT to qt4 for the duration of the transition. Meanwhile, Ubuntu already shipped the 13.04 release with Qt 5, and a shortcut was taken there instead to prevent any Qt 4 package breakage. However, after the transition period in Debian is over, that small delta can again be removed.

I will also need to continue pushing any useful packaging I do to Debian. I pushed qtimageformats and qtdoc last week, but I know I'm still behind with some "possibly interesting" git snapshot modules like qtsensors and qtpim.

Patches

More delta exists in the form of multiple patches related to the recent Ubuntu Touch efforts. I do not think they are of immediate interest to Debian – let's start packaging Qt 5 apps to Debian first. However, about all of those patches have already been upstreamed to be part of Qt 5.1 or Qt 5.2, or will be later on. Some already were for 5.0.2.

A couple of months ago Ubuntu did have some patches hanging around with no clear author information. This was a result of the heated preparation for the Ubuntu Touch launches, and the fact that patches flew (too) quickly in place into various PPA:s. I started hunting down the authors, and the situation turned out to be better than I thought. About half of the patches were already upstreamed, and work on properly upstreaming the other ones was swiftly started after my initial contact. Proper DEP3 fields do help understanding the overall situation. There are now 10 Canonical individuals in the upstream group of contributors, and in the last week's sprint it turned out more people will be joining them to upstream their future patches.

Nowadays about all the requests I get for including patches from developers are stuff that was already upstreamed, like the XEmbed support in qtbase. This is how it should be.

One big patch still being Ubuntu only is the Unity appmenu support. There was a temporary solution for 13.04 that forward-ported the Qt 4 way of doing it. This will be however removed from the first 13.10 ('saucy') upload, as it's not upstreamable (the old way of supporting Unity appmenus was deliberately dropped from Qt 5). A re-implementation via QPA plugin support is on its way, but it may be that the development version users will be without appmenu support for some duration. Another big patch is related to qtwebkit's device pixel ratio, which will need to be fixed. Apart from these two areas of work that need to be followed through, patches situation is quite nice as mentioned.

Conclusion

Free software will do world domination, and I'm happy to be part of it.

Read more

Good things have one ugly part in common — they have to end one day… For me that day will be 31st May 2013 when contract between Canonical and Linaro will end.

Those 3 years were great. I wrote a lot about it half year ago so those of you who are new – go to my previous “good bye Linaro” post before reading rest of this post.

Half year ago I was going to Canonical but got hold at Linaro for longer. Then I made a mistake by agreeing to postpone my move to Linaro instead of joining as soon as possible — my fault…

Last 6 months were full of interesting things. We went from just bootstrapped AArch64 port to fully working LAMP and SDK images built with OpenEmbedded. I integrated all Linaro layers into one repository and reorganized in a way that those who want only our toolchains can have them without using any of our changes. This move was greeted by lot of maintainers and users from OpenEmbedded community. Wherever new toolchain components were provided for tests I had them checked on first day to see how AArch64 situation got improved and provided fixes when they were needed.

Recent release of Yocto Project has several changes done by me and Riku Voipio integrated. OpenEmbedded project also made release and has even more our changes in it. Most of those were AArch64 related, some were software updates or fixes to low level stuff.

Linaro Enterprise Group has Owen Yamauchi from Facebook working on porting HipHopVM. He is using SDK created by OpenEmbedded to not worry about any build dependencies or missing libraries. With my work (and work from porters like Riku Voipio, Steve McIntyre, Yvan Roux and others) he got not only libraries but also tools he needed for his job.

Andy Johnson started OpenJDK porting — also with OpenEmbedded. Riku provided instructions which I merged into our ‘jenkins-setup’ scripts to make live easier for Andy.

Due to all that work I am often contacted by random people (not only from Linaro) wherever they have some AArch64 related questions. Sometimes even with ARMv4/EABI related like post from Nicolas Pitre a day after RMK wrote that FPU emulator has to be removed from the Linux kernel. I provided him instructions how to make such build and just to be sure that I did not made any mistakes I tried one on my machine. IIRC none of main distributions support EABI for ARMv4 (no thumb) processors.

But looks like all that has to end. Unless someone from Linaro member companies (or who knows, maybe even Linaro itself) wants to hire me. I am open for offers.

If I go outside of @linaro.org then I would like to stay around and check how things go — probably as ‘community member’ or how it is called.

And one more thing at the end. As usual when I end my work at one place I gather recommendations on LinkedIn. If you have few spare minutes and want to write something then it will be appreciated.

---

All rights reserved © Marcin Juszkiewicz
Looks like it is time for me to say good bye again was originally posted on Marcin Juszkiewicz website

Read more

Next Tuesday, May 14 starts the second Virtual Ubuntu Developer Summit. Based on feedback, this vUDS will be three days long. Don’t forget to register for the event. A list of currently approved blueprints is available on Launchpad. If you find that one is missing, you can create your own. Keep an eye out this week for scheduling to start.

Tracks…

• App Development: Alan Pope, David Planella & Michael Hall
• Community: Daniel Holbach, Nick Skaggs & Jono Bacon
• Client: Jason Warner & Sebastien Bacher
• Server & Cloud: Dave Walker & Antonio Rosales
• Foundations: Steve Langasek

Bugs…

One of the bugs that has long effected Summit has been that a blueprint had to have a specific status. This will finally no longer effect us! If a blueprint is marked anything other than Obsolete or Superseded it will now show up on the schedule as long as it is approved by a track lead. A huge thanks to Steve Kowalik and William Grant for getting this fixed!

Another issue that seemed to confuse people is having to register attendance in Launchpad in order to be able to use the features of Summit. This is no longer the case! You are now able to register as attending in Summit without any need to visit Launchpad (a Launchpad account is still required).

If you find that you have any issues with Summit or vUDS you can contact the track leads or you can contact me. If you find any bugs in Summit, please tell us about it!

See you next week!

Read more