With Ubuntu 12.04.2, the kernel team introduced the idea of the "hardware enablement kernel" (HWE), originally intended to support new hardware for bare metal server and desktop. In fact, the documentation indicates that HWE images are not suitable for Virtual or Cloud Computing environments.  The thought was that cloud and virtual environments provide stable hardware and that the newer kernel features would not be needed.

Time has proven this assumption painfully wrong. Take for example the need for drivers in virtual environments. Several of the Cloud providers that we have engaged with have requested the use of the HWE kernel by default. On GCE, the HWE kernels provide support for their NVME disks or multiqueue NIC support. Azure has benefited from having an updated HyperV driver stack resulting in better performance. When we engaged with VMware Air, the 12.04 kernel lacked the necessary drivers.

Perhaps more germane to our Cloud users is that containers are using kernel features. 12.04 users need to use the HWE kernel in order to make use of Docker. The new Ubuntu Fan project will be enabled for 14.04 via the HWE-V kernel for Ubuntu 14.04.3. If you use Ubuntu as your container host, you will likely consider using an HWE kernel.

And with that there has been a steady chorus of people requesting that we provide HWE image builds for AWS. The problem has never been the base builds; building the base bits is fairly easy. The hard part is that by adding base builds, each daily and release build goes form 96 images for AWS to 288 (needless to say that is quite a problem). Over the last few weeks -- largely in my spare time -- I've been working out what it would take to deliver HWE images for AWS.

I am happy to announce that as of today, we are now building HWE-U (3.16) and HWE-V (3.19) Ubuntu 14.04 images for AWS. To be clear, we are not making any behavioral changes to the standard Ubuntu 14.04 images. Unless users opt into using an HWE image on AWS they will continue to get the 3.13 kernel. However, for those who want newer kernels, they now have the choice.

For the time being, only amd64 and i386 builds are being published.. Over the next few weeks, we expect the HWE images to reach full feature parity including release promotions, and indexing. And I fully expect that the HWE-V version of 14.04 will include our recent Fan project once the SRU's complete.

Check them out at http://cloud-images.ubuntu.com/trusty/current/hwe-u and http://cloud-images.ubuntu.com/trusty/current/hwe-v .

As always, feedback is always welcome.

[UPDATE] The Image ID's have been updated with the latest builds which now include Docker 1.6.2, the latest LXD and of course the Ubuntu Fan driver. 

This week, Dustin Kirkland announced the Ubuntu Fan Project.  To steal from the description, "The Fan is not a software-defined network, and relies on neither distributed databases nor consensus protocols.  Rather, routes are calculated deterministically and traffic carries no additional overhead beyond routine IP tunneling.  Canonical engineers have already demonstrated The Fan operating at 5Gpbs between two Docker containers on separate hosts."

My team at Canonical is responsible for the production of these images. Once the official SRU's land, I anticipate that we will publish an official stream over at cloud-images.ubuntu.com. But until then, check back here for images and updates. As always, if you have feedback, please hop into #server on FreeNode or send email.

GCE Images

• daily-ubuntu-docker-lxd-1404-trusty-v20150620
• daily-ubuntu-docker-lxd-1504-vivid-v20150621

Amazon AWS Images

ami-7e94ac63

ami-8e93ab93

ami-f943c1e4

ami-e742c0fa

ami-543c9b54

ami-b4298eb4

ami-4ae2a73d

ami-48e7a23f

ami-fbd126bf

ami-6bd3242f

ami-63585c53

ami-875357b7

ami-7de69c47

ami-1de19b27

ami-aca4a0fe

ami-2a9b9f78

ami-95877efe

ami-e58b728e

ami-9a94ac87

ami-ae93abb3

ami-1340c20e

ami-0743c11a

ami-9c3c9b9c

ami-42379042

ami-a2e2a7d5

ami-e4e7a293

ami-4bd0270f

ami-1dd32459

ami-f9585cc9

ami-1dd32459

ami-5de69c67

ami-01e19b3b

ami-74a5a126

ami-c89b9f9a

ami-29f90042

ami-8d8a73e6

I am pleased to announce initial Vagrant images [1, 2]. These images are bit-for-bit the same as the KVM images, but have a Cloud-init configuration that allows Snappy to work within the Vagrant workflow.

Vagrant enables a cross platform developer experience on MacOS, Windows or Linux [3].

Note: due to the way that Snappy works, shared file systems within Vagrant is not possible at this time. We are working on getting the shared file system support enabled, but it will take us a little bit to get going.

If you want to use Vagrant packaged in the Ubuntu archives, in a terminal run::

• sudo apt-get -y install vagrant
• cd <WORKSPACE>
  
• vagrant init http://goo.gl/DO7a9W 
• vagrant up
• vagrant ssh
  

• vagrant init ubuntu/ubuntu-15.04-snappy-core-edge-amd64
• vagrant up
• vagrant ssh

Back when we announced that the Ubuntu 14.04 LTS Cloud Images on Azure were using the Hardware Enablement Kernel (HWE), the immediate feedback was "what about 12.04?"

One of the perennial problems in the Cloud is knowing what is the most current image and where to find it. Some Clouds provide a nice GUI console, an API, or some combination. But what has been missing is a "dashboard" showing Ubuntu across multiple Clouds.

https://cloud-images.ubuntu.com/locator

In that light, I am please to announce that we have a new beta Cloud Image Finder. This page shows where official Ubuntu images are available. As with all betas, we have some kinks to work out, like gathering up links for our Cloud Partners (so clicking an Image ID launches an image). I envision that in the future this locator page will be the default landing page for our Cloud Image Page..

A few years ago when our fine friends on the kernel team introduced the idea of the "hardware enablement" (HWE) kernel, those of us in the Cloud world looked at it as curiosity. We thought that by in large, the HWE kernel would not be needed or wanted for Virtual Cloud instances.

And we were wrong.

So wrong in fact, that the HWE kernel has found its way into the Vagrant Cloud Images, VMware's vCHS, and Google's Compute engine as the default kernel for the Certified Images. The main reason for these requests is that virtual hardware moves at a fairly quick pace. Unlike traditional hardware, Virtual Hardware can be fixed and patched at the speed that software can be deployed.

The feedback in regards to Azure has been the same: users and Microsoft has asked for the HWE kernel consistently. Microsoft has validated that the HWE kernel (3.16) running Ubuntu 14.04 on Windows Azures passes their validation testing. In our testing, we have validated that the 3.16 kernel works quite well in Azure.

For Azure users, using the 3.16 HWE kernel brings SMB 2.1 copy file support and updates LIS drivers.

Therefore, starting with the latest Windows Azure image [1], all the Ubuntu 14.04 images will track the latest hardware enablement kernel. That means that all the goodness in Ubuntu 14.10's kernel will be the default for 14.04 users launching our official images on Windows Azure.

If you want to install the LTS kernel on your existing instance(s), simply run:

• sudo apt-get update
• sudo apt-get install linux-image-virtual-lts-utopic linux-lts-utopic-cloud-tools-common walinuxagent
• sudo reboot

Snappy Launches

OVA to the rescue

• VMware products
• ESXi
• Desktop
• Fusion
• VSphere
• Virtualbox
• Citrix XenServer
• Microsoft SCVMM
• Red Hat Enterprise Virtualization
• SuSE Studio
• Oracle VM

Okay, so where can I get the OVA images?

We are pleased to announce that Ubuntu 12.04 LTS, 14.04 LTS, and 14.10 are now in beta on Google Compute Engine [1, 2, 3].

These images support both the traditional user-data as well the Google Compute Engine startup scripts. We have included the Google Cloud SDK, pre-installed as well. Users coming from other Clouds can expect to have the same great experience as on other clouds, while enjoying the features of Google Compute Engine.

From an engineering perspective, a lot of us are excited to see this launch. While we don't expect too many rough edges, it is a beta, so feedback is welcome. Please file bugs or join us in #ubuntu-server on Freenode to report any issues (ping me, utlemming, rcj or Odd_Bloke).

Finally, I wanted to thank those that have helped on this project. Launching a cloud is not an easy engineering task. You have have build infrastructure to support the new cloud, create tooling to build and publish, write QA stacks, and do packaging work. All of this spans multiple teams and disciplines. The support from Google and Canonical's Foundations and Kernel teams have been instrumental in this launch, as well the engineers on the Certified Public Cloud team.

Getting the Google Cloud SDK:

• echo "deb http://archive.canonical.com/ubuntu $(lsb_release -c -s) partner" | sudo tee /etc/apt/sources.list.d/partner.list
• sudo apt-get update
• sudo apt-get -y install google-cloud-sdk

Cloud Images and Bash Vulnerabilities

Cloud image update timeline

For years, the Ubuntu Cloud Images have been built on a timer (i.e. cronjob or Jenkins). Every week, you can reasonably expect that stable and LTS releases to be built twice a week while our development build is build once a day.  Each of these builds is given a serial in the form of YYYYMMDD. 

While time-based building has proven to be reliable, different build serials may be functionally the same, just put together at a different point in time. Many of the builds that we do for stable and LTS releases are pointless.

When the whole heartbleed fiasco hit, it put the Cloud Image team into over-drive, since it required manually triggering builds the LTS releases. When we manually trigger builds, it takes roughly 12-16 hours to build, QA, test and release new Cloud Images. Sure, most of this is automated, but the process had to be manually started by a human. This got me thinking: there has to be a better way.

What if we build the Cloud Images when the package set changes?

With that, I changed the Ubuntu 14.10 (Utopic Unicorn) build process from time-based to archive trigger-based. Now, instead of building every day at 00:30 UTC, the build starts when the archive has been updated and the packages in the prior cloud image build is older than the archive version. In the last three days, there were eight builds for Utopic. For a development version of Ubuntu, this just means that developers don't have to wait 24 hours for the latest package changes to land in a Cloud Image.

Over the next few weeks, I will be moving the 10.04 LTS, 12.04 LTS and 14.04 LTS build processes from time to archive trigger-based. While this might result less frequent daily builds, the main advantage is that the daily builds will contain the latest package sets. And if you are trying to respond to the latest CVE, or waiting on a bug fix to land, it likely means that you'll have a fresh daily that you can use the following day.

Many of our Cloud Image users have inquired about the availability of updated Ubuntu Cloud Images in response to the Heartbleed OpenSSL Vulnerability [1]. Ubuntu released update Ubuntu packages for OpenSSL 08 April 2014 [2]. Due to the exceptional circumstances and severity of the Heartbleed OpenSSL bug, Canonical has released new 12.04.4 LTS, 12.10 and 13.10 Cloud Images at [3].

Canonical is working with Amazon to get the Quickstart and the AWS Marketplace links updated. In the meantime, you can find new AMI ID's at [3] and [4]. Also, the snapshot's for Amazon have the volume-create permission granted on the latest images.

Windows Azure [5], Joyent [6] and HP [7, 8, 9] all have updated Cloud Images in their respective galleries.

If you are running an affected version of OpenSSL on 12.04 LTS, 12.10 or 13.10, you are strongly encouraged to update. For new instances, it is recommended to either use an image with a serial newer than 20140408, or update your OpenSSL  package immediately upon launch. Finally, if you need documentation on enabling unattended upgrades, please see [10].


[1] https://www.openssl.org/news/secadv_20140407.txt
[2] http://www.ubuntu.com/usn/usn-2165-1/
[3] 12.04.4 LTS: http://cloud-images.ubuntu.com/releases/precise/release-20140408/
     12.10: http://cloud-images.ubuntu.com/releases/quantal/release-20140409/
     13.10: http://cloud-images.ubuntu.com/releases/saucy/release-20140409.1/
[4] http://cloud-images.ubuntu.com/locator/ec2/
[5] Azure: Ubuntu-12_04_4-LTS-amd64-server-20140408-en-us-30GB
                 Ubuntu-12_10-amd64-server-20140409-en-us-30GB
                 Ubuntu-13_10-amd64-server-20140409.1-en-us-30GB
[6] Joyent Images:
        "ubuntu-certified-12.04", fe5aa6c0-0f09-4b1f-9bad-83e453bb74f3
        "ubuntu-certified-13.10", 049dfe64-6c37-4b88-8e89-4b8aa0f129f2
[7] HP US-West-1:
          12.04.4: 27be722e-d2d0-44f0-bebe-471c4af76039
          12.10: 065bb450-e5d0-4348-997d-e4d9e359b8fb
          13.10: 9d7d22d0-7d43-481f-a7eb-d93ea2791409
[8] HP US-East-1:
          12.04.4 8672f4c6-e33d-46f5-b6d8-ebbeba12fa02
          12.10: cbb44038-2602-48d5-b609-e05f4b61be9a
          13.10: 00398423-7429-4064-b781-fa0af00449c8
[9] Waiting on HP for replication to legacy regions az-{1,2,3}
[10] https://help.ubuntu.com/community/AutomaticSecurityUpdates

Ubuntu Server 11.04 has proven to be a venerable server platform, it has nonetheless, reached its end of life on October 28, 2012. Whether or not you are blissfully unaware, end-of-life means that updates and security patches have been discontinued.

As part of the EOL'ing of a release, the mirrors are retired over time. Last week the mirrors for 11.04 were retired from archive.ubuntu.com, which in turn propagated through to the S3 EC2 mirrors. Any person using Ubuntu 11.04 and the S3 mirrors or archive.ubuntu.com will be unable to install software.

Over the last week, the Cloud Image team has fielded several questions from distraught users caused by the continued use of 11.04. We strongly suggest that those running Ubuntu Server 11.04 and the recently expired 11.10 and 8.04 LTS upgrade to a supported release to prevent any disruptions to their infrastructure. The current supported LTS is 12.04 with 13.04 being the latest stable release. Ubuntu 10.04 LTS is supported until April of 2015.

Those who continue to run expired Ubuntu releases may experience issues and may be required to mitigate the movement of mirrors from the S3 and main archive servers to old-releases.ubuntu.com/ubuntu [2]. While Ubuntu 8.04 LTS and 11.10 are currently available at the main archive and S3 locations, they will be removed anytime, per policy [3]

The Symptom

Err http://security.ubuntu.com natty-security/main Sources
  404  Not Found [IP: 91.189.92.201 80]
Err http://security.ubuntu.com natty-security/universe Sources
  404  Not Found [IP: 91.189.92.201 80]
Err http://security.ubuntu.com natty-security/main amd64 Packages
  404  Not Found [IP: 91.189.92.201 80]
Err http://security.ubuntu.com natty-security/universe amd64 Packages
  404  Not Found [IP: 91.189.92.201 80]
Ign http://security.ubuntu.com natty-security/main Translation-en_US

Err http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ natty-updates/main libapr1 amd64 1.4.2-7ubuntu2.1
  403  Forbidden
Err http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ natty/main libaprutil1 amd64 1.3.9+dfsg-5ubuntu3
  403  Forbidden

Fix Option 1:

sudo apt-get -y update

dpkg-divert --local --rename --add /etc/cloud/templates/sources.list.tmpl

sed -i  's,http://.*ubuntu.com,http://old-releases.ubuntu.com,g' \

     /etc/cloud/templates/sources.list.tmpl.distrib \

    | sudo tee /etc/cloud/templates/sources.list.tmpl

#cloud-config

bootcmd: 

- sed -i 's,\$mirror,http://old-releases.ubuntu.com/ubuntu,g' -e 's,http://security.ubuntu.com.,http://old-releases.ubuntu.com,g' /etc/cloud/templates/sources.list.tmpl.distrib > /etc/cloud/templates/sources.list.tmpl

Over the course of working with Microsoft on Windows Azure, we have had the goal of bringing the same experience on Azure as our users have on EC2. As part of our QA process, we publish daily images (http://cloud-images.ubuntu.com) for EC2 and OpenStack users.

Today, I am pleased to announce that Windows Azure Cloud Image dailies are now being published for Ubuntu Server 12.04 LTS, 12.10, 13.04 and the current development version 13.10. Due to the way that Windows Azure image publication works, these images will appear with in a three or four hours of the EC2 images and will be published to all Windows Azure regions.

However, the daily images will not be available in the Windows Azure Gallery; these images are published to API users. In the coming weeks, we'll throw up some pages to help our API users find the current images, but for now, you can use the API Query tools to find the images.

The initial daily images are:

• Ubuntu_DAILY_BUILD-precise-12_04_2-LTS-amd64-server-20130502-en-us-30GB
• Ubuntu_DAILY_BUILD-raring-13_04-amd64-server-20130501-en-us-30GB
• Ubuntu_DAILY_BUILD-saucy-13_10-amd64-server-20130502-en-us-30GB  

We are pleased to announce that Canonical has stood up official mirrors in HP Cloud's AZ-1, 2, and 3 regions.

If you are using Ubuntu Server 12.10 Cloud Images, there is no action to take; 12.10 images are by default configured to use the new mirror address.

For Ubuntu 12.04 instances, the default Ubuntu image does not automatically use the in-HP Cloud mirrors. We are currently working with HP to publish a new image that defaults to the local mirrors. If you would like to switch to the new in-HP mirrors, simply run:
          
    $ sudo sed -i -e \
            's,^archive.ubuntu.com/ubuntu,nova.clouds.archive.ubuntu.com/ubuntu,g'  \
             /etc/apt/sources.list 

    $ sudo apt-get -y update

Note: *.clouds.archive.ubuntu.com is configured using split-horizon DNS. This means that the DNS answer to queries is based on the askering IP address; only queries originating within HP Cloud are answered with the HP Cloud mirror addresses. If your DNS resolver[s] is not based in HP Cloud, then you will be unable to benefit from these new mirrors. 
 

Shortly after introducing the Vagrant images, a number of users provided very valuable feedback. The general gist was "sure this is a nice, but useless." For Cloud Images, we definitely take our user feedback to heart. 

Earlier we announced[1] that Canonical had worked this cycle to enable more frequent releases to the Ubuntu Cloud Images stable and long term releases. As of today, we are pleased to announce that Ubuntu Server 10.04 LTS, 11.10, 12.04 LTS and 12.10 are now fully enabled to follow the kernel SRU schedule with automated update releases. This means that within 24 hours of most SRU kernel releases, a new Ubuntu Cloud Image will be published.

Please note: with this change, the release notes have been moved the http://cloud-images.ubuntu.com/releases website. You can find them under <SUITE>/release/unpacked/release-notes.txt. Effective today, all emails announcing these new updates are discontinued. 

However, at this time, 12.04 LTS and 12.10 Cloud Images are not yet being promoted automatically to Windows Azure. We expect that as Windows Azure moves closer to General Availability (i.e. moves out of preview status) that automatic promotion will be enabled.

Please use either Cloud-Images[2], the AMI Finder[3], the RSS feed[4], or "ubuntu-cloudimg-query" from the Cloud-Utils packages to find the latest released images.

[1] http://blog.utlemming.org/2013/01/ubuntu-cloud-images-automated-release.html
     https://lists.ubuntu.com/archives/ubuntu-cloud-announce/2013-January/000045.html
     https://lists.ubuntu.com/archives/ubuntu-cloud/2013-January/000879.html
     https://groups.google.com/forum/?fromgroups=#!topic/ec2ubuntu/Mg-qpfguE10
[2] http://cloud-images.ubuntu.com/releases
[3] http://cloud-images.ubuntu.com/locator/ec2/
[4] http://cloud-images.ubuntu.com/rss/

Traditionally, updates for the stable release and long term stable release Cloud Images have been on an ad-hoc basis; reasons for releasing new images were generally restricted to security, critical bugs, and stale images. This ad-hoc update cycle meant that updated images were only released every three months or so, and for older releases, as often as six months.

While quality has always been a concern and top priority, during this cycle, Canonical has worked to vastly improve the QA infrastructure to support our Cloud Images. For example, when a new kernel is released, the daily build for that image is now put through the complete QA process. This change in process has allowed us to identify and automatically evaluate whether or not an image is a good candidate for update release.


As such, we are pleased to announce in the next few weeks, we will be turning on automated updates for Ubuntu Server 10.04 LTS, 11.10, 12.04 LTS, and 12.10. This means that approximately every three to four weeks, a new, freshened image will be released. The release cadence will follow the kernel SRU process.

The first updated image to be released under this process was 10.04 LTS[1].

There are a variety of ways to find the released Cloud Images. The two easiest ways are to go the AMI Finder[2] or use http://cloud-images.ubuntu.com/releases/<SUITE>/release. For example, http://cloud-images.ubuntu.com/releases/lucid/release would bring you to the last AMI's for Ubuntu Server 10.04 LTS.

Due to this change, we will discontinuing the email notifications of updated images to the various email lists for updated images. At UDS-R in Copenhagen[3], we discussed email notifications and the decision was reached to discontinue them. Replacing email notification is the RSS feed[4] and release notes (example from 10.04 LTS)[5].

As Cloud Image suites are migrated to automated releases, we will follow up on this announcement.

Finally, for 12.04 LTS and later, this change will introduce lock-step update releases with Windows Azure. As Windows Azure moves towards GA, we have been working to have the same releases for the Ubuntu Server Cloud Images on both EC2 and Windows Azure.

As always, your feedback is most appreciated. Please feel free to follow on either this post or to email concerns direct to me.

[1] http://cloud-images.ubuntu.com/releases/lucid/release-20130124/
[2] http://cloud-images.ubuntu.com/locator/ec2/
[3] http://blueprints.launchpad.net/ubuntu/+spec/servercloud-r-cloudtesting
[4] http://cloud-images.ubuntu.com/rss/
[5] http://cloud-images.ubuntu.com/releases/lucid/release-20130124/unpacked/release_notes.txt

We are pleased to announce the availability of beta Vagrant Cloud Images. These images have been customized to work with the Vagrant development environment, and are based on the Ubuntu Cloud Images. As such, these are vanilla images. They do, however, have the Virtualboxguest additions found in the Universe archive (required for Vagrant integration). 

For those who use Vagrant, your feedback is essential. Please feel free to send feedback via the ubuntu-cloud@lists.ubuntu.com mailing list.

The images are approximately 256M in size, and are configured for 512MB of RAM. They use a custom cloud-init user-data to drive the first boot. And of course, they have the vagrant user with vagrant insecure SSH key pre-installed. During the beta period, we will not be promoting any of the Vagrant boxes with the regular releases of the Ubuntu Cloud Images and will only publish the daily image builds; after the beta period these images will be promoted with the releases.

To kick the tires on the Vagrant images, take a look at http://cloud-images.ubuntu.com/vagrant. I will be working with the fine folks at Vagrant to get the official Ubuntu Vagrant images listed at  http://www.vagrantbox.es/

If you are interested in learning about the Vagrant development environment, head on over to Vagrant for more information.

For sometime people have been asking me "when will Cloud Images sport a Twitter account?" Well, wait no longer, because the Ubuntu Cloud Image Builder now has a Twitter Account.

The Cloud Image process will now Tweet when a new image is build and published -- dailies, new release updates and new versions being releases. For right we're only Tweeting EC2 information, but once Windows Azure goes GA, we'll start Tweeting that too.

So in the meantime, you can follow our faithful Cloud Builder as it tweets merrily its build progress at @UbuCloudImages. But I'll have to warn you, the Cloud Builder won't response to tweets, so we're not snubbing you if there is no response.

If Twitter isn't your cup of tea, or coffee, or <insert liquid refreshment here>, how about RSS. A while back in November, we introduced RSS feeds for the Ubuntu Cloud Images.

There is a feed for both released images and dailies. The feeds are really simple: they show all the builds that are available. This is a great way to track new releases of the Ubuntu Cloud Images if you don't want want to follow Twitter, hate checking your email or you don't care much for reading our announcement emails.

Anyway, enjoy the RSS.