Canonical Voices

Posts tagged with 'planet ubuntu'

Stéphane Graber

This is the second blog post in this series about LXD 2.0.

LXD logo

Where to get LXD and how to install it

There are many ways to get the latest and greatest LXD. We recommend you use LXD with the latest LXC and Linux kernel to benefit from all its features but we try to degrade gracefully where possible to support older Linux distributions.

The Ubuntu archive

All new releases of LXD get uploaded to the Ubuntu development release within a few minutes of the upstream release. That package is then used to seed all the other source of packages for Ubuntu users.

If you are using the Ubuntu development release (16.04), you can simply do:

sudo apt install lxd

If you are running Ubuntu 14.04, we have backport packages available for you with:

sudo apt -t trusty-backports install lxd

The Ubuntu Core store

Users of Ubuntu Core on the stable release can install LXD with:

sudo snappy install lxd.stgraber

The official Ubuntu PPA

Users of other Ubuntu releases such as Ubuntu 15.10 can find LXD packages in the following PPA (Personal Package Archive):

sudo apt-add-repository ppa:ubuntu-lxc/stable
sudo apt update
sudo apt dist-upgrade
sudo apt install lxd

The Gentoo archive

Gentoo has pretty recent LXD packages available too, you can install those with:

sudo emerge --ask lxd

From source

Building LXD from source isn’t very difficult if you are used to building Go projects. Note however that you will need the LXC development headers. In order to run LXD, your distribution also needs a recent Linux kernel (3.13 at least), recent LXC (1.1.5 or higher), LXCFS and a version of shadow that supports user sub-uid/gid allocations.

The latest instructions on building LXD from source can be found in the upstream README.

Networking on Ubuntu

The Ubuntu packages provide you with a “lxdbr0” bridge as a convenience. This bridge comes unconfigured by default, offering only IPv6 link-local connectivity through an HTTP proxy.

To reconfigure the bridge and add some IPv4 or IPv6 subnet to it, you can run:

sudo dpkg-reconfigure -p medium lxd

Or go through the whole LXD step by step setup (see below) with:

sudo lxd init

Storage backends

LXD supports a number of storage backends. It’s best to know what backend you want to use prior to starting to use LXD as we do not support moving existing containers or images between backends.

A feature comparison table of the different backends can be found here.

ZFS

Our recommendation is ZFS as it supports all the features LXD needs to offer the fastest and most reliable container experience. This includes per-container disk quotas, immediate snapshot/restore, optimized migration (send/receive) and instant container creation from an image. It is also considered more mature than btrfs.

To use ZFS with LXD, you first need ZFS on your system.

If using Ubuntu 16.04, simply install it with:

sudo apt install zfsutils-linux

On Ubuntu 15.10, you can install it with:

sudo apt install zfsutils-linux zfs-dkms

And on older releases, you can use the zfsonlinux PPA:

sudo apt-add-repository ppa:zfs-native/stable
sudo apt update
sudo apt install ubuntu-zfs

To configure LXD to use it, simply run:

sudo lxd init

This will ask you a few questions about what kind of zfs configuration you’d like for your LXD and then configure it for you.

btrfs

If ZFS isn’t available, then btrfs offers the same level of integration with the exception that it doesn’t properly report disk usage inside the container (quotas do apply though).
btrfs also has the nice property that it can nest properly which ZFS doesn’t yet. That is, if you plan on using LXD inside LXD, btrfs is worth considering.

LXD doesn’t need any configuration to use btrfs, you just need to make sure that /var/lib/lxd is stored on a btrfs filesystem and LXD will automatically make use of it for you.

LVM

If ZFS and btrfs aren’t an option for you, you can still get some of their benefits by using LVM instead. LXD uses LVM with thin provisioning, creating an LV for each image and container and using LVM snapshots as needed.

To configure LXD to use LVM, create a LVM VG and run:

lxc config set storage.lvm_vg_name "THE-NAME-OF-YOUR-VG"

By default LXD uses ext4 as the filesystem for all the LVs. You can change that to XFS if you’d like:

lxc config set storage.lvm_fstype xfs

Simple directory

If none of the above are an option for you, LXD will still work but without any of those advanced features. It will simply create a directory for each container, unpack the image tarballs for each container creation and do a full filesystem copy on container copy or snapshot.

All features are supported except for disk quotas, but this is very wasteful of disk space and also very slow. If you have no other choice, it will work, but you should really consider one of the alternatives above.

 

More daemon configuration

The complete list of configuration options for the LXD daemon can be found here.

Network configuration

By default LXD doesn’t listen to the network. The only way to talk to it is over a local unix socket at /var/lib/lxd/unix.socket.

To have it listen to the network, there are two useful keys to set:

lxc config set core.https_address [::]
lxc config set core.trust_password some-secret-string

The first instructs LXD to bind the “::” IPv6 address, namely, all addresses on the machine. You can obviously replace this by a specific IPv4 or IPv6 address and can append the TCP port you’d like it to bind (defaults to 8443).

The second sets a password which is used for remote clients to add themselves to the LXD certificate trust store. When adding the LXD host, they will be prompted for the password, if the password matches, the LXD daemon will store their client certificate and they’ll be trusted, never needing the password again (it can be changed or unset entirely at that point).

You can also choose not to set a password and instead manually trust each new client by having them give you their “client.crt” file (from ~/.config/lxc) and add it to the trust store yourself with:

lxc config trust add client.crt

Proxy configuration

In most setups, you’ll want the LXD daemon to fetch images from remote servers.

If you are in an environment where you must go through a HTTP(s) proxy to reach the outside world, you’ll want to set a few configuration keys or alternatively make sure that the standard PROXY environment variables are set in the daemon’s environment.

lxc config set core.proxy_http http://squid01.internal:3128
lxc config set core.proxy_https http://squid01.internal:3128
lxc config set core.proxy_ignore_hosts image-server.local

With those, all transfers initiated by LXD will use the squid01.internal HTTP proxy, except for traffic to the server at image-server.local

Image management

LXD does dynamic image caching. When instructed to create a container from a remote image, it will download that image into its image store, mark it as cached and record its origin. After a number of days without seeing any use (10 by default), the image is automatically removed. Every few hours (6 by default), LXD also goes looking for a newer version of the image and updates its local copy.

All of that can be configured through the following configuration options:

lxc config set images.remote_cache_expiry 5
lxc config set images.auto_update_interval 24
lxc config set images.auto_update_cached false

Here we are instructing LXD to override all of those defaults and instead cache images for up to 5 days since they were last used, look for image updates every 24 hours and only update images which were directly marked as such (–auto-update flag in lxc image copy) but not the images which were automatically cached by LXD.

Conclusion

At this point you should have a working version of the latest LXD release, you can now start playing with it on your own or wait for the next blog post where we’ll create our first container and play with the LXD command line tool.

Extra information

The main LXD website is at: https://linuxcontainers.org/lxd
Development happens on Github at: https://github.com/lxc/lxd
Mailing-list support happens on: https://lists.linuxcontainers.org
IRC support happens in: #lxcontainers on irc.freenode.net

And if you can’t wait until the next few posts to try LXD, you can take our guided tour online and try it for free right from your web browser!

Read more
Stéphane Graber

This is the first blog post in this series about LXD 2.0.

LXD logo

A few common questions about LXD

What’s LXD?

At its simplest, LXD is a daemon which provides a REST API to drive LXC containers.

Its main goal is to provide a user experience that’s similar to that of virtual machines but using Linux containers rather than hardware virtualization.

 

How does LXD relate to Docker/Rkt?

This is by far the question we get the most, so lets address it immediately!

LXD focuses on system containers, also called infrastructure containers. That is, a LXD container runs a full Linux system, exactly as it would be when run on metal or in a VM.

Those containers will typically be long running and based on a clean distribution image. Traditional configuration management tools and deployment tools can be used with LXD containers exactly as you would use them for a VM, cloud instance or physical machine.

In contrast, Docker focuses on ephemeral, stateless, minimal containers that won’t typically get upgraded or re-configured but instead just be replaced entirely. That makes Docker and similar projects much closer to a software distribution mechanism than a machine management tool.

The two models aren’t mutually exclusive either. You can absolutely use LXD to provide full Linux systems to your users who can then install Docker inside their LXD container to run the software they want.

Why LXD?

We’ve been working on LXC for a number of years now. LXC is great at what it does, that is, it provides a very good set of low-level tools and a library to create and manage containers.

However that kind of low-level tools aren’t necessarily very user friendly. They require a lot of initial knowledge to understand what they do and how they work. Keeping backward compatibility with older containers and deployment methods has also prevented LXC from using some security features by default, leading to more manual configuration for users.

We see LXD as the opportunity to address those shortcomings. On top of being a long running daemon which lets us address a lot of the LXC limitations like dynamic resource restrictions, container migration and efficient live migration, it also gave us the opportunity to come up with a new default experience, that’s safe by default and much more user focused.

The main LXD components

There are a number of main components that make LXD, those are typically visible in the LXD directory structure, in its command line client and in the API structure itself.

Containers

Containers in LXD are made of:

  • A filesystem (rootfs)
  • A list of configuration options, including resource limits, environment, security options and more
  • A bunch of devices like disks, character/block unix devices and network interfaces
  • A set of profiles the container inherits configuration from (see below)
  • Some properties (container architecture, ephemeral or persistent and the name)
  • Some runtime state (when using CRIU for checkpoint/restore)

Snapshots

Container snapshots are identical to containers except for the fact that they are immutable, they can be renamed, destroyed or restored but cannot be modified in any way.

It is worth noting that because we allow storing the container runtime state, this effectively gives us the concept of “stateful” snapshots. That is, the ability to rollback the container including its cpu and memory state at the time of the snapshot.

Images

LXD is image based, all LXD containers come from an image. Images are typically clean Linux distribution images similar to what you would use for a virtual machine or cloud instance.

It is possible to “publish” a container, making an image from it which can then be used by the local or remote LXD hosts.

Images are uniquely identified by their sha256 hash and can be referenced by using their full or partial hash. Because typing long hashes isn’t particularly user friendly, images can also have any number of properties applied to them, allowing for an easy search through the image store. Aliases can also be set as a one to one mapping between a unique user friendly string and an image hash.

LXD comes pre-configured with three remote image servers (see remotes below):

  • “ubuntu:” provides stable Ubuntu images
  • “ubunt-daily:” provides daily builds of Ubuntu
  • “images:” is a community run image server providing images for a number of other Linux distributions using the upstream LXC templates

Remote images are automatically cached by the LXD daemon and kept for a number of days (10 by default) since they were last used before getting expired.

Additionally LXD also automatically updates remote images (unless told otherwise) so that the freshest version of the image is always available locally.

Profiles

Profiles are a way to define container configuration and container devices in one place and then have it apply to any number of containers.

A container can have multiple profiles applied to it. When building the final container configuration (known as expanded configuration), the profiles will be applied in the order they were defined in, overriding each other when the same configuration key or device is found. Then the local container configuration is applied on top of that, overriding anything that came from a profile.

LXD ships with two pre-configured profiles:

  • “default” is automatically applied to all containers unless an alternative list of profiles is provided by the user. This profile currently does just one thing, define a “eth0” network device for the container.
  • “docker” is a profile you can apply to a container which you want to allow to run Docker containers. It requests LXD load some required kernel modules, turns on container nesting and sets up a few device entries.

Remotes

As I mentioned earlier, LXD is a networked daemon. The command line client that comes with it can therefore talk to multiple remote LXD servers as well as image servers.

By default, our command line client comes with the following remotes defined

  • local: (default remote, talks to the local LXD daemon over a unix socket)
  • ubuntu: (Ubuntu image server providing stable builds)
  • ubuntu-daily: (Ubuntu image server providing daily builds)
  • images: (images.linuxcontainers.org image server)

Any combination of those remotes can be used with the command line client.

You can also add any number of remote LXD hosts that were configured to listen to the network. Either anonymously if they are a public image server or after going through authentication when managing remote containers.

It’s that remote mechanism that makes it possible to interact with remote image servers as well as copy or move containers between hosts.

Security

One aspect that was core to our design of LXD was to make it as safe as possible while allowing modern Linux distributions to run inside it unmodified.

The main security features used by LXD through its use of the LXC library are:

  • Kernel namespaces. Especially the user namespace as a way to keep everything the container does separate from the rest of the system. LXD uses the user namespace by default (contrary to LXC) and allows for the user to turn it off on a per-container basis (marking the container “privileged”) when absolutely needed.
  • Seccomp. To filter some potentially dangerous system calls.
  • AppArmor: To provide additional restrictions on mounts, socket, ptrace and file access. Specifically restricting cross-container communication.
  • Capabilities. To prevent the container from loading kernel modules, altering the host system time, …
  • CGroups. To restrict resource usage and prevent DoS attacks against the host.

Rather than exposing those features directly to the user as LXC would, we’ve built a new configuration language which abstracts most of those into something that’s more user friendly. For example, one can tell LXD to pass any host device into the container without having to also lookup its major/minor numbers to manually update the cgroup policy.

Communications with LXD itself are secured using TLS 1.2 with a very limited set of allowed ciphers. When dealing with hosts outside of the system certificate authority, LXD will prompt the user to validate the remote fingerprint (SSH style), then cache the certificate for future use.

The REST API

Everything that LXD does is done over its REST API. There is no other communication channel between the client and the daemon.

The REST API can be access over a local unix socket, only requiring group membership for authentication or over a HTTPs socket using a client certificate for authentication.

The structure of the REST API matches the different components described above and is meant to be very simple and intuitive to use.

When a more complex communication mechanism is required, LXD will negotiate websockets and use those for the rest of the communication. This is used for interactive console session, container migration and for event notification.

With LXD 2.0, comes the /1.0 stable API. We will not break backward compatibility within the /1.0 API endpoint however we may add extra features to it, which we’ll signal by declaring additional API extensions that the client can look for.

Containers at scale

While LXD provides a good command line client, that client isn’t meant to manage thousands of containers on multiple hosts. For that kind of use cases, we have nova-lxd which is an OpenStack plugin that makes OpenStack treat LXD containers in the exact same way it would treat VMs.

This allows for very large deployments of LXDs on a large number of hosts, using the OpenStack APIs to manage network, storage and load-balancing.

Extra information

The main LXD website is at: https://linuxcontainers.org/lxd
Development happens on Github at: https://github.com/lxc/lxd
Mailing-list support happens on: https://lists.linuxcontainers.org
IRC support happens in: #lxcontainers on irc.freenode.net

And if you can’t wait until the next few posts to try LXD, you can take our guided tour online and try it for free right from your web browser!

Read more
Stéphane Graber

As we are getting closer and closer to tagging the final releases of LXC, LXD and LXCFS 2.0, I figured it would be a good idea to talk a bit about everything that went into LXD since we first started that project a year and a half ago.

LXD logo

This is going to be a blog post series similar to what I’ve done for LXC 1.0 a couple years back.

The topics that will be covered are:

I’m hoping to post a couple of those every week for the coming month and a half leading to the Ubuntu 16.04 release.

If you can’t wait for all of those to come out to play with LXD, you can also take the guided tour and play with LXD, online through our online demo page.

Read more
Stéphane Graber

TLDR: NorthSec is an incredible security event, our CTF simulates a whole internet for every participating team. This allows us to create just about anything, from a locked down country to millions of vulnerable IoT devices spread across the globe. However that flexibility comes at a high cost hardware-wise, as we’re getting bigger and bigger, we need more and more powerful servers and networking gear. We’re very actively looking for sponsors so get in touch with me or just buy us something on Amazon!

What’s NorthSec?

NorthSec is one of the biggest on-site Capture The Flag (CTF), security contest in North America. It’s organized yearly over a weekend in Montreal (usually in May) and since the last edition, has been accompanied by a two days security conference before the CTF itself. The rest of this post will only focus on the CTF part though.

the-room

A view of the main room at NorthSec 2015

Teams arrive at the venue on Friday evening, get setup at their table and then get introduced to this year’s scenario and given access to our infrastructure. There they will have to fight their way through challenges, each earning them points and letting them go further and further. On Sunday afternoon, the top 3 teams are awarded their prize and we wrap up for the year.

Size wise, for the past two years we’ve had a physical limit of up to 32 teams of 8 participants and then a bunch of extra unaffiliated visitors. For the 2016 edition, we’re raising this to 50 teams for a grand total of 400 participants, thanks to some shuffling at the venue making some more room for us.

Why is it special?

The above may sound pretty simple and straightforward, however there are a few important details that sets NorthSec apart from other CTFs.

  • It is entirely on-site. There are some very big online CTFs out there but very few on-site ones. Having everyone participating in the same room is valuable from a networking point of view but also ensures fairness by enforcing fixed size teams and equal network bandwidth and latency.
  • Every team gets its very own copy of the whole infrastructure. There are no shared services in the simulated world we provide them. That means one team’s actions cannot impact another.
  • Each simulation is its own virtual world with its own instance of the internet, we use hundreds of LXC containers and thousands of VLANs and networks FOR EVERY TEAM to provide the most realistic and complete environment you can think of.
World map of our fake internet

World map of our fake internet

What’s our infrastructure like?

Due to the very high bandwidth and low latency requirements, most of the infrastructure is hosted on premises and on our hardware. We do plan on offloading Windows virtual machines to a public cloud for the next edition though.

We also provide a mostly legacy free environment to our contestants, all of our challenges are connected to IPv6-only networks and run on 64bit Ubuntu LTS  in LXC with state of the art security configurations.

Our rack

Our rack, on location at NorthSec 2015

 

All in all, for 32 teams (last year’s edition), we had:

  • 48000 virtual network interfaces
  • 2000 virtual carriers
  • 16000 BGP routers
  • 17000 Ubuntu containers
  • 100 Windows virtual machines
  • 20000 routing table entries

And all of that was running on:

  • Two firewalls (DELL SC1425)
  • Two infrastructure servers (DELL SC1425)
  • One management server (HP DL380 G5)
  • Four main contest hosts (HP DL380 G5)
  • Three backup contest hosts (DELL C6100)

On average we had 7 full simulations and 21 virtual machines running on every host (the backup hosts only had one each). That means each of the main contest hosts had:

  • 10500 virtual network interfaces
  • 435 virtual carriers
  • 3500 BGP routers
  • 3700 Ubuntu containers
  • 21 Windows virtual machines
  • 4375 routing table entries

Not too bad for servers that are (SC1425) or are getting close (DL380 G5) to being 10 years old now.

Past infrastructure challenges

In the past editions we’ve found numerous bugs in the various technologies we use when put under such a crazy load:

  • A variety of switch firmware bugs when dealing with several thousand IPv6-only networks.
  • Multiple Linux IPv6 kernel bugs (and one security issue) also related to an excess of IPv6 multicast traffic.
  • Several memory leaks and other bugs in LXC and related components that become very visible when you’re running upwards of 10000 containers.
  • Several more Linux kernel bugs related to performance scaling as we create more and more namespaces and nested namespaces.

As our infrastructure staff is very invested in these technologies by being upstream developers or contributors to the main projects we use, those bugs were all rapidly reported, discussed and fixed. We always look forward to the next NorthSec as an opportunity to test the latest technology at scale in a completely controlled environment.

How can you help?

As I mentioned, we’ve been capped at 32 teams and around 300 attendees for the past two years. Our existing hardware was barely sufficient  to handle  the load during those two editions, we urgently need to refresh our hardware to offer the best possible experience to our participants.

We’re planning on replacing most if not all of our hardware with slightly more recent equivalents, also upgrading from rotating drives to SSDs and improving our network. On the software side, we’ll be upgrading to a newer Linux kernel, possibly to Ubuntu 16.04, switch from btrfs to zfs and from LXC to LXD.

We are a Canadian non-profit organization with all our staff being volunteers so we very heavily rely on sponsors to be able to make the event a success.

If you or your company would like to help by sponsoring our infrastructure, get in touch with me. We have several sponsoring levels and can get you the visibility you’d like, ranging from a mention on our website and at the event to on-site presence with a recruitment booth and even, if our interests align, inclusion of your product in some of our challenges.

We also have an Amazon wishlist of smaller (cheaper) items that we need to buy in the near future. If you buy something from the list, get in touch so we can properly thank you!

Oh and as I briefly mentioned at the beginning, we have a two days, single-track conference ahead of the CTF. We’re actively looking for speakers, if you have something interesting to present, the CFP is here.

Extra resources

Read more
Stéphane Graber

Introduction

For the past 6 months, Serge Hallyn, Tycho Andersen, Chuck Short, Ryan Harper and myself have been very busy working on a new container project called LXD.

Ubuntu 15.04, due to be released this Thursday, will contain LXD 0.7 in its repository. This is still the early days and while we’re confident LXD 0.7 is functional and ready for users to experiment, we still have some work to do before it’s ready for critical production use.

LXD logo

So what’s LXD?

LXD is what we call our container “lightervisor”. The core of LXD is a daemon which offers a REST API to drive full system containers just like you’d drive virtual machines.

The LXD daemon runs on every container host and client tools then connect to those to manage those containers or to move or copy them to another LXD.

We provide two such clients:

  • A command line tool called “lxc”
  • An OpenStack Nova plugin called nova-compute-lxd

The former is mostly aimed at small deployments ranging from a single machine (your laptop) to a few dozen hosts. The latter seamlessly integrates inside your OpenStack infrastructure and lets you manage containers exactly like you would virtual machines.

Why LXD?

LXC has been around for about 7 years now, it evolved from a set of very limited tools which would get you something only marginally better than a chroot, all the way to the stable set of tools, stable library and active user and development community that we have today.

Over those years, a lot of extra security features were added to the Linux kernel and LXC grew support for all of them. As we saw the need for people to build their own solution on top of LXC, we’ve developed a public API and a set of bindings. And last year, we’ve put out our first long term support release which has been a great success so far.

That being said, for a while now, we’ve been wanting to do a few big changes:

  • Make LXC secure by default (rather than it being optional).
  • Completely rework the tools to make them simpler and less confusing to newcomers.
  • Rely on container images rather than using “templates” to build them locally.
  • Proper checkpoint/restore support (live migration).

Unfortunately, solving any of those means doing very drastic changes to LXC which would likely break our existing users or at least force them to rethink the way they do things.

Instead, LXD is our opportunity to start fresh. We’re keeping LXC as the great low level container manager that it is. And build LXD on top of it, using LXC’s API to do all the low level work. That achieves the best of both worlds, we keep our low level container manager with its API and bindings but skip using its tools and templates, instead replacing those by the new experience that LXD provides.

How does LXD relate to LXC, Docker, Rocket and other container projects?

LXD is currently based on top of LXC. It uses the stable LXC API to do all the container management behind the scene, adding the REST API on top and providing a much simpler, more consistent user experience.

The focus of LXD is on system containers. That is, a container which runs a clean copy of a Linux distribution or a full appliance. From a design perspective, LXD doesn’t care about what’s running in the container.

That’s very different from Docker or Rocket which are application container managers (as opposed to system container managers) and so focus on distributing apps as containers and so very much care about what runs inside the container.

There is absolutely nothing wrong with using LXD to run a bunch of full containers which then run Docker or Rocket inside of them to run their different applications. So letting LXD manage the host resources for you, applying all the security restrictions to make the container safe and then using whatever application distribution mechanism you want inside.

Getting started with LXD

The simplest way for somebody to try LXD is by using it with its command line tool. This can easily be done on your laptop or desktop machine.

On an Ubuntu 15.04 system (or by using ppa:ubuntu-lxc/lxd-stable on 14.04 or above), you can install LXD with:

sudo apt-get install lxd

Then either logout and login again to get your group membership refreshed, or use:

newgrp lxd

From that point on, you can interact with your newly installed LXD daemon.

The “lxc” command line tool lets you interact with one or multiple LXD daemons. By default it will interact with the local daemon, but you can easily add more of them.

As an easy way to start experimenting with remote servers, you can add our public LXD server at https://images.linuxcontainers.org:8443
That server is an image-only read-only server, so all you can do with it is list images, copy images from it or start containers from it.

You’ll have to do the following to: add the server, list all of its images and then start a container from one of them:

lxc remote add images images.linuxcontainers.org
lxc image list images:
lxc launch images:ubuntu/trusty/i386 ubuntu-32

What the above does is define a new “remote” called “images” which points to images.linuxcontainers.org. Then list all of its images and finally start a local container called “ubuntu-32” from the ubuntu/trusty/i386 image. The image will automatically be cached locally so that future containers are started instantly.

The “<remote name>:” syntax is used throughout the lxc client. When not specified, the default “local” remote is assumed. Should you only care about managing a remote server, the default remote can be changed with “lxc remote set-default”.

Now that you have a running container, you can check its status and IP information with:

lxc list

Or get even more details with:

lxc info ubuntu-32

To get a shell inside the container, or to run any other command that you want, you may do:

lxc exec ubuntu-32 /bin/bash

And you can also directly pull or push files from/to the container with:

lxc file pull ubuntu-32/path/to/file .
lxc file push /path/to/file ubuntu-32/

When done, you can stop or delete your container with one of those:

lxc stop ubuntu-32
lxc delete ubuntu-32

What’s next?

The above should be a reasonably comprehensive guide to how to use LXD on a single system. Of course, that’s not the most interesting thing to do with LXD. All the commands shown above can work against multiple hosts, containers can be remotely created, moved around, copied, …

LXD also supports live migration, snapshots, configuration profiles, device pass-through and more.

I intend to write some more posts to cover those use cases and features as well as highlight some of the work we’re currently busy doing.

LXD is a pretty young but very active project. We’ve had great contributions from existing LXC developers as well as newcomers.

The project is entirely developed in the open at https://github.com/lxc/lxd. We keep track of upcoming features and improvements through the project’s issue tracker, so it’s easy to see what will be coming soon. We also have a set of issues marked “Easy” which are meant for new contributors as easy ways to get to know the LXD code and contribute to the project.

LXD is an Apache2 licensed project, written in Go and which doesn’t require a CLA to contribute to (we do however require the standard DCO Signed-off-by). It can be built with both golang and gccgo and so works on almost all architectures.

Extra resources

More information can be found on the official LXD website:
https://linuxcontainers.org/lxd

The code, issues and pull requests can all be found on Github:
https://github.com/lxc/lxd

And a good overview of the LXD design and its API may be found in our specs:
https://github.com/lxc/lxd/tree/master/specs

Conclusion

LXD is a new and exciting project. It’s an amazing opportunity to think fresh about system containers and provide the best user experience possible, alongside great features and rock solid security.

With 7 releases and close to a thousand commits by 20 contributors, it’s a very active, fast paced project. Lots of things still remain to be implemented before we get to our 1.0 milestone release in early 2016 but looking at what was achieved in just 5 months, I’m confident we’ll have an incredible LXD in another 12 months!

For now, we’d welcome your feedback, so install LXD, play around with it, file bugs and let us know what’s important for you next.

Read more
Alan Pope

Ubuntu Core Apps
As we come to the end of 2014, looking forward to new devices running Ubuntu in our immediate future, it’s time for one last set of Hack Days of the year.

Next week, from Monday 8th December till Friday 12th we’re going to be having another set of Core Apps Hack Days. We’ve had a few of these this year which have been a great way to focus attention on specific applications and their dependent components in the platform. They’re also a nice gateway for getting new people into the Core Apps project and Ubuntu development in general.

The Core Apps are community maintained Free Software applications which were created for Ubuntu devices, but also work on the Ubuntu desktop. We welcome new developers, testers, autopilot writers, artists and translators to get involved in these exciting projects.

The schedule

As with previous hack days we’re going to focus on specific apps on each day, which we run from 9:00 UTC until 21:00 UTC. In summary our schedule looks like this:-

  • Monday 8th – Calculator, Terminal & Clock
  • Tuesday 9th – File Manager & Calendar
  • Wednesday 10th – Music & Document Viewer – QA Day Workshop: writing tests for the core apps (18:00UTC)
  • Thursday 11th – Shorts & Reminders
  • Friday 12th – Weather & Dekko (email)

A QA treat

Creating core apps involves close coordination between developers and designers to provide the right set of features, high usability and appealing visuals. All these would be nothing without a suite of automated tests that are run to ensure the features are rock-solid and that no regressions are introduced with new development.

All core apps include Autopilot and QML tests that we are constantly expanding to increase test coverage. Writing tests for core apps is a nice way to get started contributing. All you’ll need is some Python knowledge for Autopilot tests or QML for QML tests. Our quality man, Nicholas Skaggs will be running a live video workshop on Wednesday Dec 10th, at 18:00UTC, as an on-ramp to learn how to create tests.

Join the fest

The Hack Days will be happening live at the #ubuntu-app-devel IRC channel on Freenode

The QA Workshop will be happening also live on Ubuntu On Air. You can watch the video and ask your questions on the same IRC channel.

We’ll blog more details about the apps each day next week with links to specific bugs, tasks and goals, so stay tuned!

As always we greatly appreciate all contributions to the Core Apps project during the Hack Days, but welcome community efforts all year round, so if this week doesn’t work for you, feel free to drop by #ubuntu-app-devel on Freenode any time and speak to me, popey.

Read more
David Planella

image-phone-naturallyneat-medium

The 5 weeks after the Ubuntu Scopes Showdown announcement are coming to an end, it’s time to start putting the pencils down and submitting your scopes for the judges to do their reviews.

While you’ve still got two days to fix some bugs and do some final polish, the 3rd of December is the last day for submissions to be accepted for the Showdown. Remember that to qualify, you’ll need to:

  • Register your scope for the contest
  • Submit your scope to the Ubuntu Software Store

Registering your scope

To register your scope for the judges’ review, you’ll simply need a couple of minutes to fill in the registration form. It might be worth filling it in advance, even if you are planning to upload your app at the last minute.

You can submit the form now and still upload new revisions of your app until the 3rd of December.

Register your scope for participation

Submitting your scope

Submitting your scope to the store should also be quick and easy. The upload workflow is exactly the same as for apps, and with automated reviews it takes just a few minutes from upload to your scope being available for everyone on the Ubuntu Software Store.

To ensure your scope is discoverable and looks good, you might want to check out the scope upload tips ›

And when you’re ready to start the upload, you can follow the 5-step process to get it published ›

Need help?

If you need help with any of the above, feel free to reach out in any of the channels below:

Looking forward to seeing your scopes in the store!

Read more
David Callé

Now that your scope is in a working state, it’s time to get it ready for publication. In this tutorial you will learn how to make your scope look good when the user is browsing the store or the list of scopes installed on the phone.

In the next steps, we are going to prepare a few graphics, edit the <scope>.ini file located in the data directory of your project and package the scope for the store.

Read…

scope_prev_all

Read more
David Planella

.tg {border-collapse:collapse;border-spacing:0;border-color:#ccc;margin:20px auto;width:680px !important;} .tg td{font-size:14px;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;border-color:#ccc;color:#333;background-color:#fff;} .tg th{font-size:14px;font-weight:normal;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;border-color:#ccc;color:#333;background-color:#f0f0f0;} .tg .tg-4eph{background-color:#f9f9f9}

image-phone-naturallyneat-medium

This is a week full of exciting events in the Ubuntu world! Following on the series of Ubuntu Scopes Workshops for the Ubuntu Scopes Showdown, we’re thrilled to announce more Scopes Workshops sessions as part of the Ubuntu Online Summit.

Scopes workshops: learn more and ask your questions

In order to support participants of the Scope Showdown, we’re organizing a series of workshops around different topics on writing scopes. These will be 1 hour hands-on sessions where the presenter will be demonstrating the topic live on video, with real code and using the Ubuntu SDK.

These are also meant to be interactive, so during and after the session the presenter will be answering the questions posted in real time by developers on the chat widget on each session page. Here’s the schedule for the workshops:

Workshop Time Presenter
Online Accounts for Scopes Developers Thursday, 13th Nov at 14:00UTC Alberto Mardegan
Scope Development How-Tos Thursday, 13th Nov at 15:00UTC Thomas Strehl & the Unity API Team

In a nutshell:

  • WHAT: Scopes workshops at the Ubuntu Online Summit
  • WHEN: Thursday, 13th November, starting at 14:00 UTC
  • WHERE: At the Ubuntu Online Summit

Looking forward to seeing you there!

Read more
David Callé

Departments are a way for the user to navigate the data source exposed by a scope. A music scope can use them to allow browsing by genre, a Youtube scope could list channels and playlists, a news scope could use them for listing topics, etc. Departments can also display a full hierarchy of sub-departments.

In this tutorial, you are going to learn how to create and add them to your scope.

Read…

scope_dep8 scope_dep9

Read more
David Planella

More content coming up for app and scope developers targeting Ubuntu on devices: this time around Online Accounts!

Learn the concepts on how to write account providers for online services using the Ubuntu Online Accounts API (UOA), and let the API take care of all the complexity and security for you, so that you can concentrate on your code.

Go to the Online Accounts Developer Guide >

Read more
David Planella

.tg {border-collapse:collapse;border-spacing:0;border-color:#ccc;margin:20px auto;width:680px !important;} .tg td{font-size:14px;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;border-color:#ccc;color:#333;background-color:#fff;} .tg th{font-size:14px;font-weight:normal;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;border-color:#ccc;color:#333;background-color:#f0f0f0;} .tg .tg-4eph{background-color:#f9f9f9}

image-phone-naturallyneat-medium

Hot on the heels of the Ubuntu Scopes Showdown start, we’re thrilled to announce the Scopes Workshops sessions to get developers kickstarted and ask all of their questions!

Scopes workshops: learn and ask your questions

In order to support participants of the Scope Showdown, we’re organizing a series of workshops around different topics on writing scopes. These will be 1 hour hands-on sessions where the presenter will be demonstrating the topic live on video, with real code and using the Ubuntu SDK.

These are also meant to be interactive, so during and after the session the presenter will be answering the questions posted in real time by developers on the chat widget on Ubuntu on air. Here’s the schedule for the workshops:

Workshop Time Presenter
Setting up your scopes development environment Thursday, 6th Nov at 16:00UTC David Planella
Introduction to scopes development Thursday, 6th Nov at 17:00UTC David Callé
Adding location support to your scope Thursday, 6th Nov at 18:00UTC Kyle Nitzsche

In a nutshell:

  • WHAT: Scopes workshops
  • WHEN: Thursday, 6th November, starting at 16:00 UTC
  • WHERE: On Ubuntu on air

More on scopes at the Ubuntu Online Summit

Next week another key event in the Ubuntu world is coming: the Ubuntu Online Summit (UOS): 3 days of discussions, workshops and presentations about the upcoming work and plans for the next Ubuntu release.

As part of UOS, we’ll be running another set of workshops, so stay tuned for the schedule to learn more about scopes. Register to attend UOS >

Looking forward to seeing the scopes everyone comes up with!

Read more
David Callé

Ubuntu has a solid location stack, allowing users to select which applications have access to the device location. This also applies to scopes and is very easy to add to your code.

In this short tutorial, you are going to learn how to bring location awareness to your scope.

Read…

scope-blue1scope-blue0

Read more
Michael Hall

With all of the new documentation coming to support the development of Unity Scopes, it’s time for us to have another development shodown! Contestants will have five (5) weeks to develop a project, from scratch, and submit it to the Ubuntu Store. But this time all of the entries must be Scopes.

Be sure to update to the latest SDK packages to ensure that you have the correct template and tools. You should also create a new Click chroot to get the latest build and runtime packages.

Prizes

prizesWe’ve got some great prizes lined up for the winners of this competition.

  • 1st place will win a new Dell XPS 13 Laptop, Developer Edition (preloaded with Ubuntu)
  • Runners up will receive one of:
    • Logitech UE Boom Bluetooth speakers
    • Nexus 7 running Ubuntu
    • An Ubuntu bundle, featuring:
      • Ubuntu messenger bag
      • Ubuntu Touch Infographic T-shirt
      • Ubuntu Neoprene Laptop Sleeve
    • An Ubuntu bundle, featuring:
      • Ubuntu backpack
      • Ubuntu Circle of Friends Dot Design T-shirt
      • Ubuntu Neoprene Laptop Sleeve

Judging

Scope entries will be reviewed by a panel of judges from a variety of backgrounds and specialties, all of whom will evaluate the scope based on the following criteria:

  • General Interest – Scopes that are of more interest to general phone users will be scored higher. We recommend identifying what kind of content phone users want to have fast, easy access to and then finding an online source where you can query for it
  • Creativity – Scopes are a unique way of bringing content and information to a user, and we’ve only scratched the surface of what they can do. Thinking outside the box and providing something new and exciting will lead to a higher score for your Scope
  • Features – There’s more to scopes than basic searching, take advantage of the departments, categories and settings APIs to enhance the functionality of your Scope
  • Design – Scopes offer a variety of ways to customize the way content is displayed, from different layouts to visual styling. Take full advantage of what’s possible to provide a beautiful presentation of your results.
  • Awareness / Promotion – we will award extra points to those of you who blog, tweet, facebook, Google+, reddit, and otherwise share updates and information about your scope as it progresses.

The judges for this contest are:

  • Chris Wayne developer behind a number of current pre-installed Scopes
  • Joey-Elijah Sneddon Author and editor of Omg!Ubuntu!
  • Victor Thompson Ubuntu Core Apps developer
  • Jouni Helminen Designer at Canonical
  • Alan Pope from the Ubuntu Community Team at Canonical

Learn how to write Ubuntu Scopes

To get things started we’ve recently introduced a new Unity Scope project template into the Ubuntu SDK, you can use this to get a working foundation for your code right away. Then you can follow along with our new SoundCloud scope tutorial to learn how to tailor your code to a remote data source and give your scope a unique look and feel that highlights both the content and the source. To help you out along the way, we’ll be scheduling a series of online Workshops that will cover how to use the Ubuntu SDK and the Scope APIs. In the last weeks of the contest we will also be hosting a hackathon on our IRC channel (#ubuntu-app-devel on Freenode) to answer any last questions and help you get your c If you cannot join those, you can still find everything you need to know in our scope developer documentation.

How to participate

If you are not a programmer and want to share some ideas for cool scopes, be sure to add and vote for scopes on our reddit page. The contest is free to enter and open to everyone. The five week period starts on the Thursday 30th October and runs until Wednesday 3rd December 2014! Enter the Ubuntu Scope Showdown >

Read more
David Callé

A scope can provide persistent settings for simple customizations, such as allowing the user to configure an email address or select a distance unit as metric or imperial.

In this tutorial, you well learn how to add settings to your scope and allow users to customize their experience.

Read…

scope-settings_coffeenearby2 scope-settings_visitparis2 scope-settings_indieconcerts1

Read more
David Callé

Scopes come with a very flexible customization system. From picking the text color to rearranging how results are laid out, a scope can easily look like a generic RSS reader, a music library or even a store front.

In this new article, you will learn how to make your scope shine by customizing its results, changing its colors, adding a logo and adapting its layout to present your data in the best possible way. Read…

screenshot20145615_125616591

Read more
Stéphane Graber

I often have to deal with VPNs, either to connect to the company network, my own network when I’m abroad or to various other places where I’ve got servers I manage.

All of those VPNs use OpenVPN, all with a similar configuration and unfortunately quite a lot of them with overlapping networks. That means that when I connect to them, parts of my own network are no longer reachable or it means that I can’t connect to more than one of them at once.

Those I suspect are all pretty common issues with VPN users, especially those working with or for companies who over the years ended up using most of the rfc1918 subnets.

So I thought, I’m working with containers every day, nowadays we have those cool namespaces in the kernel which let you run crazy things as a a regular user, including getting your own, empty network stack, so why not use that?

Well, that’s what I ended up doing and so far, that’s all done in less than 100 lines of good old POSIX shell script :)

That gives me, fully unprivileged non-overlapping VPNs! OpenVPN and everything else run as my own user and nobody other than the user spawning the container can possibly get access to the resources behind the VPN.

The code is available at: git clone git://github.com/stgraber/vpn-container

Then it’s as simple as: ./start-vpn VPN-NAME CONFIG

What happens next is the script will call socat to proxy the VPN TCP socket to a UNIX socket, then a user namespace, network namespace, mount namespace and uts namespace are all created for the container. Your user is root in that namespace and so can start openvpn and create network interfaces and routes. With careful use of some bind-mounts, resolvconf and byobu are also made to work so DNS resolution is functional and we can start byobu to easily allow as many shell as you want in there.

In the end it looks like this:

stgraber@dakara:~/vpn$ ./start-vpn stgraber.net ../stgraber-vpn/stgraber.conf 
WARN: could not reopen tty: No such file or directory
lxc: call to cgmanager_move_pid_abs_sync(name=systemd) failed: invalid request
Fri Sep 26 17:48:07 2014 OpenVPN 2.3.2 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [eurephia] [MH] [IPv6] built on Feb  4 2014
Fri Sep 26 17:48:07 2014 WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
Fri Sep 26 17:48:07 2014 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Fri Sep 26 17:48:07 2014 Attempting to establish TCP connection with [AF_INET]127.0.0.1:1194 [nonblock]
Fri Sep 26 17:48:07 2014 TCP connection established with [AF_INET]127.0.0.1:1194
Fri Sep 26 17:48:07 2014 TCPv4_CLIENT link local: [undef]
Fri Sep 26 17:48:07 2014 TCPv4_CLIENT link remote: [AF_INET]127.0.0.1:1194
Fri Sep 26 17:48:09 2014 [vorash.stgraber.org] Peer Connection Initiated with [AF_INET]127.0.0.1:1194
Fri Sep 26 17:48:12 2014 TUN/TAP device tun0 opened
Fri Sep 26 17:48:12 2014 Note: Cannot set tx queue length on tun0: Operation not permitted (errno=1)
Fri Sep 26 17:48:12 2014 do_ifconfig, tt->ipv6=1, tt->did_ifconfig_ipv6_setup=1
Fri Sep 26 17:48:12 2014 /sbin/ip link set dev tun0 up mtu 1500
Fri Sep 26 17:48:12 2014 /sbin/ip addr add dev tun0 172.16.35.50/24 broadcast 172.16.35.255
Fri Sep 26 17:48:12 2014 /sbin/ip -6 addr add 2001:470:b368:1035::50/64 dev tun0
Fri Sep 26 17:48:12 2014 /etc/openvpn/update-resolv-conf tun0 1500 1544 172.16.35.50 255.255.255.0 init
dhcp-option DNS 172.16.20.30
dhcp-option DNS 172.16.20.31
dhcp-option DNS 2001:470:b368:1020:216:3eff:fe24:5827
dhcp-option DNS nameserver
dhcp-option DOMAIN stgraber.net
Fri Sep 26 17:48:12 2014 add_route_ipv6(2607:f2c0:f00f:2700::/56 -> 2001:470:b368:1035::1 metric -1) dev tun0
Fri Sep 26 17:48:12 2014 add_route_ipv6(2001:470:714b::/48 -> 2001:470:b368:1035::1 metric -1) dev tun0
Fri Sep 26 17:48:12 2014 add_route_ipv6(2001:470:b368::/48 -> 2001:470:b368:1035::1 metric -1) dev tun0
Fri Sep 26 17:48:12 2014 add_route_ipv6(2001:470:b511::/48 -> 2001:470:b368:1035::1 metric -1) dev tun0
Fri Sep 26 17:48:12 2014 add_route_ipv6(2001:470:b512::/48 -> 2001:470:b368:1035::1 metric -1) dev tun0
Fri Sep 26 17:48:12 2014 Initialization Sequence Completed


To attach to this VPN, use: byobu -S /home/stgraber/vpn/stgraber.net.byobu
To kill this VPN, do: byobu -S /home/stgraber/vpn/stgraber.net.byobu kill-server
or from inside byobu: byobu kill-server

After that, just copy/paste the byobu command and you’ll get a shell inside the container. Don’t be alarmed by the fact that you’re root in there. root is mapped to your user’s uid and gid outside the container so it’s actually just your usual user but with a different name and with privileges against the resources owned by the container.

You can now use the VPN as you want without any possible overlap or conflict with any route or VPN you may be running on that system and with absolutely no possibility that a user sharing your machine may access your running VPN.

This has so far been tested with 5 different VPNs, on a regular Ubuntu 14.04 LTS system with all VPNs being TCP based. UDP based VPNs would probably just need a couple of tweaks to the socat unix-socket proxy.

Enjoy!

Read more
David Planella

OSM GPS dump

We’re very excited to announce an agreement with Nokia HERE to provide A-GPS support on Ubuntu. The new platform service will enable developers to obtain accurate positioning data for their location-based apps in under two minutes, a significantly shorter Time To First Fix (TTFF) than the average for raw GPS technologies.

Faster positioning

While Ubuntu already features GPS-based location, it has always been a key requirement for the OS to provide application developers with rapid and efficient location positioning capabilities.

The new positioning service will be a hybrid solution integrating A-GPS and WiFi positioning, a powerful combo to help obtaining a very fast and accurate TTFF. The system is to be functional by the Release To Manufacturer (RTM) milestone, and available on the regular Ubuntu builds and for retail phones shipping Ubuntu.

Privacy and security

With the user’s explicit consent, anonymous data related to signal strength of local WiFi signals and radio cells can be contributed to crowd-sourcing location services, with the purpose of improving the overall quality of the positioning service for all users.

In line with Ubuntu’s privacy policy, no personal data of any nature is to be collected and released. Users will also be able to opt-out of this service if they do not wish their mobile handset to collect this type of data.

The positioning system will also be run under strict confinement, so that the service and its data cannot be accessed without the user explicitly granting access. With Ubuntu’s trust model, a confined application has to be granted trust by the user to gain access to security- or privacy-relevant system components.

Mapping capabilities

As the new service is to be focused on positioning, it will be decoupled from any mapping solution. Ubuntu Developers, as before, will have a choice of mapping services to use for their applications, including Nokia HERE, OpenStreetMap and others.

Header image based on “openstreetmap gps coverage” by Steven Kay, CC-BY-SA 2.0.

Read more
Michael Hall

As part of the continued development of the Ubuntu platform, the Content Hub has gained the ability to share links (and soon text) as a content type, just as it has been able to share images and other file-based content in the past. This allows applications to more easily, and more consistently, share things to a user’s social media accounts.

Consolidating APIs

facebook-sharing
Thanks to the collaborative work going on between the Content Hub and the Ubuntu Webapps developers, it is now possible for remote websites to be packaged with local user scripts that provide deep integration with our platform services. One of the first to take advantage of this is the Facebook webapp, which while displaying remote content via a web browser wrapper, is also a Content Hub importer. This means that when you go to share an image from the Gallery app, the Facebook webapp is displayed as an optional sharing target for that image. If you select it, it will use the Facebook web interface to upload that image to your timeline, without having to go through the separate Friends API.

This work not only brings the social sharing user experience inline with the rest of the system’s content sharing experience, it also provide a much simpler API for application developers to use for accomplishing the same thing. As a result, the Friends API is being deprecated in favor of the new Content Hub functionality.

What it means for App Devs

Because this is an API change, there are things that you as an app developer need to be aware of. First, though the API is being deprecated immediately, it is not being removed from the device images until after the release of 14.10, which will continue to support the ubuntu-sdk-14.04 framework which included the Friends API. The API will not be included in the final ubuntu-sdk-14.10 framework, or any new 14.10-dev frameworks after -dev2.

After the 14.10 release in October, when device images start to build for utopic+1, the ubuntu-sdk-14.04 framework will no longer be on the images. So if you haven’t updated your Click package by then to use the ubuntu-sdk-14.10 framework, it won’t be available to install on devices with the new image. If you are not using the Friends API, this would simply be a matter of changing your package metadata to the new framework version.  For new apps, it will default to the newer version to begin with, so you shouldn’t have to do anything.

Read more
David Planella

Ubuntu loves HTML5Here’s a reminder about next Monday’s 7th of July Ubuntu HTML5 apps session in Barcelona.

At this free event, I’ll be presenting Ubuntu’s HTML5 development story, together with a live coding session and a Q&A round at the end. You’ll learn how to use the Ubuntu SDK and the UI toolkit to easily reuse your web skills to create stunning Ubuntu apps.

HTML5 is the other side of the coin of the Ubuntu app developer offering, where both web and native are first class citizens, offering a very flexible yet focused approach for application development. Teaming up with BeMyApp meetups, the session will start at 7 p.m. at Barcelona’s Mobile World Centre.

I look forward to seeing you there!

Register here for the HTML5 session >

Read more