Canonical Voices

Posts tagged with 'ubuntu core'

abeato

Ubuntu Core (UC) is Canonical’s take in the IoT space. There are pre-built images for officially supported devices, like Raspberry Pi or Intel NUCs, but if we have something else and there is no community port, we need to create the UC image ourselves. High level instructions on how to do this are found in the official docs. The process is straightforward once we have two critical components: the kernel and the gadget snap.

Creating these snaps is not necessarily complex, but there can be bumps in the road if you are new to the task. In this post I explain how I created them for the Jetson TX1 developer kit board, and how they were used to create a UC image for said device, hoping this will provide new tricks to hackers working on ports for other devices. All the sources for the snaps and the build scripts are available in github:
https://github.com/alfonsosanchezbeato/jetson-kernel-snap
https://github.com/alfonsosanchezbeato/jetson-gadget-snap
https://github.com/alfonsosanchezbeato/jetson-ubuntu-core

So, let’s start with…

The kernel snap

The Linux kernel that we will use needs some kernel configuration options to be activated, and it is also especially important that it has a modern version of apparmor so snaps can be properly confined. The official Jetson kernel is the 4.4 release, which is quite old, but fortunately Canonical has a reference 4.4 kernel with all the needed patches for snaps backported. Knowing this, we are a git format-patch command away to obtain the patches we will use on top of the nvidia kernel. The patches include also files with the configuration options that we need for snaps, plus some changes so the snap could be successfully compiled on Ubuntu 18.04 desktop.

Once we have the sources, we need, of course, to create a snapcraft.yaml file that will describe how to build the kernel snap. We will walk through it, highlighting the parts more specific to the Jetson device.

Starting with the kernel part, it turns out that we cannot use easily the kernel plugin, due to the special way in which the kernel needs to be built: nvidia distributes part of the needed drivers as separate repositories to the one used by the main kernel tree. Therefore, I resorted to using the nil plugin so I could hand-write the commands to do the build.

The pull stage that resulted is

override-pull: |
  snapcraftctl pull
  # Get kernel sources, which are distributed across different repos
  ./source_sync.sh -k tegra-l4t-r28.2.1
  # Apply canonical patches - apparmor stuff essentially
  cd sources/kernel/display
  git am ../../../patch-display/*
  cd -
  cd sources/kernel/kernel-4.4
  git am ../../../patch/*

which runs a script to retrieve the sources (I pulled this script from nvidia Linux for Tegra -L4T- distribution) and applies Canonical patches.

The build stage is a few more lines, so I decided to use an external script to implement it. We will analyze now parts of it. For the kernel configuration we add all the necessary Ubuntu bits:

make "$JETSON_KERNEL_CONFIG" \
    snappy/containers.config \
    snappy/generic.config \
    snappy/security.config \
    snappy/snappy.config \
    snappy/systemd.config

Then, to do the build we run

make -j"$num_cpu" Image modules dtbs

An interesting catch here is that zImage files are not supported due to lack of a decompressor implementation in the arm64 kernel. So we have to build an uncompressed Image instead.

After some code that stages the built files so they are included in the snap later, we retrieve the initramfs from the core snap. This step is usually hidden from us by the kernel plugin, but this time we have to code it ourselves:

# Get initramfs from core snap, which we need to download
core_url=$(curl -s -H "X-Ubuntu-Series: 16" -H "X-Ubuntu-Architecture: arm64" \
                "https://search.apps.ubuntu.com/api/v1/snaps/details/core?channel=stable" \
               | jq -r ".anon_download_url")
curl -L "$core_url" > core.snap
# Glob so we get both link and regular file
unsquashfs core.snap "boot/initrd.img-core*"
cp squashfs-root/boot/initrd.img-core "$SNAPCRAFT_PART_INSTALL"/initrd.img
ln "$SNAPCRAFT_PART_INSTALL"/initrd.img "$SNAPCRAFT_PART_INSTALL"/initrd-"$KERNEL_RELEASE".img

Moving back to the snapcraft recipe we also have an initramfs part, which takes care of doing some changes to the default initramfs shipped by UC:

initramfs:
  after: [ kernel ]
  plugin: nil
  source: ../initramfs
  override-build: |
    find . | cpio --quiet -o -H newc | lzma >> "$SNAPCRAFT_STAGE"/initrd.img

Here we are taking advantage of the fact that the initramfs can be built as a concatenation of compressed cpio archives. When the kernel decompresses it, the files included in the later archives overwrite the files from the first ones, which allows us to modify easily files in the initramfs without having to change the one shipped with core. The change that we are doing here is a modification to the resize script that allows UC to get all the free space in the disk on first boot. The modification makes sure this happens in the case when the partition is already taken all available space but the filesystem does not. We could remove this modification when these changes reach the core snap, thing that will happen eventually.

The last part of this snap is the firmware part:

firmware:
  plugin: nil
  override-build: |
    set -xe
    wget https://developer.nvidia.com/embedded/dlc/l4t-jetson-tx1-driver-package-28-2-ga -O Tegra210_Linux_R28.2.0_aarch64.tbz2
    tar xf Tegra210_Linux_R28.2.0_aarch64.tbz2 Linux_for_Tegra/nv_tegra/nvidia_drivers.tbz2
    tar xf Linux_for_Tegra/nv_tegra/nvidia_drivers.tbz2 lib/firmware/
    cd lib; cp -r firmware/ "$SNAPCRAFT_PART_INSTALL"
    mkdir -p "$SNAPCRAFT_PART_INSTALL"/firmware/gm20b
    cd "$SNAPCRAFT_PART_INSTALL"/firmware/gm20b
    ln -sf "../tegra21x/acr_ucode.bin" "acr_ucode.bin"
    ln -sf "../tegra21x/gpmu_ucode.bin" "gpmu_ucode.bin"
    ln -sf "../tegra21x/gpmu_ucode_desc.bin" "gpmu_ucode_desc.bin"
    ln -sf "../tegra21x/gpmu_ucode_image.bin" "gpmu_ucode_image.bin"
    ln -sf "../tegra21x/gpu2cde.bin" "gpu2cde.bin"
    ln -sf "../tegra21x/NETB_img.bin" "NETB_img.bin"
    ln -sf "../tegra21x/fecs_sig.bin" "fecs_sig.bin"
    ln -sf "../tegra21x/pmu_sig.bin" "pmu_sig.bin"
    ln -sf "../tegra21x/pmu_bl.bin" "pmu_bl.bin"
    ln -sf "../tegra21x/fecs.bin" "fecs.bin"
    ln -sf "../tegra21x/gpccs.bin" "gpccs.bin"

Here we download some files so we can add firmware blobs to the snap. These files come separate from nvidia kernel sources.

So this is it for the kernel snap, now you just need to follow the instructions to get it built.

The gadget snap

Time now to take a look at the gadget snap. First, I recommend to start by reading great ogra’s post on gadget snaps for devices with u-boot bootloader before going through this section. Now, same as for the kernel one, we will go through the different parts that are defined in the snapcraft.yaml file. The first one builds the u-boot binary:

uboot:
  plugin: nil
  source: git://nv-tegra.nvidia.com/3rdparty/u-boot.git
  source-type: git
  source-tag: tegra-l4t-r28.2
  override-pull: |
    snapcraftctl pull
    # Apply UC patches + bug fixes
    git am ../../../uboot-patch/*.patch
  override-build: |
    export ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-
    make p2371-2180_defconfig
    nice make -j$(nproc)
    cp "$SNAPCRAFT_PART_BUILD"/u-boot.bin $SNAPCRAFT_PART_INSTALL"/

We decided again to use the nil plugin as we need to do some special quirks. The sources are pulled from nvidia’s u-boot repository, but we apply some patches on top. These patches, along with the uboot environment, provide

  • Support for loading the UC kernel and initramfs from disk
  • Support for the revert functionality in case a core or kernel snap installation goes wrong
  • Bug fixes for u-boot’s ext4 subsystem – required because the just mentioned revert functionality needs to call u-boot’s command saveenv, which happened to be broken for ext4 filesystems in tegra’s u-boot

More information on the specifics of u-boot patches for UC can be found in this great blog post.

The only other part that the snap has is uboot-env:

uboot-env:
  plugin: nil
  source: uboot-env
  override-build: |
    mkenvimage -r -s 131072 -o uboot.env uboot.env.in
    cp "$SNAPCRAFT_PART_BUILD"/uboot.env "$SNAPCRAFT_PART_INSTALL"/
    # Link needed for ubuntu-image to work properly
    cd "$SNAPCRAFT_PART_INSTALL"/; ln -s uboot.env uboot.conf
  build-packages:
    - u-boot-tools

This simply encodes the uboot.env.in file into a format that is readable by u-boot. The resulting file, uboot.env, is included in the snap.

This environment is where most of the support for UC is encoded. I will not delve too much into the details, but just want to mention that the variables that need to be edited usually for new devices are

  • devnum, partition, and devtype to set the system boot partition, from which we load the kernel and initramfs
  • fdtfile, fdt_addr_r, and fdt_high to determine the name of the device tree and where in memory it should be loaded
  • ramdisk_addr_r and initrd_high to set the loading location for the initramfs
  • kernel_addr_r to set where the kernel needs to be loaded
  • args contains kernel arguments and needs to be adapted to the device specifics
  • Finally, for this device, snappy_boot was changed so it used booti instead of bootz, as we could not use a compressed kernel as explained above

Besides the snapcraft recipe, the other mandatory file when defining a gadget snap is the gadget.yaml file. This file defines, among other things, the image partitioning layout. There is more to it, but in this case, partitioning is the only thing we have defined:

volumes:
  jetson:
    bootloader: u-boot
    schema: gpt
    structure:
      - name: system-boot
        role: system-boot
        type: 0FC63DAF-8483-4772-8E79-3D69D8477DE4
        filesystem: ext4
        filesystem-label: system-boot
        offset: 17408
        size: 67108864
      - name: TBC
        type: EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
        size: 2097152
      - name: EBT
        type: EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
        size: 4194304
      - name: BPF
        type: EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
        size: 2097152
      - name: WB0
        type: EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
        size: 6291456
      - name: RP1
        type: EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
        size: 4194304
      - name: TOS
        type: EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
        size: 6291456
      - name: EKS
        type: EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
        size: 2097152
      - name: FX
        type: EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
        size: 2097152
      - name: BMP
        type: EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
        size: 134217728
      - name: SOS
        type: EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
        size: 20971520
      - name: EXI
        type: EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
        size: 67108864
      - name: LNX
        type: 0FC63DAF-8483-4772-8E79-3D69D8477DE4
        size: 67108864
        content:
          - image: u-boot.bin
      - name: DTB
        type: EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
        size: 4194304
      - name: NXT
        type: EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
        size: 2097152
      - name: MXB
        type: EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
        size: 6291456
      - name: MXP
        type: EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
        size: 6291456
      - name: USP
        type: EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
size: 2097152

The Jetson TX1 has a complex partitioning layout, with many partitions being allocated for the first stage bootloader, and many others that are undocumented. So, to minimize the risk of touching a critical partition, I preferred to keep most of them untouched and do just the minor amount of changes to fit UC into the device. Therefore, the gadget.yaml volumes entry mainly describes the TX1 defaults, with the main differences comparing to the original being:

  1. The APP partition is renamed to system-boot and reduced to only 64MB. It will contain the uboot environment file plus the kernel and initramfs, as usual in UC systems with u-boot bootloader.
  2. The LNX partition will contain our u-boot binary
  3. If a partition with role: system-data is not defined explicitly (which is the case here), a partition which such role and with label “writable” is implicitly defined at the end of the volume. This will take all the available space left aside by the reduction of the APP partition, and will contain the UC root filesystem. This will replace the UDA partition that is the last in nvidia partitioning scheme.

Now, it is time to build the gadget snap by following the repository instructions.

Building & flashing the image

Now that we have the snaps, it is time to build the image. There is not much to it, you just need an Ubuntu One account and to follow the instructions to create a key to be able to sign a model assertion. With that just follow the README.md file in the jetson-ubuntu-core repository. You can also download the latest tarball from the repository if you prefer.

The build script will generate not only a full image file, but also a tarball that will contain separate files for each partition that needs to be flashed in the device. This is needed because unfortunately there is no way we can fully flash the Jetson device with a GPT image, instead we can flash only individual partitions with the tools nvidia provides.

Once the build finishes, we can take the resulting tarball and follow the instructions to get the necessary partitions flashed. As can be read there, we have to download the nvidia L4T package. Also, note that to be able to change the partition sizes and files to flash, a couple of patches have to be applied on top of the L4T scripts.

Summary

After this, you should have a working Ubuntu Core 18 device. You can use the serial port or an external monitor to configure it with your launchpad account so you can ssh into it. Enjoy!

Read more
ssweeny

My team at work has been focused on snaps this year and one thing we’ve tried to do internally is establish a set of best practices for snap packaging software. Toward that end I’ve been working on a little tool I’m calling snaplint to encode those practices and verify that we’re following them.

Right now you can run snaplint against your snapcraft project directory
and it will scan the prime subdirectory for the following things:

  • copyright (basically that you included usr/share/doc/*copyright*) for
    any stage-packages
  • developer cruft (things like header and object files or static libs
    that might have made their way into your snap)
  • libraries (examine the ELF files in your snap and look for libraries
    which aren’t used)

The next things I’m planning on adding are:

  • checking for copyright info from apps/parts themselves.
  • checking for mixing of incompatible licenses

I would love to hear suggestions on further improvements.

You can find the source at https://github.com/ssweeny/snaplint

And, of course if you’re running Ubuntu 16.04 or later you can try it on your own machine with:
$ snap install snaplint
$ snaplint path/to/your/project

Read more
Daniel Holbach

Ubuntu 16.04 is out!

Ubuntu 16.04 – yet another LTS?

Ubuntu 16.04 LTS, aka the Xenial Xerus, has just been released. It’s incredible that it’s already the 24th Ubuntu release and the 6th LTS release. If you have been around for a while and need a blast from the past, check out this video:

Click here to view it on youtube. It’s available in /usr/share/example-content on a default desktop install as well.

You would think that after such a long time releases get somewhat inflationary and less important and while I’d very likely always say on release day “yes, this one is the best of all so far”, Ubuntu 16.04 is indeed very special to me.

Snappy snappy snappy

Among the many pieces of goodness to come to your way is the snapd package. It’s installed by default on many flavours of Ubuntu including Ubuntu Desktop and is your snappy connection to myApps.

Snappy Ubuntu Core 2.0 landing just in time for the 16.04 LTS release only happened due to the great and very hard work of many teams and individuals. I also see it as the implementation of lots of feedback we have been getting from third party app developers, ISVs and upstream projects over the years. Basically what all of them wanted was in a nutshell: a solid Ubuntu base, flexibility in handling their app and the relevant stack, independence from distro freezes, dead-simple packaging, bullet-proof upgrades and rollbacks, and an app store model established with the rise of the smartphones. Snappy Ubuntu Core is exactly that and more. What it also brings to Ubuntu is a clear isolation between apps and a universal trust model.

As most of you know, I’ve been trying to teach people how to do packaging for Ubuntu for years and it continued to improve and get easier, but all in all, it still is hard to get right. snapcraft makes this so much easier. It’s just beautiful. If you have been doing some packaging in the past, just take a look at some of the examples.

Landing a well-working and stable snapd with clear-cut and stable set of APIs was the most important aspect, especially considering that almost everyone will be basing their work on 16.04 LTS, which is going to be supported for five years. This includes being able to use snapcraft on the LTS.

Today you can build a snap, upload it to the store using snapcraft upload, having it automatically reviewed and published by the store and Desktop users can install it on their system. This brings you in a position where you can easily share your software with millions of users, without having to wait for somebody to upload it to the distro for you, without having your users add yet another PPA, etc.

So, what’s still missing? Quite a few things actually. Because you have to bundle your dependencies, packages are still quite big. This will change as soon as the specifics of OS and library snaps are figured out. Apart from that many new interfaces will need to be added to make Ubuntu Core really useful and versatile. There are also still a few bugs which need figuring out.

If you generally like what you’re reading here, come and talk to us. Introduce yourselves, talk to us and we’ll figure out if anything you need it still missing.

If you’re curious you can also check out some blog posts written by people who worked on this relentlessly in the last weeks:

Thanks a lot everyone – I thoroughly enjoyed working with you on this and I’m looking forward to all the great things we are going to deliver together!

Bring your friends, bring your questions!

The Community team moved the weekly Ubuntu Community Q&A to be tomorrow, Friday 2016-04-22 15:00 UTC on https://ubuntuonair.com as usual. If you have questions, tune in and bring your friends as well!

Read more
Daniel Holbach

ubucon

I’m very excited about UbuCon Summit which will bring many many Ubuntu people from all parts of its community together in January. David Planella did a great job explaining why this event is going to be just fantastic.

I look forward to meeting everyone and particularly look forward to what we’ve got to show in terms of Snappy Ubuntu Core.

Manik Taneja and Sergio Schvezov

We are going to have Manik Taneja and Sergio Schvezov there who are going to give the following talk:

Internet of Things gets ‘snappy’ with Ubuntu Core

Snappy Ubuntu Core is the new rendition of Ubuntu, designed from the ground up to power the next generation of IoT devices. The same Ubuntu and its vast ecosystem, but delivered in a leaner form, with state-of-the art security and reliable update mechanisms to ensure devices and apps are always up-to-date.

This talk will introduce Ubuntu Core, the technologies of its foundations and the developer experience with Snapcraft. We will also discuss how public and branded stores can kickstart a thriving app ecosystem and how Ubuntu meets the needs of connected device manufacturers, entrepreneurs and innovators.

And there’s more! Sergio Schvezov will also give the following workshop:

Hands-on demo: creating Ubuntu snaps with Snapcraft

Overview the snapcraft features and demo how easily a snap can be created using multiple parts from different sources. We will also show how to create a plugin for unhandled source types.

In addition to that we are going to have a few nice things at our booth, so we can show give you a Snappy experience there as well.

If you want to find out more, like check the entire schedule or register for the event, do it at ubucon.org.

I’m looking forward to seeing you there! </p>
            <a href=Read more

Daniel Holbach

Ubuntu Online Summit featured more than 70 sessions this time around and quite a big turnout. You can find the full schedule with links to session videos and session notes in summit.ubuntu.com.

Here’s a quick summary of what happened in Snappy Ubuntu Core land:

  • Testing Snappy: In this Show & Tell session Leo Arias showcased a lot of the QA work which has been done on Ubuntu Core along with many useful techniques to run tests and easily bring up Snappy in a number of different scenario.
  • Creating more Snappy frameworks: Frameworks are an effective way to bring functionality to Ubuntu Core which can then be shared by apps. The session attracted quite a few users of Snappy who wanted to know if their use-case could be addressed by a framework. We discussed some more technical difficulties, possible solutions and learned that bluetooth and connectivity (based on network-manager) frameworks are in the works.
  • Snappy Clinic: bringing ROS apps to Snappy Ubuntu Core: Ted Gould showed off the great work which has been put into the catkin plugin of Snapcraft. Taking a simple ROS app and bringing it to Ubuntu Core is very easy. The interest from members of the ROS community was great to see and their feedback will help us improve the support even further.
  • Snap packages for phone and desktop apps: Alejandro Cura and Kyle Fazzari brought up their analysis of snappy on the phone/desktop and discussed a plan on what would need to land to make snappy apps on the Ubuntu desktop and phone a reality.
  • Your feedback counts: the Snappy onboarding experience: This session brought together a number of different users of Snappy who shared their experience and what they would like to do. The feedback was great and will be factored into our upcoming documentation plans.
  • Snappy Developer Community Resources: In this session Thibaut Rouffineau and I had a chat about our online support options and community resources. We gathered a number of ideas and will look into creating workshop and presentation materials this cycle as well.
  • Porting popular apps/software to Snappy: Many interesting apps and appliances exist for a variety of boards, most notably the Raspberry Pi. We put together a plan on how we could start a community initiative for bringing them over to Snappy Ubuntu Core.

Thanks to everyone who participated and helped to make this such a great UOS!

Read more