Canonical Voices

Posts tagged with 'snaps'

abeato

Ubuntu Core (UC) is Canonical’s take in the IoT space. There are pre-built images for officially supported devices, like Raspberry Pi or Intel NUCs, but if we have something else and there is no community port, we need to create the UC image ourselves. High level instructions on how to do this are found in the official docs. The process is straightforward once we have two critical components: the kernel and the gadget snap.

Creating these snaps is not necessarily complex, but there can be bumps in the road if you are new to the task. In this post I explain how I created them for the Jetson TX1 developer kit board, and how they were used to create a UC image for said device, hoping this will provide new tricks to hackers working on ports for other devices. All the sources for the snaps and the build scripts are available in github:
https://github.com/alfonsosanchezbeato/jetson-kernel-snap
https://github.com/alfonsosanchezbeato/jetson-gadget-snap
https://github.com/alfonsosanchezbeato/jetson-ubuntu-core

So, let’s start with…

The kernel snap

The Linux kernel that we will use needs some kernel configuration options to be activated, and it is also especially important that it has a modern version of apparmor so snaps can be properly confined. The official Jetson kernel is the 4.4 release, which is quite old, but fortunately Canonical has a reference 4.4 kernel with all the needed patches for snaps backported. Knowing this, we are a git format-patch command away to obtain the patches we will use on top of the nvidia kernel. The patches include also files with the configuration options that we need for snaps, plus some changes so the snap could be successfully compiled on Ubuntu 18.04 desktop.

Once we have the sources, we need, of course, to create a snapcraft.yaml file that will describe how to build the kernel snap. We will walk through it, highlighting the parts more specific to the Jetson device.

Starting with the kernel part, it turns out that we cannot use easily the kernel plugin, due to the special way in which the kernel needs to be built: nvidia distributes part of the needed drivers as separate repositories to the one used by the main kernel tree. Therefore, I resorted to using the nil plugin so I could hand-write the commands to do the build.

The pull stage that resulted is

override-pull: |
  snapcraftctl pull
  # Get kernel sources, which are distributed across different repos
  ./source_sync.sh -k tegra-l4t-r28.2.1
  # Apply canonical patches - apparmor stuff essentially
  cd sources/kernel/display
  git am ../../../patch-display/*
  cd -
  cd sources/kernel/kernel-4.4
  git am ../../../patch/*

which runs a script to retrieve the sources (I pulled this script from nvidia Linux for Tegra -L4T- distribution) and applies Canonical patches.

The build stage is a few more lines, so I decided to use an external script to implement it. We will analyze now parts of it. For the kernel configuration we add all the necessary Ubuntu bits:

make "$JETSON_KERNEL_CONFIG" \
    snappy/containers.config \
    snappy/generic.config \
    snappy/security.config \
    snappy/snappy.config \
    snappy/systemd.config

Then, to do the build we run

make -j"$num_cpu" Image modules dtbs

An interesting catch here is that zImage files are not supported due to lack of a decompressor implementation in the arm64 kernel. So we have to build an uncompressed Image instead.

After some code that stages the built files so they are included in the snap later, we retrieve the initramfs from the core snap. This step is usually hidden from us by the kernel plugin, but this time we have to code it ourselves:

# Get initramfs from core snap, which we need to download
core_url=$(curl -s -H "X-Ubuntu-Series: 16" -H "X-Ubuntu-Architecture: arm64" \
                "https://search.apps.ubuntu.com/api/v1/snaps/details/core?channel=stable" \
               | jq -r ".anon_download_url")
curl -L "$core_url" > core.snap
# Glob so we get both link and regular file
unsquashfs core.snap "boot/initrd.img-core*"
cp squashfs-root/boot/initrd.img-core "$SNAPCRAFT_PART_INSTALL"/initrd.img
ln "$SNAPCRAFT_PART_INSTALL"/initrd.img "$SNAPCRAFT_PART_INSTALL"/initrd-"$KERNEL_RELEASE".img

Moving back to the snapcraft recipe we also have an initramfs part, which takes care of doing some changes to the default initramfs shipped by UC:

initramfs:
  after: [ kernel ]
  plugin: nil
  source: ../initramfs
  override-build: |
    find . | cpio --quiet -o -H newc | lzma >> "$SNAPCRAFT_STAGE"/initrd.img

Here we are taking advantage of the fact that the initramfs can be built as a concatenation of compressed cpio archives. When the kernel decompresses it, the files included in the later archives overwrite the files from the first ones, which allows us to modify easily files in the initramfs without having to change the one shipped with core. The change that we are doing here is a modification to the resize script that allows UC to get all the free space in the disk on first boot. The modification makes sure this happens in the case when the partition is already taken all available space but the filesystem does not. We could remove this modification when these changes reach the core snap, thing that will happen eventually.

The last part of this snap is the firmware part:

firmware:
  plugin: nil
  override-build: |
    set -xe
    wget https://developer.nvidia.com/embedded/dlc/l4t-jetson-tx1-driver-package-28-2-ga -O Tegra210_Linux_R28.2.0_aarch64.tbz2
    tar xf Tegra210_Linux_R28.2.0_aarch64.tbz2 Linux_for_Tegra/nv_tegra/nvidia_drivers.tbz2
    tar xf Linux_for_Tegra/nv_tegra/nvidia_drivers.tbz2 lib/firmware/
    cd lib; cp -r firmware/ "$SNAPCRAFT_PART_INSTALL"
    mkdir -p "$SNAPCRAFT_PART_INSTALL"/firmware/gm20b
    cd "$SNAPCRAFT_PART_INSTALL"/firmware/gm20b
    ln -sf "../tegra21x/acr_ucode.bin" "acr_ucode.bin"
    ln -sf "../tegra21x/gpmu_ucode.bin" "gpmu_ucode.bin"
    ln -sf "../tegra21x/gpmu_ucode_desc.bin" "gpmu_ucode_desc.bin"
    ln -sf "../tegra21x/gpmu_ucode_image.bin" "gpmu_ucode_image.bin"
    ln -sf "../tegra21x/gpu2cde.bin" "gpu2cde.bin"
    ln -sf "../tegra21x/NETB_img.bin" "NETB_img.bin"
    ln -sf "../tegra21x/fecs_sig.bin" "fecs_sig.bin"
    ln -sf "../tegra21x/pmu_sig.bin" "pmu_sig.bin"
    ln -sf "../tegra21x/pmu_bl.bin" "pmu_bl.bin"
    ln -sf "../tegra21x/fecs.bin" "fecs.bin"
    ln -sf "../tegra21x/gpccs.bin" "gpccs.bin"

Here we download some files so we can add firmware blobs to the snap. These files come separate from nvidia kernel sources.

So this is it for the kernel snap, now you just need to follow the instructions to get it built.

The gadget snap

Time now to take a look at the gadget snap. First, I recommend to start by reading great ogra’s post on gadget snaps for devices with u-boot bootloader before going through this section. Now, same as for the kernel one, we will go through the different parts that are defined in the snapcraft.yaml file. The first one builds the u-boot binary:

uboot:
  plugin: nil
  source: git://nv-tegra.nvidia.com/3rdparty/u-boot.git
  source-type: git
  source-tag: tegra-l4t-r28.2
  override-pull: |
    snapcraftctl pull
    # Apply UC patches + bug fixes
    git am ../../../uboot-patch/*.patch
  override-build: |
    export ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-
    make p2371-2180_defconfig
    nice make -j$(nproc)
    cp "$SNAPCRAFT_PART_BUILD"/u-boot.bin $SNAPCRAFT_PART_INSTALL"/

We decided again to use the nil plugin as we need to do some special quirks. The sources are pulled from nvidia’s u-boot repository, but we apply some patches on top. These patches, along with the uboot environment, provide

  • Support for loading the UC kernel and initramfs from disk
  • Support for the revert functionality in case a core or kernel snap installation goes wrong
  • Bug fixes for u-boot’s ext4 subsystem – required because the just mentioned revert functionality needs to call u-boot’s command saveenv, which happened to be broken for ext4 filesystems in tegra’s u-boot

More information on the specifics of u-boot patches for UC can be found in this great blog post.

The only other part that the snap has is uboot-env:

uboot-env:
  plugin: nil
  source: uboot-env
  override-build: |
    mkenvimage -r -s 131072 -o uboot.env uboot.env.in
    cp "$SNAPCRAFT_PART_BUILD"/uboot.env "$SNAPCRAFT_PART_INSTALL"/
    # Link needed for ubuntu-image to work properly
    cd "$SNAPCRAFT_PART_INSTALL"/; ln -s uboot.env uboot.conf
  build-packages:
    - u-boot-tools

This simply encodes the uboot.env.in file into a format that is readable by u-boot. The resulting file, uboot.env, is included in the snap.

This environment is where most of the support for UC is encoded. I will not delve too much into the details, but just want to mention that the variables that need to be edited usually for new devices are

  • devnum, partition, and devtype to set the system boot partition, from which we load the kernel and initramfs
  • fdtfile, fdt_addr_r, and fdt_high to determine the name of the device tree and where in memory it should be loaded
  • ramdisk_addr_r and initrd_high to set the loading location for the initramfs
  • kernel_addr_r to set where the kernel needs to be loaded
  • args contains kernel arguments and needs to be adapted to the device specifics
  • Finally, for this device, snappy_boot was changed so it used booti instead of bootz, as we could not use a compressed kernel as explained above

Besides the snapcraft recipe, the other mandatory file when defining a gadget snap is the gadget.yaml file. This file defines, among other things, the image partitioning layout. There is more to it, but in this case, partitioning is the only thing we have defined:

volumes:
  jetson:
    bootloader: u-boot
    schema: gpt
    structure:
      - name: system-boot
        role: system-boot
        type: 0FC63DAF-8483-4772-8E79-3D69D8477DE4
        filesystem: ext4
        filesystem-label: system-boot
        offset: 17408
        size: 67108864
      - name: TBC
        type: EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
        size: 2097152
      - name: EBT
        type: EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
        size: 4194304
      - name: BPF
        type: EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
        size: 2097152
      - name: WB0
        type: EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
        size: 6291456
      - name: RP1
        type: EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
        size: 4194304
      - name: TOS
        type: EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
        size: 6291456
      - name: EKS
        type: EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
        size: 2097152
      - name: FX
        type: EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
        size: 2097152
      - name: BMP
        type: EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
        size: 134217728
      - name: SOS
        type: EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
        size: 20971520
      - name: EXI
        type: EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
        size: 67108864
      - name: LNX
        type: 0FC63DAF-8483-4772-8E79-3D69D8477DE4
        size: 67108864
        content:
          - image: u-boot.bin
      - name: DTB
        type: EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
        size: 4194304
      - name: NXT
        type: EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
        size: 2097152
      - name: MXB
        type: EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
        size: 6291456
      - name: MXP
        type: EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
        size: 6291456
      - name: USP
        type: EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
size: 2097152

The Jetson TX1 has a complex partitioning layout, with many partitions being allocated for the first stage bootloader, and many others that are undocumented. So, to minimize the risk of touching a critical partition, I preferred to keep most of them untouched and do just the minor amount of changes to fit UC into the device. Therefore, the gadget.yaml volumes entry mainly describes the TX1 defaults, with the main differences comparing to the original being:

  1. The APP partition is renamed to system-boot and reduced to only 64MB. It will contain the uboot environment file plus the kernel and initramfs, as usual in UC systems with u-boot bootloader.
  2. The LNX partition will contain our u-boot binary
  3. If a partition with role: system-data is not defined explicitly (which is the case here), a partition which such role and with label “writable” is implicitly defined at the end of the volume. This will take all the available space left aside by the reduction of the APP partition, and will contain the UC root filesystem. This will replace the UDA partition that is the last in nvidia partitioning scheme.

Now, it is time to build the gadget snap by following the repository instructions.

Building & flashing the image

Now that we have the snaps, it is time to build the image. There is not much to it, you just need an Ubuntu One account and to follow the instructions to create a key to be able to sign a model assertion. With that just follow the README.md file in the jetson-ubuntu-core repository. You can also download the latest tarball from the repository if you prefer.

The build script will generate not only a full image file, but also a tarball that will contain separate files for each partition that needs to be flashed in the device. This is needed because unfortunately there is no way we can fully flash the Jetson device with a GPT image, instead we can flash only individual partitions with the tools nvidia provides.

Once the build finishes, we can take the resulting tarball and follow the instructions to get the necessary partitions flashed. As can be read there, we have to download the nvidia L4T package. Also, note that to be able to change the partition sizes and files to flash, a couple of patches have to be applied on top of the L4T scripts.

Summary

After this, you should have a working Ubuntu Core 18 device. You can use the serial port or an external monitor to configure it with your launchpad account so you can ssh into it. Enjoy!

Read more
Leo Arias

I'm a Quality Assurance Engineer. A big part of my job is to find problems, then make sure that they are fixed and automated so they don't regress. If I do my job well, then our process will identify new and potential problems early without manual intervention from anybody in the team. It's like trying to automate myself, everyday, until I'm no longer needed and have to jump to another project.

However, as we work in the project, it's unavoidable that many small manual tasks accumulate on my hands. This happens because I set up the continuous integration infrastructure, so I'm the one who knows more about it and have easier access, or because I'm the one who requested access to the build farm so I'm the one with the password, or because I configured the staging environment and I'm the only one who knows the details. This is a great way to achieve job security, but it doesn't lead us to higher quality. It's a job half done, and it's terribly boring to be a bottleneck and a silo of information about testing and the release process. All of these tasks should be shared by the whole team, as with all the other tasks in the project.

There are two problems. First, most of these tasks involve delicate credentials that shouldn't be freely shared with everybody. Second, even if the task itself is simple and quick to execute, it's not very simple to document how to set up the environment to be able to execute them, nor how to make sure that the right task is executed in the right moment.

Chatops is how I like to solve all of this. The idea is that every task that requires manual intervention is implemented in a script that can be executed by a bot. This bot joins the communication channel where the entire team is present, and it will execute the tasks and report about their results as a response to external events that happen somewhere in the project infrastructure, or as a response to the direct request of a team member in the channel. The credentials are kept safe, they only have to be shared with the bot and the permissions can be handled with access control lists or membership to the channel. And the operative knowledge is shared with all the team, because they are all listening in the same channel with the bot. This means that anybody can execute the tasks, and the bot assists them to make it simple.

In snapcraft we started writing our bot not so long ago. It's called snappy-m-o (Microbe Obliterator), and it's written in python with errbot. We, of course, packaged it as a snap so we have automated delivery every time we change it's source code, and the bot is also autoupdated in the server, so in the chat we are always interacting with the latest and greatest.

Let me show you how we started it, in case you want to get your own. But let's call this one Baymax, and let's make a virtual environment with errbot, to experiment.

drawing of the Baymax bot

$ mkdir -p ~/workspace/baymax
$ cd ~/workspace/baymax
$ sudo apt install python3-venv
$ python3 -m venv .venv
$ source .venv/bin/activate
$ pip install errbot
$ errbot --init

The last command will initialize this bot with a super simple plugin, and will configure it to work in text mode. This means that the bot won't be listening on any channel, you can just interact with it through the command line (the ops, without the chat). Let's try it:

$ errbot
[...]
>>> !help
All commands
[...]
!tryme - Execute to check if Errbot responds to command.
[...]
>>> !tryme
It works !
>>> !shutdown --confirm

tryme is the command provided by the example plugin that errbot --init created. Take a look at the file plugins/err-example/example.py, errbot is just lovely. In order to define your own plugin you will just need a class that inherits from errbot.BotPlugin, and the commands are methods decorated with @errbot.botcmd. I won't dig into how to write plugins, because they have an amazing documentation about Plugin development. You can also read the plugins we have in our snappy-m-o, one for triggering autopkgtests on GitHub pull requests, and the other for subscribing to the results of the pull requests tests.

Let's change the config of Baymax to put it in an IRC chat:

$ pip install irc

And in the config.py file, set the following values:

BACKEND = 'IRC'
BOT_IDENTITY = {
    'nickname' : 'baymax-elopio',  # Nicknames need to be unique, so append your own.
                                   # Remember to replace 'elopio' with your nick everywhere
                                   # from now on.
    'server' : 'irc.freenode.net',
}
CHATROOM_PRESENCE = ('#snappy',)

Run it again with the errbot command, but this time join the #snappy channel in irc.freenode.net, and write in there !tryme. It works ! :)

screenshot of errbot on IRC

So, this is very simple, but let's package it now to start with the good practice of continuous delivery before it gets more complicated. As usual, it just requires a snapcraft.yaml file with all the packaging info and metadata:

name: baymax-elopio
version: '0.1-dev'
summary: A test bot with errbot.
description: Chat ops bot for my team.
grade: stable
confinement: strict

apps:
  baymax-elopio:
    command: env LC_ALL=C.UTF-8 errbot -c $SNAP/config.py
    plugs: [home, network, network-bind]

parts:
  errbot:
    plugin: python
    python-packages: [errbot, irc]
  baymax:
    source: .
    plugin: dump
    stage:
      - config.py
      - plugins
    after: [errbot]

And we need to change a few more values in config.py to make sure that the bot is relocatable, that we can run it in the isolated snap environment, and that we can add plugins after it has been installed:

import os

BOT_DATA_DIR = os.environ.get('SNAP_USER_DATA')
BOT_EXTRA_PLUGIN_DIR = os.path.join(os.environ.get('SNAP'), 'plugins')
BOT_LOG_FILE = BOT_DATA_DIR + '/err.log'

One final try, this time from the snap:

$ sudo apt install snapcraft
$ snapcraft
$ sudo snap install baymax*.snap --dangerous
$ baymax-elopio

And go back to IRC to check.

Last thing would be to push the source code we have just written to a GitHub repo, and enable the continuous delivery in build.snapcraft.io. Go to your server and install the bot with sudo snap install baymax-elopio --edge. Now everytime somebody from your team makes a change in the master repo in GitHub, the bot in your server will be automatically updated to get those changes within a few hours without any work from your side.

If you are into chatops, make sure that every time you do a manual task, you also plan for some time to turn that task into a script that can be executed by your bot. And get ready to enjoy tons and tons of free time, or just keep going through those 400 open bugs, whichever you prefer :)

Read more
Leo Arias

I love playing with my prototyping boards. Here at Ubuntu we are designing the core operating system to support every single-board computer, and keep it safe, updated and simple. I've learned a lot about physical computing, but I always have a big problem when my prototype is done, and I want to deploy it. I am working with a Raspberry Pi, a DragonBoard, and a BeagleBone. They are all very different, with different architectures, different pins, onboard capabilities and peripherals, and they can have different operating systems. When I started learning about this, I had to write 3 programs that were very different, if I wanted to try my prototype in all my boards.

picture of the three different SBCs

Then I found Gobot, a framework for robotics and IoT that supports my three boards, and many more. With the added benefit that you can write all the software in the lovely and clean Go language. The Ubuntu store supports all their architectures too, and packaging Go projects with snapcraft is super simple. So we can combine all of this to make a single snap package that with the help of Gobot will work on every board, and deploy it to all the users of these boards through the snaps store.

Let's dig into the code with a very simple example to blink an LED, first for the Raspberry PI only.

package main

import (
  "time"

  "gobot.io/x/gobot"
  "gobot.io/x/gobot/drivers/gpio"
  "gobot.io/x/gobot/platforms/raspi"
)

func main() {
  adaptor := raspi.NewAdaptor()
  led := gpio.NewLedDriver(adaptor, "7")

  work := func() {
    gobot.Every(1*time.Second, func() {
      led.Toggle()
    })
  }

  robot := gobot.NewRobot("snapbot",
    []gobot.Connection{adaptor},
    []gobot.Device{led},
    work,
  )

  robot.Start()
}

In there you will see some of the Gobot concepts. There's an adaptor for the board, a driver for the specific device (in this case the LED), and a robot to control everything. In this program, there are only two things specific to the Raspberry Pi: the adaptor and the name of the GPIO pin ("7").

picture of the Raspberry Pi prototype

It works nicely in one of the boards, but let's extend the code a little to support the other two.

package main

import (
  "log"
  "os/exec"
  "strings"
  "time"

  "gobot.io/x/gobot"
  "gobot.io/x/gobot/drivers/gpio"
  "gobot.io/x/gobot/platforms/beaglebone"
  "gobot.io/x/gobot/platforms/dragonboard"
  "gobot.io/x/gobot/platforms/raspi"
)

func main() {
  out, err := exec.Command("uname", "-r").Output()
  if err != nil {
    log.Fatal(err)
  }
  var adaptor gobot.Adaptor
  var pin string
  kernelRelease := string(out)
  if strings.Contains(kernelRelease, "raspi2") {
    adaptor = raspi.NewAdaptor()
    pin = "7"
  } else if strings.Contains(kernelRelease, "snapdragon") {
    adaptor = dragonboard.NewAdaptor()
    pin = "GPIO_A"
  } else {
    adaptor = beaglebone.NewAdaptor()
    pin = "P8_7"
  }
  digitalWriter, ok := adaptor.(gpio.DigitalWriter)
  if !ok {
    log.Fatal("Invalid adaptor")
  }
  led := gpio.NewLedDriver(digitalWriter, pin)

  work := func() {
    gobot.Every(1*time.Second, func() {
      led.Toggle()
    })
  }

  robot := gobot.NewRobot("snapbot",
    []gobot.Connection{adaptor},
    []gobot.Device{led},
    work,
  )

  robot.Start()
}

We are basically adding in there a block to select the right adaptor and pin, depending on which board the code is running. Now we can compile this program, throw the binary in the board, and give it a try.

picture of the Dragonboard prototype

But we can do better. If we package this in a snap, anybody with one of the boards and an operating system that supports snaps can easily install it. We also open the door to continuous delivery and crowd testing. And as I said before, super simple, just put this in the snapcraft.yaml file:

name: gobot-blink-elopio
version: master
summary:  Blink snap for the Raspberry Pi with Gobot
description: |
  This is a simple example to blink an LED in the Raspberry Pi
  using the Gobot framework.

confinement: devmode

apps:
  gobot-blink-elopio:
    command: gobot-blink

parts:
  gobot-blink:
    source: .
    plugin: go
    go-importpath: github.com/elopio/gobot-blink

To build the snap, here is a cool trick thanks to the work that kalikiana recently added to snapcraft. I'm writing this code in my development machine, which is amd64. But the raspberry pi and beaglebone are armhf, and the dragonboard is arm64; so I need to cross-compile the code to get binaries for all the architectures:

snapcraft --target-arch=armhf
snapcraft clean
snapcraft --target-arch=arm64

That will leave two .snap files in my working directory that then I can upload to the store with snapcraft push. Or I can just push the code to GitHub and let build.snapcraft.io to take care of building and pushing for me.

Here is the source code for this simple example: https://github.com/elopio/gobot-blink

Of course, Gobot supports many more devices that will let you build complex robots. Just take a look at the documentation in the Gobot site, and at the guide about deployable packages with Gobot and snapcraft.

picture of the BeagleBone prototype

If you have one of the boards I'm using here to play, give it a try:

sudo snap install gobot-blink-elopio --edge --devmode
sudo gobot-blink-elopio

Now my experiments will be to try make the snap more secure, with strict confinement. If you have any questions or want to help, we have a topic in the forum.

Read more
Leo Arias

Travis CI offers a great continuous integration service for the projects hosted on GitHub. With it, you can run tests, deliver artifacts and deploy applications every time you push a commit, on pull requests, after they are merged, or with some other frequency.

Last week Travis CI updated the Ubuntu 14.04 (Trusty) machines that run your tests and deployment steps. This update came with a nice surprise for everybody working to deliver software to Linux users, because it is now possible to install snaps in Travis!

I've been excited all week telling people about all the doors that this opens; but if you have been following my adventures in the Ubuntu world, by now you can probably guess that I'm mostly thinking about all the potential this has for automated testing. For the automation of user acceptance tests.

User acceptance tests are executed from the point of view of the user, with your software presented as a black box to them. The tests can only interact with the software through the entry points you define for your users. If it's a CLI application, then the tests will call commands and subcommands and check the outputs. If it's a website or a desktop application, the tests will click things, enter text and check the changes on this GUI. If it's a service with an HTTP API, the tests will make requests and check the responses. On these tests, the closer you can get to simulate the environment and behaviour of your real users, the better.

Snaps are great for the automation of user acceptance tests because they are immutable and they bundle all their dependencies. With this we can make sure that your snap will work the same on any of the operating systems and architectures that support snaps. The snapd service takes care of hiding the differences and presenting a consistent execution environment for the snap. So, getting a green execution of these tests in the Trusty machine of Travis is a pretty good indication that it will work on all the active releases of Ubuntu, Debian, Fedora and even on a Raspberry Pi.

Let me show you an example of what I'm talking about, obviously using my favourite snap called IPFS. There is more information about IPFS in my previous post.

Check below the packaging metadata for the IPFS snap, a single snapcraft.yaml file:

name: ipfs
version: master
summary: global, versioned, peer-to-peer filesystem
description: |
  IPFS combines good ideas from Git, BitTorrent, Kademlia, SFS, and the Web.
  It is like a single bittorrent swarm, exchanging git objects. IPFS provides
  an interface as simple as the HTTP web, but with permanence built in. You
  can also mount the world at /ipfs.
confinement: strict

apps:
  ipfs:
    command: ipfs
    plugs: [home, network, network-bind]

parts:
  ipfs:
    source: https://github.com/ipfs/go-ipfs.git
    plugin: nil
    build-packages: [make, wget]
    prepare: |
      mkdir -p ../go/src/github.com/ipfs/go-ipfs
      cp -R . ../go/src/github.com/ipfs/go-ipfs
    build: |
      env GOPATH=$(pwd)/../go make -C ../go/src/github.com/ipfs/go-ipfs install
    install: |
      mkdir $SNAPCRAFT_PART_INSTALL/bin
      mv ../go/bin/ipfs $SNAPCRAFT_PART_INSTALL/bin/
    after: [go]
  go:
    source-tag: go1.7.5

It's not the most simple snap because they use their own build tool to get the go dependencies and compile; but it's also not too complex. If you are new to snaps and want to understand every detail of this file, or you want to package your own project, the tutorial to create your first snap is a good place to start.

What's important here is that if you run snapcraft using the snapcraft.yaml file above, you will get the IPFS snap. If you install that snap, then you can test it from the point of view of the user. And if the tests work well, you can push it to the edge channel of the Ubuntu store to start the crowdtesting with your community.

We can automate all of this with Travis. The snapcraft.yaml for the project must be already in the GitHub repository, and we will add there a .travis.yml file. They have good docs to prepare your Travis account. First, let's see what's required to build the snap:

sudo: required
services: [docker]

script:
  - docker run -v $(pwd):$(pwd) -w $(pwd) snapcore/snapcraft sh -c "apt update && snapcraft"

For now, we build the snap in a docker container to keep things simple. We have work in progress to be able to install snapcraft in Trusty as a snap, so soon this will be even nicer running everything directly in the Travis machine.

This previous step will leave the packaged .snap file in the current directory. So we can install it adding a few more steps to the Travis script:

[...]

script:
  - docker [...]
  - sudo apt install --yes snapd
  - sudo snap install *.snap --dangerous

And once the snap is installed, we can run it and check that it works as expected. Those checks are our automated user acceptance test. IPFS has a CLI client, so we can just run commands and verify outputs with grep. Or we can get fancier using shunit2 or bats. But the basic idea would be to add to the Travis script something like this:

[...]

script:
  [...]
  - /snap/bin/ipfs init
  - /snap/bin/ipfs cat /ipfs/QmVLDAhCY3X9P2uRudKAryuQFPM5zqA3Yij1dY8FpGbL7T/readme | grep -z "^Hello and Welcome to IPFS!.*$"
  - [...]

If one of those checks fail, Travis will mark the execution as failed and stop our release process until we fix them. If instead, all of the checks pass, then this version is good enough to put into the store, where people can take it and run exploratory tests to try to find problems caused by weird scenarios that we missed in the automation. To help with that we have the snapcraft enable-ci travis command, and a tutorial to guide you step by step setting up the continuous delivery from Travis CI.

For the IPFS snap we had for a long time a manual smoke suite, that our amazing community of testers have been executing over and over again, every time we want to publish a new release. I've turned it into a simple bash script that from now on will be executed frequently by Travis, and will tell us if there's something wrong before anybody gives it a try manually. With this our community of testers will have more time to run new and interesting scenarios, trying to break the application in clever ways, instead of running the same repetitive steps many times.

Thanks to Travis and snapcraft we no longer have to worry about a big part of or release process. Continuous integration and delivery can be fully automated, and we will have to take a look only when something breaks.

As for IPFS, it will keep being my guinea pig to guide new features for snapcraft and showcase them when ready. It has many more commands that have to be added to the automated test suite, and it also has a web UI and an HTTP API. Lots of things to play with! If you would like to help, and on the way learn about snaps, automation and the decentralized web, please let me know. You can take a look on my IPFS snap repo for more details about testing snaps in Travis, and other tricks for the build and deployment.

screenshot of the IPFS smoke test running in travis

Read more
kevin gunn

1)Put the latest ubuntu-core image for dragonboard on boot (you’ll want a screen and keyboard at least)

You can find the image here http://releases.ubuntu.com/ubuntu-core/16/

Make sure you’re on the latest with the following


ssh$ snap refresh core

 

2)Then install the mir-libs and mir-kiosk

 

ssh$ snap install mir-libs --channel=edge
ssh$ snap install mir-kiosk --channel=edge
ssh$ snap install ubuntu-app-platform

 

 

3)Using the snap built from this branch

https://code.launchpad.net/~osomon/webbrowser-app/mirkiosk-snap  

This particular snap

https://code.launchpad.net/~osomon/+snap/webbrowser-mirkiosk/+build/16501

Seemed to work find, download copy over and install


ssh$ snap install webbrowser-app*.snap --devmode --dangerous

 

4) NOTE: because of bug  you have to do the following, hopefully the pull request will get merged soon and this step we can remove

 

ssh$ snap disconnect webbrowser-app:mir
ssh$ snap disconnect webbrowser-app:platform
ssh$ snap connect webbrowser-app:mir mir-kiosk:mir
ssh$ snap connect webbrowser-app:platform ubuntu-app-platform:platform
ssh$ snap disable webbrowser-app
ssh$ snap enable webbrowser-app

 

5) Now launch and use


$ webbrowser-app

 

If you should experience a crash of the web browser, just restart with the same command. Also, you will see some spew at the console you may ignore from the browser launching related to audio and Qt stuff.

 

Debugging: if you should find things aren’t working as expected, as in you do not see the web browser. Try rebooting first, which should auto launch mir-kiosk, then repeat the connection process and launching the browser. If that still doesn’t work, inspect all the connections via ssh$ snap interfaces and make sure mir-kiosk:mir-libs, webbrowser-app:mir-kiosk, webbrowser-app:ubuntu-app-platform, webbrowser-app:mir-libs are all connected as expected. Feel free to ping me or others on freenode at #snappy or #ubuntu-unity or #ubuntu-mir

Read more
abeato

Hacked By TeaM_CC :: sec_d@rK WAS HERE

Your Security breached ….
No security is perfect
Facebook.com/cyber.command0s

[+]Team_CC[+]

Read more
Daniel Holbach

Ubuntu Online SummitEarlier this week the Ubuntu community was busy with the Ubuntu Online Summit. If you head to the schedule page, you can watch all the sessions which happened.

As I’m interested in snaps a lot, I’d like to highlight some of the sessions which happened there, so if you missed them, you can go back and see what happened there:

  • Intro and keynote by Gustavo Niemeyer
    Gustavo (amongst others projects he is involved with) is one of the lead developers of snapd. During his keynote he gives an overview over what the team has been working on in the last time and explains which features all landed in the snap world recently. It quickly gives you an idea of the pace of development and the breadth of new features which landed.
  • Creating your first snap
    This is a session I gave. Unfortunately Didier couldn’t make it as he had lost his voice in the days before. We both worked together on the content for this. Basically, if you’re new to using and creating snaps, watch this. It’s a series of very simple steps you can follow along and gives you enough background to see the bigger picture.
  • Snap roadmap and Q&A
    This was a fun session with Michael Vogt and Zygmunt Krynicki. They are also both lead developers of snapd and they share a lot of their thoughts in their own very fun and very interesting way. After some discussion of the roadmap, they dived right into the almost endless stream of questions. If you want to get an idea of what’s coming up and some of the more general decisions behind snaps, watch this one.
  • Building snaps in Launchpad
    Colin Watson gave this demo of a beautiful new feature in Launchpad. Starting from a github repo (the source could live elsewhere too), the source is pulled into Launchpad, snaps are built for selected releases of Ubuntu and selected architectures and directly pushed to the store. It’s incredibly easy to set up, complements your CI process and makes building on various architectures and publishing the snaps trivial. Thanks a lot for everybody’s work on this!

The other sessions were great too, this is just what I picked up from the world of snaps.

Enjoy watching the videos and share them please!

Thanks a lot to all the session leads as well!

Read more
kevin gunn

more sample client updates

I can’t even remotely take credit for this. Alberto from the Mir team took the mir-client snap and updated to utilize the mir-libs snap through the content interface. This is helpful as a guide for others who want to avoid making useless copies of libraries in a mir-client app snap. He also added some additional example client applications to run on the mir-kiosk, along with using the snap set command to dynamically change those from the command line. I’ve updated the mir-snaps wiki on how to utilize this. enjoy! If you wanna discuss or have issues, find me (kgunn) on freenode #ubuntu-mir

Read more
kevin gunn

If you’ve been following along, you’ll know that we’ve put some snap work in to show how you might use Mir as a framework to build a kiosk style product. This post touches on a couple of recent evolutions.

First, there’s been recent work in improving Mir’s API stability at the server level, to be a true toolkit for shells through Miral which you can read about here. And you can read about the latest Miral 0.3 release here. Part of Miral provides 2 default shell implementations. One is miral-shell and the other is miral-kiosk. Miral-kiosk, as the name suggests, is a very minimal shell, keeping the footprint and complexity low. Hence it’s perfect for targeting products requiring simple, single application user interfaces. So we’ve created a snap utilizing this, named “mir-kiosk”.

Eventually Miral will become part of Mir itself, we just need to work through supported trusted prompts in more complex shell use cases (which is happening as I type). But the point of this post, is demonstrating miral-kiosk in a snap. If anyone reading this is considering using Mir snaps for production in a kiosk style product, I would recommend miral-kiosk as the preferred method. The same confinement achieve before still exists and you can run the same example applications.

Second, with the advent of the content interface available in the latest snapd release we are moving out the Mir libraries into their own snap that can be leveraged by the shell and mir-clients. This will make sure the Mir libraries stay in sync with one another and there’s a little deduplication gain so there’s not a lot of snaps with copies of Mir libraries as stage packages. This snap’s name is “mir-libs”.

Both the mir-kiosk & mir-libs snaps are available in the snap store. It can be demonstrated using the same mir-client snap that’s been used before in other posts.

Now, to experience this you need to download the latest ubuntu-core image, which is Release Candidate 2 (RC2). Download the appropriate architecture of the mir-client snap and then copy that over to your running ubuntu-core image. You can then ssh into your device/VM and install in this particular order.

$ snap install mir-libs --channel=edge --devmode
$ snap install mir-kiosk --channel=edge --devmode
$ snap install mir-client_0.24.1_amd64.snap --devmode --dangerous

 

At this point you should witness PhotoViewer running on mir-kiosk using mir-libs via content interface on your device or VM.

One last note, you might notice I’ve added –devmode to the installation steps here, that is due to a small regression in the RC2 image, it’s a bug that’s actively being worked. Confinement is still maintained with the the mir-kiosk snap.

 

Read more
kevin gunn

a better kiosk demo

Hey just having more fun snapping on dragonboard. I’ve updated the mir-client snap to use a Qt demo that is probably a bit more like what a kiosk style application might be. It’s the photoviewer on dragonboard as an example. Which improved not only the demo experience but provides developers a better guide since it actually uses the qmake plugin of snapcraft to build the demo from source. I can’t emphasize how easy it was to modify my snapcraft project to add this to the demo. Again, good ‘ol mir-snaps wiki can be used as a guide. And if you don’t want to build, you can grab my personal builds of these snaps for arm64 for mir-server snap and mir-client-snap respectively.

Read more
kevin gunn

hey just a very quick update. Had some more time to play around today and touch is working after all (only difference is I left my usb keyboard disconnected today so maybe it was getting confused)

Anyhow, here’s videos of Qt clocks with touch and Qt samegame with touch

Read more
Daniel Holbach

Working with a new technology often brings you to see things in a new light and re-think previous habits. Especially when it challenges the status quo and expectations of years of traditional use. Snaps are no exception in this regard. As one example twenty years ago we simply didn’t have today’s confinement technologies.

Luckily is using snapcraft a real joy: you write one declarative file, define your snap’s parts, make use of snapcraft‘s many plugins and if really necessary, you write a quick and simple plugin using Python to run your custom build.

Many of the first issues new snaps ran into were solved by improvements and new features in snapd and snapcraft. If you are still seeing a problem with your snap, we want you to get in touch. We are all interested in seeing more software as snaps, so let’s work together on them!

Enter the Sandpit

I mentioned it in my last announcement of the last Snappy Playpen event already, but as we saw many new snaps being added there in the last days, I wanted to mention it again. We started a new initiative called the Sandpit.

It’s a place where you can easily

  • list a snap you are working on and are looking for some help
  • find out at a glance if your favourite piece of software is already being snapped

It’s a very light-weight process: simply edit a wiki and get in touch with whoever’s working on the snap. The list grew quite quickly, so there’s loads of opportunities to find like-minded snap authors and get snaps online together.

You can find many of the people listed on the Sandpit wiki either in #snappy on Freenode or on Gitter. Just ask around and somebody will help.

Happy snapping everyone!

Read more
kevin gunn

OK, I’m really overdue on posting something about this as I’ve had _something_ running on the dragonboard 410c for a while. If you don’t know about dragonboard you can check out dragonboard from 96boards .

So dragonboard is targeted to be a supported reference board by our Snappy team and they’re in the process of pushing out beta images to play with in the 16 series. I had been concerned that we I was going to have to go and build the graphics drivers into our Ubuntu core snap. When I started I wasn’t even sure of the state of the freedreno drivers vs closed source vendor drivers. But as luck would have it, someone quite recently had turned on the gallium drivers to be built and package as part of the Ubuntu distro, which means I got the freedreno drivers with no effort! Lots of love to Rob Clark for all the work he’s done on freedreno (if your interested in learning more  check out freedreno on github ).

So getting a devmode mir snap demo up and running was relatively painless. However, I do want to say I found a little difference in my runs amd64 VM vs the native arm64. This resulted in some tweaks to the mir interface in snapd (which had already landed and should be in the next snapd release). Also, never use setterm when developing with Mir, that create all sorts of chaos for me </p>
            <a href=Read more

kevin gunn

So first, if you didn’t catch it, series 16 Ubuntu Core beta images are available here http://cdimage.ubuntu.com/ubuntu-snappy/16.04/current/

I just verified the Mir snaps are functioning and made some small updates to match here https://developer.ubuntu.com/en/snappy/build-apps/mir-snaps/

One of which being a command line switch for installing the snaps locally… –dangerous, what a great flag name </p>
            <a href=Read more

Daniel Holbach

I’m looking forward to next week, as

Going-To-Akademy-2016On Wednesday I’m going to give this workshop

So if you are interested in learning how to publish software easily and directly to users, this might be just for you.

Snaps are self-contained, confined apps, which run across a variety of Linux systems. The process of snapping software is very straight-forward and publishing them is very quick as well. The whole process offers many things upstreams and app publishers have been asking for years.

The workshop is interactive, all that’s required is that you either have VirtualBox or qemu installed or run any flavour of Ubuntu 16.04 or later. I’m going to bring USB sticks with images.

The workshop will consist of three very straight-forward parts:

  • Using the snap command to find, install, remove, update and revert software installations.
  • Using snapcraft to build and publish software.
  • Taking a look at KDE/Qt software and see how it’s snapped.

A few words about your host of the session: I’m Daniel Holbach, I have been part of the Ubuntu community since its very early days and work for Canonical on the Community team. Right now I’m working closely with the Snappy team on making publishing software as much fun as it should be.

See you next Wednesday!

Read more
kevin gunn

Wanted to quickly share some work-in-progress. I’d previously posted about running unconfined snaps on unity8 , this post is about a small but important incremental step in the progress. Since the mir interface is now part of snapd – you can begin confining apps as snaps targeting Unity8. What this means is, those developers who’ve been delivering and maintaining clicks for Unity8 will have a method to test out their snap on the Unity8 desktop. This will help those developers eventually migrate completely to snaps quickly and easily while keeping their confinement.

So if you want to give it a shot….

Here’s the pull request for adding mir to implicit interfaces , which will make the mir interface an available slot on the system. You can use the mod and rebuild snapd using Zyga’s tools  (If you’ve never modified snapd, recommend reading the README  on github  )

And one other hot tip that wasn’t obvious when I started tinkering with this…when rebuilding/running snapd, just use the one-shot method like “$ ./refresh-bits snap snapd setup run-snapd” this will build snap, snapd, stop the running installed snapd and run the newly built snapd all in one go.

Once you’ve got snapd rebuilt and running, you just need to rebuild ubuntu-clock-app from snappy playpen. Again, it’s only a one-line change to add “mir” as a plug to the ubuntu-clock-app’s yaml. Here’s a branch with the change.

Anyhow, I hope you give it a shot and let us know about any struggles you might encounter. You can find me in either #snappy or #unity8 on freenode.

Read more
kevin gunn

Mir interface landed!

So the Mir interface has landed in snapd. This means that the mir-server snap can be downloaded and taken into use in a fully confined mode, along with matching mir-client snap. I’ve been keeping updated instructions here on the best way to operate and develop your mir-client snap. The one caveat is there’s still a snappy bug (launchpad bug# 1577897)  with snap auto-connections.  This means that you have to manually make the plug-slot connection of the mir-server and mir-client, and then restart the mir-client service. I’ve just uploaded a new mir-server snap which is on Mir version 0.23.5, it should be reflected in the store soon.

At any rate, I’d love to hear back from folks trying this out. In the coming weeks I’m hoping to spend some time on dragonboard and getting this working there.

Read more
Daniel Holbach

Distributing software has never been easier. snapcraft makes it easy to build any kind of app, snapd and snap-confine bring security and hassle-free updates. Maintaining the app in the store is simple and you get lots of flexibility with different release channels.

If you’re interested or curious, adding your software to the Snappy Playpen, might be a good first step. Tomorrow, Tuesday 12th July 2016, we are working together on getting more snaps landed, getting things improved, updating our docs, helping out the snapd/snapcraft people, and upstreaming snaps.

It’s easy to get in touch, we are both hanging out in

We are looking forward to seeing you there.

Read more
kevin gunn

Snap on Unity8 classic

I wanted to provide a quick guide for those who might be tinkering around with snaps. This is a demonstration for developers interested in the ability to create snaps of their applications as well for Unity8, analogous to the experience of snaps on classic Ubuntu on Unity7, as outlined at https://developer.ubuntu.com/en/desktop/get-started/. These instructions assume you’ve already installed and experienced ubuntu-core, the ubuntu-snappy-cli, installing and running snaps on Unity7. You should also be running an up-to-date Ubuntu 16.04 system, since there are updates consistently being made to snapcraft and snapd.

In following this guide you’ll eventually run ubuntu-clock-app as the demonstration. In order to avoid confusion, ensure you do not have the clock application installed on your ubuntu-core from previous experimentation. The reason for this is that ubuntu-clock-app in the store assumes it is running on an X11 based system, but the snap we are going to build and install here assumes it is running on a Mir based system and there is no X11 in the Unity8 session. So before we begin, while logged into your Unity7 session run the following:

 

$ sudo snap remove ubuntu-clock-app

 

If you haven’t already, you need to install the unity8-desktop-session-mir package and add ppa:ci-train-ppa-service/stable-phone-overlay (the addition of the ppa is in order to keep up to date with the latest Unity8, UI toolkit and Mir changes that are being regularly released for the phone).

Note1: we are in the process of SRU’ing a change to Unity8-desktop-session-mir that will automatically add the PPA, but for now this is required as the SRU process will take time.

Note2: you must be using free graphics drivers for the unity8-desktop-session-mir, for instance if you are on an Intel gpu you will have no problems, if you are on an AMD gpu you will need radeon, if you are on Nvidia you will need Nouvea.

Open a terminal and run the following commands:

 

$ sudo apt-add-repository ppa:ci-train-ppa-service/stable-phone-overlay

$ sudo apt-get update

$ sudo apt-get dist-upgrade

$ sudo apt-get install unity8-desktop-session-mir

 

At this point, you should be able to log out of your Unity7 session and toggle the greeter icon to Unity8 and login. Once in your Unity8 session, you can navigate to the Ubuntu Store scope on the Dash via the Apps scope. Search for the terminal app and install it.

NOTE: i’m trying to get the terminal app updated in the store to be a “fat package” which means working for both amd64 and armhf architectures. Until then you can find and install the terminal app from https://launchpad.net/~dpm/+archive/ubuntu/ppa

$ sudo add-apt-repository ppa:dpm/ppa

$ sudo apt update

$ sudo apt install ubuntu-terminal-app

Open up the terminal app to download the ubuntu-clock-app source that has relevant snapcraft files and modifications to support running on the Unity8-Mir system. If you want to see the what modifications I had to add you may compare the branch we’ll be building, lp:~kgunn72/snappy-desktop-exmaples/try-mir-take1 to lp:snappy-desktop-examples . The modifications are mainly that the snap needs to contain the qtmir plugin since Unity8 is running on the Mir stack for graphics and X11 is not running.

 

You’ll notice on the installation step we install with –devmode, as we are just now in the process of creating the interfaces for Unity8 and Mir. Running in –devmode means that the application is not confined. Then enjoy building and running the snap:

 

$ bzr branch lp:~kgunn72/snappy-desktop-examples/try-mir-take1

$ cd ./try-mir-take1/ubuntu-clock-app

$ snapcraft

$ sudo snap install ubuntu-clock-app*.snap –devmode

 

OK congrats, you’ve built a snap for an application that runs on unity8. There is one bug (https://bugs.launchpad.net/ubuntu/+source/unity8-desktop-session/+bug/1590439) that we need to account for to add /snap/bin to our path. But once done, you should be able to launch the app.

 

$ export PATH=$PATH: /snap/bin

$ ubuntu-clock-app.clock

Hope you enjoyed this exercise. In theory this could be applied to any application that might have been created for the Ubuntu phone. We’ll be working to make this process less manual over time.

Read more
Daniel Holbach

We are in the second week of the Snappy Playpen and it’s simply beautiful to see how new folks are coming in and collaborate on getting snaps done, improve existing ones, answer questions and work together. The team spirit is strong and we’re all learning loads.

Keep up the good work everyone! </p>
            <a href=Read more