Canonical Voices

Posts tagged with 'ci'

pitti

This week from Tuesday to Thursday four Canonical Foundations team members held a virtual sprint about the proposed-migration infrastructure. It’s been a loooong three days and nightshifts, but it was absolutely worth it. Thanks to Brian, Barry, and Robert for your great work!

I started the sprint on Tuesday with a presentation (slides) about the design and some details about the involved components, and showed how to deploy the whole thing locally in juju-local. I also prepared a handful of bite-size improvements which were good finger-exercises for getting familiar with the infrastructure and testing changes. I’m happy to report that all of those got implemented and are running in production!

The big piece of work which we all collaborated on was providing a web-based test retry for all Ubuntu developers. Right now this is limited to a handful of Canonical employees, but we want Ubuntu developers to be able to retry autopkgtest regressions (which stop their package from landing in Ubuntu) by themselves. I don’t know the first thing about web applications and OpenID, so I’m really glad that Barry and Robert came up with a “hello world” kind of Flask webapp which uses Ubuntu SSO authentication to verify that the requester is an Ubuntu Developer. I implemented the input variable validation and sending the actual test requests over AMQP.

Now we have a nice autopkgtest-retrier git with the required functionality and 100% (yes, complete!) test coverage. With that, requesting tests in a local deployment works! So what’s left to do for me now is to turn this into a CGI script, configure apache for it, enable SSL on autopkgtest.ubuntu.com, and update the charms to set this all up automatically. So this moved from “ugh, I don’t know where to start” from “should land next week” in these three days!

We are going to have similar sprints for Brian’s error tracker, Robert’s CI train, and Barry’s system-image builder in the next weeks. Let’s increase all those bus factors from the current “1” to at least “4” ☺ . Looking forward to these!

Read more
bmichaelsen

But I believe in this and it’s been tested by research
— The Clash, Death and Glory

Thanks to Norbert’s efforts, the LibreOffice project now has a Jenkins setup that not only gives us visibility on how healthy our master branch is, with the results being reported to the ESC regularly: In addition it allows everyone easily testing commits and branches on all major LibreOffice platforms (Linux, OS X, Windows) just by uploading a change to gerrit. Doing so is really easy once you are set up:

./logerrit submit                      # a little helper script in our repo
git push logerrit HEAD:refs/for/master # alternative: plain old git
git review                             # alternative: needs to install the git-review addon

Each of the above commands alone send your work for review and testbuilding to gerrit. The last one needs an additional setup, that is however really helpful and worth it for people working with gerrit from the command-line regulary.

So, what if you have a branch that you want to testbuild? Well, just pushing the branch to gerrit as suggested above still works: gerrit then will create a change for every commit, mark them as depending on each other and testbuild every commit. This is great for a small branch of a handful of commits, but will be annoying and somewhat wasteful for a branch with more than 10-15 commits. In the latter case you might not want a manual review for each commit and also not occupy our builders for each of them. So what’s the alternative, if you have a branch ${mybranch} and want to at least test the final commit to build fine everywhere?

git checkout -b ${mybranch}-ci ${mybranch} # switch to branch ${mybranch}-ci
git rebase -i remotes/logerrit/master      # rebase the branch on master interactively

Now your favourite editor comes up showing the commits of the branch. As your favourite editor will be vim, you can then type:

:2,$s/^pick/s/ | x

To squash all the commits of the branch into one commit. Then do:

git checkout -                                   # go back to whatever branch we where on before
git push logerrit ${mybranch}-ci:refs/for/master # push squashed branch to gerrit for testbuilding
git branch -D ${mybranch}-ci                     # optional: delete squashed branch locally

Now only wait for the builder on Jenkins to report back. This allowed me to find out that our compiler on OS X didnt think of this new struct as a POD-type, while our compilers on Linux and Windows where fine with it (see: “Why does C++ require a user-provided default constructor to default-construct a const object?” for the gory details). Testbuilding on gerrit allowed me to fix this before pushing something broken on a platform to master, which would have spoiled the nifty ability to test your commit before pushing for everyone else: Duly testing your commit on gerrit only to find that the master you build upon was broken by someone else on some platform is not fun.

The above allows you to ensure the end of your branch builds fine on all platforms. But what about the intermediate commits and our test-suites? Well, you can test that each and every commit passes tests quite easily locally:

git rebase -i remotes/logerrit/master --exec 'make check'

This rebases your branch on master (even if its already up to date) and builds and runs all the tests on each commit along the way. In case there is a test breakage, git stops and lets you fix things (just like with traditional troubles on rebases like changes not applying cleanly).

Note: gerrit will close the squashed branch change if you push the branch to master: The squashed commit message ends with the Change-Id of the final commit of the branch. So once that commit is pushed, the gerrit closes the review for the squashed change.

Another note: If the above git commands are too verbose for you (they are for me), consider using gitsh and aliases. Combined they help quite a lot in reducing redundant typing when working with git.


Read more
bmichaelsen

But I believe in this and it’s been tested by research
— The Clash, Death and Glory

Thanks to Norbert’s efforts, the LibreOffice project now has a Jenkins setup that not only gives us visibility on how healthy our master branch is, with the results being reported to the ESC regularly: In addition it allows everyone easily testing commits and branches on all major LibreOffice platforms (Linux, OS X, Windows) just by uploading a change to gerrit. Doing so is really easy once you are set up:

./logerrit submit                      # a little helper script in our repo
git push logerrit HEAD:refs/for/master # alternative: plain old git
git review                             # alternative: needs to install the git-review addon

Each of the above commands alone send your work for review and testbuilding to gerrit. The last one needs an additional setup, that is however really helpful and worth it for people working with gerrit from the command-line regulary.

So, what if you have a branch that you want to testbuild? Well, just pushing the branch to gerrit as suggested above still works: gerrit then will create a change for every commit, mark them as depending on each other and testbuild every commit. This is great for a small branch of a handful of commits, but will be annoying and somewhat wasteful for a branch with more than 10-15 commits. In the latter case you might not want a manual review for each commit and also not occupy our builders for each of them. So what’s the alternative, if you have a branch ${mybranch} and want to at least test the final commit to build fine everywhere?

git checkout -b ${mybranch}-ci ${mybranch} # switch to branch ${mybranch}-ci
git rebase -i remotes/logerrit/master      # rebase the branch on master interactively

Now your favourite editor comes up showing the commits of the branch. As your favourite editor will be vim, you can then type:

:2,$s/^pick/s/ | x

To squash all the commits of the branch into one commit. Then do:

git checkout -                                   # go back to whatever branch we where on before
git push logerrit ${mybranch}-ci:refs/for/master # push squashed branch to gerrit for testbuilding
git branch -D ${mybranch}-ci                     # optional: delete squashed branch locally

Now only wait for the builder on Jenkins to report back. This allowed me to find out that our compiler on OS X didnt think of this new struct as a POD-type, while our compilers on Linux and Windows where fine with it (see: “Why does C++ require a user-provided default constructor to default-construct a const object?” for the gory details). Testbuilding on gerrit allowed me to fix this before pushing something broken on a platform to master, which would have spoiled the nifty ability to test your commit before pushing for everyone else: Duly testing your commit on gerrit only to find that the master you build upon was broken by someone else on some platform is not fun.

The above allows you to ensure the end of your branch builds fine on all platforms. But what about the intermediate commits and our test-suites? Well, you can test that each and every commit passes tests quite easily locally:

git rebase -i remotes/logerrit/master --exec 'make check'

This rebases your branch on master (even if its already up to date) and builds and runs all the tests on each commit along the way. In case there is a test breakage, git stops and lets you fix things (just like with traditional troubles on rebases like changes not applying cleanly).

Note: gerrit will close the squashed branch change if you push the branch to master: The squashed commit message ends with the Change-Id of the final commit of the branch. So once that commit is pushed, the gerrit closes the review for the squashed change.

Another note: If the above git commands are too verbose for you (they are for me), consider using gitsh and aliases. Combined they help quite a lot in reducing redundant typing when working with git.


Read more
Nicholas Skaggs

Keeping ubuntu healthy: Core Apps

Continuing our discussion of testing within ubuntu, today's post will talk about how you can help the community core apps stay healthy.

As you recall the core apps go through a series of QA before being released to the store. However bugs in the application, or in the platform itself can still be exposed. The end result is that the dashboard contains tests failures for that application. To release a new stable image, we need a green dashboard, and more importantly we need to make sure the applications work properly.

Getting plugged in
So to help out, it's important to first plug into the communication stream. After all, we're building these applications and images as a community! First, join the ubuntu phone group on launchpad and sign up for the phone mailing list. The list is active and discussing all issues pertaining to the ubuntu phone. Most importantly, you will see landing team emails that summarize and coordinate issues with the phone images.

From there you can choose a community core app to help improve from a quality perspective. These applications all have development teams and it's helpful to stay in contact with them. Your merge proposal can serve as an introduction!

Finding something to work on
So what needs fixing? A landing team email might point out a failing test. You might notice a test failure on the dashboard yourself. In addition each application keeps a list of bugs reported against it, including bugs that point out failing tests or testing needs. For example here's the list of all new autopilot tests that need to be written for all of the core apps. Pick an app, browse the buglist, assign a bug to yourself, and fix it.

For example, here's the list of bugs for music app. As of this writing you can see several tests that need written, as well as a bug for a test improvement.

You can also simply enhance the app's existing testsuite by fixing a flaky test, or improving the test to use best practices, etc. As a bonus for those reading this near it's original publication date, we just had a session @ vUDS covering the core apps and the testing needs we have. Watch the session / browse the pad and pick something to work on.

Fixing things
Look into any failures you find and have a look at the tests. Often the tests can use a little improvement (or maybe an additional test), and you can help out here! Sometimes failures won't happen every run -- this is the sign of a weird bug, or more likely a flaky test.  Fix the test(s), improve them, or add to them. Then commit your work and submit a merge proposal. Follow the guide on the wiki if you need help with doing this.

Remember, you can iteratively run the tests on your device as you work. Read my post on click-buddy for help with this. If you are lacking a device, run the tests on your desktop instead and a reviewer can test against a real device before merging.

Getting Help
For realtime help, check out #ubuntu-quality and #ubuntu-autopilot on freenode. You'll find a group of folks like yourself working on tests, hacking on autopilot and sharing advice. If IRC isn't your thing, feel free to contact us through another method instead. Happy hacking!

Read more
Nicholas Skaggs

Since just before the last LTS, quality has been a buzzword within the ubuntu community. We've come a long way since precise and I wanted to provide some help and prospective on what ubuntu's process for quality looks like this cycle. In simple terms. Or as reddit would say, "explain to me like I'm 5".

I'll try and define terms as we go. First let me define CI, which is perhaps the buzzword of this cycle, lest I lose all of you! CI stands for continuous integration, and it means we are testing ubuntu. All the time. Non-stop. Every change we make, we test. The goal behind this idea is to find and fix problems, before well, they become problems on your device!

CI Dashboard
The CI dashboard then is a way to visually see the results of this testing. It acts as a representation of the health of ubuntu as a distribution. At least once a day runs are executed, apps and images are tested and benchmarked, and the results are populated on ci.ubuntu.com. This is perhaps the most visible part of the CI (our continuous testing efforts) that is happening within ubuntu. But let's step back a minute and look at how the overall CI process works within ubuntu.

CI Process
App developers hack on a bit of code, fixing bugs or adding new features to the codebase. Once the code is ready, a merge proposal1 is created by the developer and feedback is sought. If the code passes the peer review and the application's tests, it will then become part of the image after a journey through the CI train.


For the community core apps, the code is merged after peer review, and then undergoes a similar journey to the store where it will become part of the image as well. Provided of course it meets further review criteria by myself and Alan (we'll just call him the gatekeeper).
Though menacing, Alan assures me he doesn't bite

Lest we forget, upstream2 uploads3 are done as well. We can hope some form of testing was done on them before we received them. Nevertheless, tests are run on these as well, and if they pass successfully, the new packages will enter the archive4 and become part of the image.

Generating Images
Now it's time to generate some images. For the desktop a snapshot of what's in the ubuntu archive is taken each day, built, and then subjected to a series of installation tests. If the tests pass, it is released for general testing called Image (or ISO) testing. An image is tested and declared stable as part of a milestone (or testing event) and can become the next version of ubuntu!
Adopted images are healthy images!

On the ubuntu phone side of things, all the new uploads are gathered and evaluated for risk. If something is a large change, it might be prudent to not land it with other large changes so we can tell what broke should the image not work properly. Once everything is ready, a new image is created and is released for testing.  The OTA updates (over-the-air; system updates) on your ubuntu phone come from this process!

How you can help?
Still with me I hope? As you can see there's many things happening each day in regards to quality and lots of places where you can create a positive change for the health of the distro! In my next few posts, I'll cover each of the places you can plug in to help ubuntu be healthy everyday!

1. A merge proposal is a means of changing an applications code via peer review.
2. By upstream, I mean the communities and people who make things we use inside of ubuntu, but are not directly a part of it. Something like the browser (firefox) and kernel are good examples.
3. This can happen via a general sync at the beginning of the cycle from debian. This sync copies part of the debian archive into the ubuntu archive, which in effect causes applications to be updated. Applications are also updated whenever a core ubuntu developer or a MOTU uploads a new version to the archive. 
4. In case you are wondering, "the archive" is the big repository where all of your updates and new applications come from!

Read more
Jussi Pakkanen

The quest for software quality has given us lots of new tools: new compiler warnings, making the compiler treat all warnings as errors, style checkers, static analyzers and the like. These are all good things to have. Sooner or later in a project someone will decide to make them mandatory. This is fine as well, and the reason we have continuous integration servers.

Still, some people might, at times, propose MRs that cause CI to emit errors. The issues are fixed, some time is lost when doing this but on the whole it is no big deal. But then someone comes to the conclusion that these checks should be mandatory on every build so the errors never get to the CI server. Having all builds pristine all the time is great, the reasoning goes, because then errors are found as soon as possible. This is as per the agile manifesto and universally a good thing to have.

Except that it is not. It is terrible! It is a massive drain on productivity and the kind of thing that makes people hate their job and all things related to it.

This is a strong and somewhat counter-intuitive statement. Let’s explore it with an example. Suppose we have this simple snippet of code.

  x = this_very_long_function_that_does_something(foo, bar, baz10, foofoofoo);

Now let’s suppose we have a bug somewhere. As part of the debugging cycle we would like to check what would happen if x had the value 3 instead of whatever value the function returns. The simple way to check is to change the code like this.

  x = this_very_long_function_that_does_something(foo, bar, baz10, foofoofoo);
  x = 3;

This does not give you the result you want. Instead you get a compile/style/whatever checker error. Why? Because you assign to variable x twice without using the first value for anything. This is called a dead assignment. It may cause latent bugs, so the checker issues an error halting the build.

Fair enough, let’s do this then.

  this_very_long_function_that_does_something(foo, bar, baz10, foofoofoo);
  x = 3;

This won’t work either. The code is ignoring the return value of the function, which is an error (in certain circumstances but not others).

Grumble, grumble. On to iteration 3.

  //x = this_very_long_function_that_does_something(foo, bar, baz10, foofoofoo);
  x = 3;

This will also fail. The line with the comment is over 80 characters wide and this is not tolerated by many style guides, presumably because said code bases are being worked on by people who only have access to text consoles. On to attempt 4.

//x = this_very_long_function_that_does_something(foo, bar, baz10, foofoofoo);
  x = 3;

This won’t work either for two different reasons. Case 1: the variables used as arguments might not be used at all and will therefore trigger unused variable warnings. Case 2: if any argument is passed by reference or pointer, their state might not be properly updated.

The latter case is the worst, because no static checker can detect it. Requiring the code to conform to a cosmetic requirement caused it to grow an actual bug.

Getting this kind of test code through all the testers is a lot of work. Sometimes more than the actual debug work. Most importantly, it is completely useless work. Making this kind of exploratory code fully polished is useless because it will never, ever enter any kind of production. If it does, your process has bigger issues than code style. Any time spent working around a style checker is demotivational wasted effort.

But wait, it gets worse!

Type checkers are usually slow. They can take over ten times longer to do their checks than just plain compiling the source code. Which means that developer productivity with these tools is, correspondingly, several times lower than it could be. Programmers are expensive, having them sitting around watching preprocessor checker text scroll slowly by in a terminal is not a very good use of their time.

Fortunately there is a correct solution for this. Make the style checks a part of the unit test suite, possibly as part of an optional suite of slow tests. Run said tests on every CI merge. This allows developers to be fast and productive most of the time but precise and polished when required.

Read more
David Murphy (schwuk)

As part of our self-improvement and knowledge sharing within Canonical, within our group (Professional and Engineering Services) we regularly – at least once a month – run what we call an “InfoSession”. Basically it is Google Hangout on Air with a single presenter on a topic that is of interest/relevance to others, and one of my responsibilities is organising them. Previously we have had sessions on:

  • Go (a couple of sessions in fact)
  • SystemTap
  • Localization (l10n) and internationalization (i18n)
  • Juju
  • Graphviz
  • …and many others…

Today the session was on continuous integration with Tarmac and Vagrant, presented by Daniel Manrique from our certification team. In his own words:

Merge requests and code reviews are a fact of life in Canonical. Most projects start by manually merging approved requests, including running a test suite prior to merging.

This infosession will talk about tools that automate this workflow (Tarmac), while leveraging your project’s test suite to ensure quality, and virtual machines (using Vagrant) to provide multi-release, repeatable testing.

Like most of our sessions it is publicly available, here it is is for your viewing pleasure:

Read more