Canonical Voices

Posts tagged with 'testing'

Nicholas Skaggs

As promised, here is your reminder that we are indeed fast approaching the final image for trusty. It's release week, which means it's time to put your energy and focus into finding and getting the remaining bugs documented or fixed in time for the release.

We need you!
The images are a culmination of effort from everyone. I know many have already tested and installed trusty and reported any issues encountered. Thank you! If you haven't yet tested, we need to hear from you!

How to help
The final milestone and images are ready; click here to have a look.

Execute the testcases for ubuntu and your favorite flavor images. Install or upgrade your machine and keep on the lookout for any issues you might find, however small.

I need a guide!
Sound scary? It's simpler than you might think. Checkout the guide and other links at the top of the tracker for help.

I got stuck!
Help is a simple email away, or for real-time help try #ubuntu-quality on freenode. Here are all the ways of getting ahold of the quality team who would love to help you.

Plan to help test and verify the images for trusty and take part in making ubuntu! You'll join a community of people who do there best everyday to ensure ubuntu is an amazing experience. Here's saying thanks, from me and everyone else in the community for your efforts. Happy testing!

Read more
Nicholas Skaggs

Time to test trusty!

Say that three times fast. Time to test trusty,
time to test trusty, time to test trusty!
Ahh it's my favorite time of the cycle. This is the part were we all get serious, go a little bit crazy, and end super excited to release a new version of ubuntu into the world. This time it's even more special as the new version is a brand new LTS, which we look forward to supporting for the next 5 years.

The developers and early adopters have been working hard all cycle to put forth the best version of ubuntu to date. For you! For all of us! It's time to fix bugs, do last minute polish and prepare for the release candidate which will occur around April 11th.

We need you!
This is were you dear reader come in. You see despite their good looks and wonderful sense of humor and charm, the release team doesn't like to release final images of ubuntu that haven't been thoroughly tested.

The release team is ready to pounce on untested images
We need testing, and further, we need the results of that testing! We need to hear from you. Passing test results matter just as much as failures. The way to record these results is via the isotracker; we can't read your mind sadly!

How to help
Mark your calendars now for April 11th - April 16th. Pick a good date for you and plan to download and test the release candidate image. You'll see a new milestone on the tracker, and an announcement here as well when the image is ready. I won't let you forget, promise!

Execute the testcases for ubuntu and your favorite flavor images. Install or upgrade your machine and keep on the lookout for any issues you might find, however small.

I need a guide!
Sound scary? It's simpler than you might think. Checkout the guide and other links at the top of the tracker for help.

I got stuck!
Help is a simple email away, or for realtime help try #ubuntu-quality on freenode. Here's all the ways of getting ahold of the quality team who would love to help.

Plan to help test and verify the images for trusty and take part in making ubuntu! You'll join a community of people who do there best everyday to ensure ubuntu is an amazing experience. Here's saying thanks, from me and everyone else in the community for your efforts. Happy testing!

Read more
David Murphy (schwuk)

Today I was adding tox and Travis-CI support to a Django project, and I ran into a problem: our project doesn’t have a Of course I could have added one, but since by convention we don’t package our Django projects (Django applications are a different story) – instead we use virtualenv and pip requirements files - I wanted to see if I could make tox work without changing our project.

Turns out it is quite easy: just add the following three directives to your tox.ini.

In your [tox] section tell tox not to run

skipsdist = True

In your [testenv] section make tox install your requirements (see here for more details):

deps = -r{toxinidir}/dev-requirements.txt

Finally, also in your [testenv] section, tell tox how to run your tests:

commands = python test

Now you can run tox, and your tests should run!

For reference, here is a the complete (albeit minimal) tox.ini file I used:

envlist = py27
skipsdist = True

deps = -r{toxinidir}/dev-requirements.txt
setenv =
    PYTHONPATH = {toxinidir}:{toxinidir}
commands = python test

The post Using tox with Django projects appeared first on David Murphy.

Read more

Our current autopkgtest machinery uses Jenkins (a private and a public one) and lots of “rsync state files between hosts”, both of which have reached a state where they fall over far too often. It’s flakey, hard to maintain, and hard to extend with new test execution slaves (e. g. for new architectures, or using different test runners). So I’m looking into what it would take to replace this with something robust, modern, and more lightweight.

In our new Continuous Integration world the preferred technologies are RabbitMQ for doing the job distribution (which is delightfully simple to install and use from Python), and OpenStack’s swift for distributed data storage. We have a properly configured swift in our data center, but for local development and experimentation I really just want a dead simple throw-away VM or container which gives me the swift API. swift is quite a bit more complex, and it took me several hours of reading and exercising various tutorials, debugging connection problems, and reading stackexchange to set it up. But now it’s working, and I condensed the whole setup into a single shell script.

You can run this in a standard ubuntu container or VM as root:

sudo apt-get install lxc
sudo lxc-create -n swift -t ubuntu -- -r trusty
sudo lxc-start -n swift
# log in as ubuntu/ubuntu, and wget or scp
sudo ./

Then get swift’s IP from sudo lxc-ls --fancy, install the swift client locally, and talk to it:

$ sudo apt-get install python-swiftclient
$ swift -A -U testproj:testuser -K testpwd stat

Caveat: Don’t use this for any production machine! It’s configured to maximum insecurity, with static passwords and everything.

I realize this is just poor man’s juju, but juju-local is currently not working for me (I only just analyzed that). There is a charm for swift as well, but I haven’t tried that yet. In any case, it’s dead simple now, and maybe useful for someone else.

Read more
Nicholas Skaggs

Continuing our discussion of the testing within ubuntu, today's post will talk about how you can help ubuntu stay healthy by manually testing the images produced. No amount of robots or automated testing in the world can replace you (well, at least not yet, heh), and more specifically your workflow and usage patterns.

As discussed, everyday new images are produced for ubuntu for all supported architecture types. I would encourage you to follow along and watch the progression of the OS through these images and your testing. Every data point matters and testing on a regular basis is helpful. So how to get started?

Settle in with a nice cup of tea while testing!

The Desktop
For the desktop images everything you need is on the image tracker. There is a wonderful video and text tutorial for helping you get started. You report your results on the tracker itself in a simple web form, so you'll need a launchpad account if you don't have one.

The secondary way to help keep the desktop images in good shape is to install and run the development version of ubuntu on your machine. Each day you can check for updates and update your machine to stay in sync. Use your pc as you normally would, and when you find a bug, report it! Bugs found before the release are much easier to fix than after release.

Now for the phablet images you will need a device capable of running the image. Check out the list. Grab your device and go through the installation process as described on the wiki. Make sure to select the '-proposed' channel when you install so you will see updates to get the latest images being worked on every time they are built. From there you can update everyday. Use the device and when you find a bug, report it! Here's a wiki page to help guide your testing and help you understand how and where to report bugs.

Don't forget there's a whole team of people within ubuntu dedicated to testing just like you. And they would love to have you join them!

Read more
Nicholas Skaggs

Recently the ubuntu core app developers and myself have been on an adventure towards adopting cmake for the all the core applications. While some of the applications are pure qml it's still been useful to embark on adopting a singular build system for all of the projects. Now that (most) of the pain of transitioning is gone, I'm going to talk about one of the useful features of setting up cmake for your project; click-buddy!

Click-buddy is an evolving tool that helps you build and deploy click packages to your phablet device. In addition, it has the ability to setup the device to run your autopilot test suite. So, rather than writing anything further, let's cover an example. You are going to need phablet-tools installed for this to work. I'm going to branch the clock app, build the click package, install it on my device, and finally run the tests.

bzr branch lp:ubuntu-clock-app
cd ubuntu-clock-app
click-buddy --dir . --provision
phablet-test-run -v ubuntu_clock_app

Click-buddy is also gaining the ability to build your project, even it involves a plugin and you are interested in building for your phablet device (armhf). Once landed you will be able to run something like this for non-qml applications.

sudo click chroot -a armhf create
click-buddy --arch armhf --provision

This will setup a chroot automagically for you and compile and build your application. Give it a try!

Note, as of this writing, emulator support for the ubuntu-ui-toolkit emulator is not yet built in. If your tests fail with a module import, run this line from your connected pc. It will copy over the ubuntu-ui-toolkit emulator (provided you have it installed on your pc :-) ) and your tests should now properly run.

adb push /usr/lib/python2.7/dist-packages/ubuntuuitoolkit /home/phablet/autopilot/ubuntuuitoolkit

Read more

Today’s autopilot release provides a new feature for test case writers. Unless the widget you want to test has a direct object name (GtkBuilder ID/Qt objectName), it is often not that easy to find a widget in a deeply nested hierarchy in autopilot vis.

With the new version, if you have some parent widget (like the containing dialog) w in your test, you can now call w.print_tree() to dump the paths and properties of that widget and all its children to stdout. That’s easy enough to grep, so provides a “poor man’s full tree search”. You can also specify a different output sink, like a file object or a file name: w.print_tree('/tmp/dump.txt').

This is a first step towards making it easier to find widgets and properties you are interested in. Arguably this is mostly just a crutch, but I found it to be rather effective. Before this feature I often wrote little snippets like in LP#1241312, now this becomes much easier. A better solution for this would certainly be a “full tree search” in vis itself, but that’s not that easy to implement. It is on the roadmap for this cycle, though.

I am also currently working on a real-time property change monitor for autopilot-gtk, which may also help in some cases. Unfortunately we cannot build such a thing for autopilot-qt, as due to the nature of Qt object properties, changes of them cannot be monitored.

Read more
Anthony Dillon

I was recently asked to attend a cloud sprint in San Francisco as a front-end developer for the new Juju GUI product. I had the pleasure of finally meeting the guys that I have collaboratively worked with and ultimately been helped by on the project.

Here is a collection of things I learnt during my week overseas.

Mocha testing

Mocha is a JavaScript test framework that tests asynchronously in a browser. Previously I found it difficult to imagine a use case when developing a site, but I now know that any interactive element of a site could benefit from Mocha testing.

This is by no means a full tutorial or features set of Mocha but my findings from a week with the UI engineering team.

Breakdown small elements of your app or website its logic test

If you take a system like a user’s login and register, it is much easier to test each function of the system. For example, if the user hits the signup button you should test the registration form is then visible to the user. Then work methodically through each step of the process, testing as many different inputs you can think of.

Saving your bacon

Testing undoubtedly slows down initial development but catches a lot of mistakes and flaws in the system before anything lands in the main code base. It also means if a test fails you don’t have to manually check each test again by hand — you simply run the test suite and see the ticks roll in.

Speeds up bug squashing

Bug fixing becomes easier to the reporter and the developer. If the reporter submits a test that fails due to a bug, the developer will get the full scope of the issue and once the test passes the developer and reporter can be confident the problem no longer exists.


While I have read a lot about linting in the past but have not needed to use it on any projects I have worked on to date. So I was very happy to use and be taught the linting performed by the UI engineering team.

Enforces a standard coding syntax

I was very impressed with the level of code standards it enforces. It requires all code to be written in a certain way, from indenting and commenting to unused variables. This results in anyone using the code, being able to pick up it up and read it as if created by one person when in fact it may have contributed by many.

Code reviews

In my opinion code reviews should be performed on all front-end work to discourage sloppy code and encourage shared knowledge.

Mark up

Mark up should be very semantic. This can be a case of opinion, but shared discussion will get the team to an agreed solution, which will then be reused again by others in the similar situations.


CSS can be difficult as there are different ways to achieve a similar result, but with a code review the style used will be common practise within the team.


A perfect candidate as different people have different methods of coding. With a review, it will catch any sloppy or short cuts in the code. A review makes sure  your code is refactored to best-practise the first time.


Test driven development (TDD) does slow the development process down but enforces better output from your time spend on the code and less bugs in the future.

If someone writes a failing test for your code which is expected to pass, working on the code to produce a passing test is a much easier way to demonstrate the code now works, along with all the other test for that function.

I truly believe in code reviews now. Previously I was sceptical about them. I used to think that  “because my code is working” I didn’t need reviews and it would slow me down. But a good reviewer will catch things like “it works but didn’t you take a shortcut two classes ago which you meant to go back and refactor”. We all want our code to be perfect and to learn from others on a daily basis. That is what code reviews give us.

Read more
Nicholas Skaggs

Since I last posted on the topic of our communities future, we've helped push out a new release of ubuntu. Saucy and fresh, the little salamander is now out in the wild. With the release out it's time to move forward with these changes.

(c) Carla Sella
The beginning of a new cycle is always a time to breathe for us in quality and these past few days of relative calm has indeed been a welcome respite from the craziness of testing leading up to a release.

We can rebuild it. We have the technology. Better than it was before. Better, stronger, faster.

So, let's talk about some of the changes coming to quality for this new cycle.

Defined Roles
I want to help our new members become a productive part of ubuntu quality as soon as possible. To this end I've created a list of roles for the quality team. By defining roles within the team it is easy to see how you can contribute as a 'tester', 'test writer', or 'developer'.

Contribute anytime
While release milestones and calls for testing will still be important, contributing to ubuntu quality can be a daily task. There are activities you can perform any day and anytime in the cycle. The roles pages list activities for each role that can be done right now. If you are a tester, check out the activities you can do right now to help!

More exploratory testing
We're iterating faster and faster. Builds of new code are landing each day, and in some cases several times a day. We can't afford to only test every other week with cadence testing. Instead, we've ramped up efforts to automatically test on each of these new builds. But we still need manual testing!

As a tester you can provide testing results for images, packages and hardware at any time! In addition, exploratory testing is highly encouraged. This is were we as manual testers shine! I want to encourage ongoing exploratory testing all throughout the cycle. Run and use the development version of ubuntu on your machine all the time!

Tackling some big projects
One of the things I wanted to push us as a team towards was tackling some projects that have a wider impact than just us. To that end, you can see several big projects on the activities pages for each role.

For testers, we are undertaking making reporting issues better for users. For the test writers, one of the largest projects is spearheading the effort to make manual image testing less burdensome and more automated by automating ubiquity testing. And for developers, the autopilot emulators are big projects as well and need help.

More involvement with bugs
As a quality community with interact with many bugs. Sometimes we are finding the bug, other times we might confirm them or verify a fix works. However, we haven't always gone the extra step towards doing work with SRU verifications and bug triage. The bugsquad and others have traditionally performed these tasks. If you'll notice these activities are now encouraged for those in the tester role. Let's dive more into the bug world.

A potential expansion of the team
With the mention of the bugsquad and the encouraged involvement with bugs for our team, I would like to propose a union between the quality and bugsquad teams. I would encourage current bugsquad members to take up tester roles, and consider some of the additional opportunities that are available. For those who have been testing with the quality team in past cycles, let's get more invovled with bugs and traditional bugsquad activities as mentioned above.

Making a quality LTS
Trusty Tahr is going to be the next LTS release. Those of you who remember and use Precise will agree that it is going to be a tough act to follow. The bar is set high, but I am confident we can reach it and do better.

We can't do this alone. We need testers, test writers, and developers! If you are interested in helping us achieve our goal, please join us! Now is an excellent time to learn and grow with the rest of us on the team. Thanks for helping making ubuntu better!

Read more
Nicholas Skaggs

Here are 3 easy steps to testing prowess. Do your part to be prepared! Let's help make saucy a great release!

1) Review both the critical bugs and those found during isotesting throughout the cycle. This is handy in case you find something during testing and want to know if someone has already reported it or not.

2) Make sure you know how to use the tracker and how to test an iso. When the milestone is created and the images appear on October 10th, you want to be ready to go!

3) Test and report your results to the tracker ;-) Note, at this point in the cycle we prefer real hardware results. Upgrade your machine! Try a fresh install. If you've been sitting on raring, now is the time to migrate to saucy and let us know of any bugs you find.

Happy Testing!

Read more
Nicholas Skaggs

As I eluded to in my previous post, it's time for us as a community to fully embrace the place of automated testing in our workflows. Over the past year we've learned how to write testcases and to then apply that knowledge towards writing autopilot tests. At the same time ubuntu has been building testing infrastructure and launching a CI team to help run and maintain it.

With these changes and our acquired knowledge we can construct a sustainable vision for testing. So just what is that vision?

Let's make manual testing fun again. As we've ramped up our workloads, we find ourselves repetitively testing the same packages again and again (and running the same tests!). We'd call this regression testing and it's a perfect candidate for automated testing. Let's automate these tests and keep our focus on where we as humans excel.

In addition, we as a community participate in "calls for testing". These calls have become more and more exploratory in nature. In general, we have moved further away from a defined testcase ("perform steps 1, 2, and 3"). Instead, we encourage testers to adopt a try and break it attitude. This is where we as humans can excel, and you know what, have some fun at trying! Remember QA is the only place we encourage you to break things!

So let's get manual testing back to the exploratory testing we love.

Thank you little testing robot!
Let's expand our automated tests. We can increase testing coverage forever with a one time investment of writing a testcase. Developers, write tests for your code and feature sets as you develop them. As a quality community, let's do our best to make it easy for developers to do so.

In addition, a well written bug report can become an excellent testcase to be added to the application testuite. Let's look at bug reports for potential testcases. We can write the test and then forget it. The little robots will tell us if it becomes a problem again in the future and even prevent the bad code from getting shipped where it can break our machines (again). Not bad little robots!

So let's focus on ensuring tests exist for critical regressions when they get fixed.

Let's tackle some big projects. Who said we can't achieve amazing things in the quality community? This vision would be nothing more than an idea without some actions behind it!

Right now we as a community already have a chance to put these ideas in action. Exploratory testing is happening right now on the phablet devices. Get involved! This is the manual, have fun and try and break it, exploratory testing we all love. Even without a phablet device, regressions can be turned into autopilot tests. This meets our goal to expand our automated tests and to look at bugs as potential sources for those tests.

All test and no play makes for a sad image
Moving forward there are still a some big projects to tackle. One of the largest is the amount of manual testing effort required in cadence and milestone testing. We can bring automated testing to the mix to both reduce our on-going effort and raise the quality bar in our releases.

So let's be the leaders for change in ubuntu.

Are you in?

Read more

I recently created a test for digicam photo import for Shotwell (using autopilot and umockdev), and made that run as an autopkgtest. It occurred to me that this might be interesting for other desktop applications as well.

The community QA team has written some autopkgtests for desktop applications such as evince, nautilus, or Firefox. We run them regularly in Jenkins on real hardware in a full desktop environment, so that they can use the full desktop integration (3D, indicators, D-BUS services, etc). But of course for those the application already needs to be in Ubuntu.

If you only want to test functionality from the application itself and don’t need 3D, a proper window manager, etc., you can also call your autopilot tests from autopkgtest with a wrapper script like this:

set -e

# start X
(Xvfb :5 >/dev/null 2>&1 &)
export DISPLAY=:5

# start local session D-BUS
eval `dbus-launch`
export XAUTHORITY=/dev/null

# change to the directory where your autopilot tests live, and run them
cd `dirname $0`
autopilot run autopilot_tests

This will set up the bare minimum: Xvfb and a session D-BUS, and then run your autopilot tests. Your debian/tests/control should have Depends: yourapp, xvfb, dbus-x11, autopilot-desktop, libautopilot-gtk for this to work. (Note: I didn’t manage to get this running with xvfb-run; any hints to how to simplify this appreciated, but please test that it actually works.)

Please note that this does not replace the “run in full desktop session” tests I mentioned earlier, but it’s a nice addition to check that your package has correct dependencies and to automatically block new libraries/dependencies which break your package from entering Ubuntu.

Read more
Nicholas Skaggs

Automated Testing in ubuntu

So with all the automated testing buzz occurring in the quality world of ubuntu this cycle, I wanted to speak a little about why we're doing the work, and what the output of the work looks like.

The Why
So, why? Why go through the trouble of writing tests for the code you write? I'll save everyone a novel and avoid re-hashing the debate about whether testing is a proper use of development time or not. Simply put, for developers, it will prevent bugs from reaching your codebase, alleviate support and maintenance burdens, and will give you more confidence and feedback during releases. If you want your application to have that extra polish and completeness, testing needs to be a part. For users, the positives are similar and simple. Using well tested software prevents regressions and critical bugs from affecting you. Every bug found during testing is a bug you don't have to deal with as an end user. Coincidentally, if you like finding bugs in software, we'd love to have you on the team :-)

The How
In general, three technologies are being used for automated testing.
Autopilot, Autopkg, and QML tests. Click to learn more about writing tests for each respectively.

Any color is good, as long as it's green
The Results
The QA Dashboard
The dashboard holds most of the test results, and gives you a much nicer view of the results than just looking at a jenkins build screen. Take some time to explore all of the different tabs to see what's available here. I wanted to highlight just a couple areas in particular.
  • Autopkg Tests 
    • These testcases run at build time and are great testcases for low level libraries and integral parts of ubuntu. Check out the guide for help on contributing a testcase or two to these. Regressions have been spotted and fixed before even hitting the ubuntu archives.
  • Smoke Tests for ubuntu touch
    • This is some of the newest tests to come online and displays the results of the ubuntu phone image and applications, including the core apps which are written entirely by community members. Want to know how well an image is running on your device? This is the page to find it.
Ubiquity Installer Testing
Curious about how well the installer is working? Yep, we've got tests for those as well. The tests are managed via the ubiquity project on launchpad.

Ubuntu Desktop Tests
Wondering how well things like nautilus, gedit and your other favorite desktop applications are doing? Indeed, thanks to our wonderful quality community, we've got tests for those as well. The tests can be found here.

The Next Steps
We want to continue to grow and expand all areas of testing. If you've got the skills or the willingness to learn, try your hard at helping improve our automated testcases. There's a wide variety to choose from, and all contributions are most welcome!

Sorry robot, all our tests are belong to us
Don't have those skills? Don't worry, not only are machines not taking over the world, they aren't taking over testing completely either. We need your brainpower to help test other applications and to test deeper than a machine can do. Join us for our cadence weeks and general calls for testing and sample new software while you help ubuntu. In fact, we're testing this week, focusing on Mir.

Finally, remember your contributions (automated or manual) help make ubuntu better for us all! Thank you!

Read more

umockdev 0.3 introduced the notion of an “umockdev script”, i. e. recording the read()s and write()s that happen on a device node such as ttyUSB0. With that one can successfully run ModemManager in an umockdev testbed to pretend that one has e. g. an USB 3G stick.

However, this didn’t yet apply to the Ubuntu phone stack, where ofonod talks to Android’s “rild” (Radio Interface Layer Daemon) through the Unix socket /dev/socket/rild. Thus over the last days I worked on extending umockdev’s script recording and replaying to Unix sockets as well (which behave quite different and quite a bit more complex than ordinary files and character devices). This is released in 0.4, however you should actually get 0.4.1 if you want to package it.

So you now can make a script from ofonod how it makes a phone call (or other telephony action) through rild, and later replay that in an umockdev testbed without having to have a SIM card, or even a phone. This should help with reproducing and testing bugs like ofonod goes crazy when roaming: It’s enough to record the communication for a person who is in a situation to reproduce the bug, then a developer can study what’s going wrong independent of harware and mobile networks.

How does it work? If you have used umockdev before, the pattern should be clear now: Start ofonod under umockdev-record and tell it to record the communication on /dev/socket/rild:

  sudo pkill ofonod; sudo umockdev-record -s /dev/socket/rild=phonecall.script -- ofonod -n -d

Now launch the phone app and make a call, send a SMS, or anything else you want to replay later. Press Control-C when you are done. After that you can run ofonod in a testbed with the mocked rild:

  sudo pkill ofonod; sudo umockdev-run -u /dev/socket/rild=phonecall.script -- ofonod -n -d

Note the new --unix-stream/-u option which will create /tmp/umockdev.XXXXXX/dev/socket/rild, attach some server threads to accept client connections, and replay the script on each connection.

But wait, that fails with some

   ERROR **: ScriptRunner op_write[/dev/socket/rild]: data mismatch; got block '...', expected block '...'

error! Apparently ofono’s messages are not 100% predictable/reproducible, I guess there are some time stamps or bits of uninitialized memory involved. Normally umockdev requires that the program under test sticks to the previously recorded write() parts of the script, to ensure that the echoed read()s stay in sync and everything works as expected. But for cases like these were some fuzz is expected, umockdev 0.4 introduces setting a “fuzz percentage” in scripts. To allow 5% byte value mismatches, i. e. in a block of n bytes there can be n*0.05 bytes which are different than the script, you’d put a line

  f 5 -

before the ‘w’ block that will get jitter, or just put it at the top of the file to allow it for all messages. Please see the script format documentation for details.

After doing that, ofonod works, and you can do the exact same operations that you recorded, with e. g. the phone app. Doing other operations will fail, of course.

As always, umockdev-run -u is of course just a CLI convenience wrapper around the umockdev API. If you want to do the replay in a C test suite, you can call

   umockdev_testbed_load_socket_script(testbed, "/dev/socket/rild",
                                       SOCK_STREAM, "path/to/phonecall.script", &error);

or the equivalent in Python or Vala, as usual.

If you are an Ubuntu phone developer and want to use this, please don’t hesitate to talk to me. This is all in saucy now, so on the Ubuntu phone it’s a mere “sudo apt-get install umockdev” away.

Read more
Nicholas Skaggs

This month has been all about dogfooding, and part of that has been the drive to automate the testing of the functionality found in the core applications. Nothing will replace true user testing, but we can certainly do a lot to help prevent bugs and regressions. So, during my "test all the things" series we went through each of the core applications and highlighted areas for testing. So the question is, how are we doing as this month draws to a close?

Well, fortunately, I've setup this wiki page as a reflection of where we stand.
As you can see, we've come a long way on many of the applications, but I wanted to especially point out where we still need help.

Dropping Letters
Adrian has been working on Dropping Letters and could use some help in finishing the testcases. Leo has proposed a merge that brings the testsuite up to date with the new sdk, so it's ready for you to add tests!

RSS Reader
RSS Reader, aka shorts, just went through some large UI changes and is now ready for us to add tests again. Carla has begun working on updating her old tests for adding and editing a feed, but there's more work to do!

Calendar is currently undergoing some drastic changes, and it's testcases will need to be re-written once that's complete. Hang tight if your keen to help in this area!

Music has come a long way in a couple weeks and we have tests to prove it! Thanks to Daniel and Victor, we can test play, pause and library load. But there are still more tests needed. John is also volunteering on this application to help get the tests in shape. Thanks guys!

The following applications are very close to completion:

Nekhelesh has hacked together most of the tests, but I know he won't complain if you help him finish. Review the list and help bring it over the line!

As of this morning, the final calculator testcase has been written, testing tear-off. Pending a bugfix in the sdk, the testcase will get merged. Fingers crossed. Thanks Riccardo!

100% DONE!
Congratulations to the following teams and folks who helped us reach 100% status on the following core apps:

terminal, weather, sudoku, stock ticker, file manager

Remember though, as we move forward it's important to keep the tests aligned with new features :-)

Still reading this? What are you waiting for? Jump in and help! Happy Automated Testing!

Read more
Nicholas Skaggs

A little over a month ago, I posted about creating an autopilot emulator for
ubuntu sdk applications. Thanks to some hard work by Leo Arias and the ubuntu sdk team, I'm happy to announce the little side project from my +junk branch is all grown up and even more ready for your consumption. Leo has made the emulator a proper part of the sdk, complete with tests for all of it's functions!  You can now easily install it and incorporate it into your project. Here's how to snag your copy.

1) Add the ubuntu sdk team ppa if you haven't already

sudo add-apt-repository ppa:ubuntu-sdk-team/ppa

2) Install the ubuntu-ui-toolkit-autopilot package

sudo apt-get update && sudo apt-get install ubuntu-ui-toolkit-autopilot

This will install the emulator as a python module, ready for you to import. If you want to checkout what the module can do, have a look at the documentation.

Incorporating the module might seem a little tricky, so Leo has also put together an example of one of the ubuntu touch core apps using the new module. Check out it. Here's a branch showing off the work done for ubuntu-filemanager-app. And here's one for dropping-letters.

Please do check out the module and incorporate it into your ubuntu sdk project. Feedback is encouraged, and bug reports too! Please file any issues you find against the ubuntu-ui-toolkit project. Happy Automated Testing!

Read more
Nicholas Skaggs

The music app. A vital and necessary piece of my running setup. I've run in silence before, and while it's nice, sometimes you need some jams to keep you motivated on longer runs.

Recently I've been running in silence but not by choice, definitely needing a way to play music via my phone. Thanks to the hack day for music last week and the ongoing work on the music app developers that's all changing. Today the app is feature complete enough to begin work on some autopilot tests. It can for instance, play music now!

Consider helping the music app developers keep development going.  Grab the music app branchadd a testcase from the list of needsfollow the tutorial for help if needed, and propose a merge. Thanks for helping to ensure quality for ubuntu touch!

If you need help getting started, there is a wonderful video tutorial to help you. In addition the logs, and a FAQ from the workshops are now available and ready for you to consume and understand. Happy test writing!

Read more
Nicholas Skaggs

Today I thought we'd take a break from the testing all the things series to take a look at what we're doing, how far we've come, and most importantly, where we need to get to.

The core apps project has come a long way since it's inception at the beginning of this cycle. So as the development started maturing, testing was the natural step for me to get involved and check out what the apps had to offer. With that in mind, let's look at the burndown chart for the core apps shall we?

Can you tell where the tests were added?

So even as the work items has increased from adding things like tests, this month the community have been working hard at finishing work items and bringing us back to the trendline. See that nice driving downwards the last couple weeks? Excellent job! Let's get down to that trendline!

From the quality side, we've had several people come forward, learn about autopilot and the core apps and then get a merge in. A special thanks to Carla Sella, Daniel Kessel, Michael Spencer, Adrian Goodyer, Riccardo Padovani and Arturas Norkus for your contributions last week to core apps projects. Well done, and I look forward to seeing and merging more of your work!

For those of you still working on getting commits in, or sitting on the sidelines, let me help! You can be a part of this effort! There's still more work to do, and the entire community around core apps would appreciate your help. Hack on an app, a testcase, wherever your skills and interests lie. For autopilot tests specifically, check out this recipe on and watch this video.

I look forward to merging your work!

Read more

I’m happy to announce a new release 0.3 of umockdev.

The big new feature is the ability to fake character devices and provide recording and replaying of communications on them. This work is driven by our need to create automatic tests for the Ubuntu phone stack, i. e. pretending that we have a 3G or phone driver and ensuring that the higher level stacks behaves as expected without actually having to have a particular modem. I don’t currently have a phone capable of running Ubuntu, so I tested this against the standard ModemManager daemon which we use in the desktop. But the principle is the same, it’s “just” capturing and replaying read() and write() calls from/to a device node.

In principle it ought to work in just the same way for other device nodes than tty, e. g. input devices or DRI control; but that will require some slight tweaks in how the fake device nodes are set up; please let me know if you are intested in a particular use case (preferably as a bug report).

With just using the command line tools, this is how you would capture ModemManager’s talking to an USB 3G stick which creates /dev/ttyUSB{0,1,2}. The communication gets recorded into a text file, which umockdev calls “script” (yay my lack of imagination for names!):

# Dump the sysfs device and udev properties
$ umockdev-record /dev/ttyUSB* > huawei.umockdev

# Record the communication
$ umockdev-record -s /dev/ttyUSB0=0.script -s /dev/ttyUSB1=1.script \
     -s /dev/ttyUSB2=2.script -- modem-manager --debug

The –debug option for ModemManager is not necessary, but it’s nice to see what’s going on. Note that you should shut down the running system instance for that, or run this on a private D-BUS.

Now you can disconnect the stick (not necessary, just to clearly prove that the following does not actually talk to the stick), and replay in a test bed:

$ umockdev-run -d huawei.umockdev -s /dev/ttyUSB0=0.script -s /dev/ttyUSB1=1.script \
    -s /dev/ttyUSB2=2.script -- modem-manager --debug

Please note that the CLI options of umockdev-record and umockdev-run changed to be more consistent and fit the new features.

If you use the API, you can do the same with the new umockdev_testbed_load_script() method, which will spawn a thread that replays the script on the faked device node (which is just a PTY underneath).

If you want full control, you can also do all the communication from your test cases manually: umockdev_testbed_get_fd("/dev/mydevice") will give you a (bidirectional) file descriptor of the “master” end, so that whenever your program under test connects to /dev/mydevice you can directly talk to it and pretend that you are an actual device driver. You can look at the t_tty_data() test case for how this looks like (that’s the test for the Vala binding, but it works in just the same way in C or the GI bindings).

I’m sure that there are lots of open ends here still, but as usual this work is use case driven; so if you want to do something with this, please let me know and we can talk about fine-tuning this.

In other news, with this release you can also cleanly remove mocked devices (umockdev_testbed_remove_device()), a feature requested by the Mir developers. Finally there are a couple of bug fixes; see the release notes for details.

I’ll upload this to Saucy today. If you need it for earlier Ubuntu releases, you can have a look into my daily builds PPA.

Let’s test!

Read more
Nicholas Skaggs

It's almost FRIDAY! To celebrate, let's look at the second game that's become part of the core apps, dropping letters.

That best score! It was my first game, I can do much better now :)

The name of the game in dropping letters is to form words as the letters fall and try and prevent the screen from filling up. At the moment, the game is dogfoodable, but needs your help for tests. There are still some bugs within the code, and autopilot tests will help ensure those bugs don't sneak back into the codebase.

Consider helping the dropping letters developers keep development going.  Grab the dropping letters branchadd a testcase from the list of needsfollow the tutorial for help if needed, and propose a merge. Thanks for helping to ensure quality for ubuntu touch!

If you need help getting started, check out the logs, and a FAQ from the workshops. Feel free to contact me with any questions or help. Happy test writing!

Read more