In my last next post, I discussed will discuss notable autopilot features and talk about how autopilot has matured since it became an independent project.
In the meantime I would be remiss if I didn't also talk about the different test runners commonly used with autopilot tests. In addition to the autopilot binary which can be executed to run the tests, different tools have cropped up to make running tests easier.
This tool ships with autopilot itself and was developed as a way to run autopilot test suites on your desktop in a sane manner. Run the autopilot3-sandbox-run command with --help to see all the options available. By default, the tests will run in an Xvfb server, all completely behind the scenes with the results being reported to you upon completion. This is a great way to run tests with no interference on your desktop. If you are a visual person like me, you may instead wish to pass -X to enable the test runs to occur in a Xephyr window allowing you to see what's happening, but still retaining control of your mouse and keyboard.
I need this tool!
sudo apt-get install python3-autopilot
I want to run tests on my desktop without losing control of my mouse!
I want to run tests on my desktop without losing control of my mouse, but I still want to see what's happening!
autopilot3-sandbox-run -X my_testsuite_name
Autopkgtest was developed as a means to automatically test Debian packages, "as-installed". Recently support was added to also test click packages and to run on phablet devices. Autopkgtest will take care of dependencies, setting up autopilot, and unlocking the device. You can literally plug in a device and wait for the results. You should really checkout the README pages, including those on running tests. That said, here's a quick primer on running tests using autopkgtest.
I need this tool!
sudo apt-get install autopkgtest
If you are on trusty, grab and install the utopic deb from here.
I want to run tests for a click package installed on my device!
Awesome. This one is simple. Connect the device and then run:
adt-run --click my.click.name --- ssh -s adb
adt-run --click com.ubuntu.music --- ssh -s adb
will run the tests for the installed version of the music app on your device. You don't need to do anything else. For the curious, this works by reading the manifest file all click packages have. Read more here.
I want to run the tests I wrote/modified against an installed click package!
For this you need to also pass your local folder containing the tests. You will also want to make sure you installed the new version of the click package if needed.
adt-run my-folder/ --click my.click.name --- ssh -s adb
Autopkgtest can also run in a lxc container, QEMU, a chroot, and other fun targets. In the examples above, I passed --- ssh -s adb as the target, instructing autopkgtest to use ssh and adb and thus run the tests on a connected phablet device. If you want to run autopilot tests on a phablet device, I recommend using autopkgtest as it handles everything for you.
This tool is part of the greater phablet-tools package. It was originally developed as an easy way to execute tests on your phablet device. Note however that copying the tests and any dependencies to the phablet device is left to you. The phablet-tools package provides some other useful utilities to help you with this (checkout phablet-click-test-setup for example).
I need this tool!
sudo apt-get install phablet-tools
I want to run the tests I wrote/modified against an installed click package!
First copy the tests to the device. You can use the ubuntu sdk or click-buddy for this, or even do it manually via adb. Then run phablet-test-run. It takes the same arguments as autopilot itself.
phablet-test-run -v my_testsuite
Note the tools looks for the testsuite and any dependencies of the testsuite inside the /home/phablet/autopilot folder. It's up to you to make sure everything that is needed to run your tests are located there or else it will fail.
There are of course other possible test runners that wrap around autopilot to make executing tests easier. Perhaps you've written a script yourself. Just remember at the end of the day the autopilot binary will be running the tests. It simply needs to be able to find the testsuite and all of it's dependencies in order to run. For this reason, don't be afraid to execute autopilot3 and run the tests yourself. Happy test runs!
We're having our first hackfest of the utopic cycle this week on Tuesday, July 15th. You can catch us live in a hangout on ubuntuonair.com starting at 1900 UTC. Everything you need to know can be found on the wiki page for the event.
During the hangout, we'll be demonstrating writing a new manual testcase, as well as reviewing writing automated testcases. We'll be answering any questions you have as well about contributing a testcase.
We need your help to write some new testcases! We're targeting both manual and automated testcase, so everyone is welcome to pitch in.
We are looking at writing and finishing some testcases for ubuntu studio and some other flavors. All you need is some basic tester knowledge and the ability to write in English.
If you know python, we are also going to be hacking on the toolkit helper for autopilot for the ubuntu sdk. That's a mouthful! Specifically it's the helpers that we use for writing autopilot tests against ubuntu-sdk applications. All app developers make use of these helpers, and we need more of them to ensure we have good coverage for all components developers use.
Don't worry about getting stuck, we'll be around to help, and there's guides to well, guide you!
Hope to see everyone there!
Recently the ubuntu core app developers and myself have been on an adventure towards adopting cmake for the all the core applications. While some of the applications are pure qml it's still been useful to embark on adopting a singular build system for all of the projects. Now that (most) of the pain of transitioning is gone, I'm going to talk about one of the useful features of setting up cmake for your project; click-buddy!
Click-buddy is an evolving tool that helps you build and deploy click packages to your phablet device. In addition, it has the ability to setup the device to run your autopilot test suite. So, rather than writing anything further, let's cover an example. You are going to need phablet-tools installed for this to work. I'm going to branch the clock app, build the click package, install it on my device, and finally run the tests.
bzr branch lp:ubuntu-clock-app
click-buddy --dir . --provision
phablet-test-run -v ubuntu_clock_app
Click-buddy is also gaining the ability to build your project, even it involves a plugin and you are interested in building for your phablet device (armhf). Once landed you will be able to run something like this for non-qml applications.
sudo click chroot -a armhf create
click-buddy --arch armhf --provision
This will setup a chroot automagically for you and compile and build your application. Give it a try!
Note, as of this writing, emulator support for the ubuntu-ui-toolkit emulator is not yet built in. If your tests fail with a module import, run this line from your connected pc. It will copy over the ubuntu-ui-toolkit emulator (provided you have it installed on your pc :-) ) and your tests should now properly run.
adb push /usr/lib/python2.7/dist-packages/ubuntuuitoolkit /home/phablet/autopilot/ubuntuuitoolkit
Autopilot has continued to change with the times, and the pending release of 1.4 brings even more goodies; including some performance fixes. But today I wanted to cover a newly landed feature from the minds of Martin and Jean-Baptiste (thanks guys!).
If you've developed autopilot tests in the past you will have noticed how cumbersome it can be to run the tests. If you run a test on your desktop, you lose control of your mouse and keyboard for the duration, and you might even accidentally cause a test to fail. This can be especially noticeable when you are iterating over getting your test to run "just right" while wanting to keep your introspection tree in vis open, or reviewing someone else's code while wanting to verify the tests run. Enter sandbox mode.
A new command called autopilot-sandbox-run lets you easily run a testsuite inside your choice of two sandboxes; Xvfb by default, or if you want to visually see the output, Xephyr. Have a quick look at the command options below as of this writing:
Usage: autopilot-sandbox-run [OPTIONS...] TEST [TEST...]
Runs autopilot tests in a 'fake' Xserver with Xvfb or Xephyr. autopilot runs
in Xvfb by default.
TEST: autopilot tests to run
-h, --help This help
-d, --debug Enable debug mode
-a, --autopilot ARG Pass arguments ARG to 'autopilot run'
-X, --xephyr Run in nested mode with Xephyr
-s, --screen WxHxD Sets screen width, height, and depth to W, H, and D respectively (default: 1024x768x24)
The next time you want to get your hands dirty with some autopilot tests, try out the sandbox. I'm sure you'll find a very nice use for it in your workflow; after all wouldn't it be handy to run multiple testsuites at once?
As of today, we are exactly one month away from the release of Saucy Salamander. As part of that release, ubuntu is committed to delivering an image of ubuntu-touch, ready to install on supported devices.
And while folks have been dogfooding the images since May, many changes have continued to land as the images mature. As such, the qa team is committing to test each of the stable images released, and do exploratory testing against new features and specific packagesets.
If you have a device, I would encourage you to join this effort! Everything you need to know can be found upon this wiki page. You'll need a nexus device and a little time to spend with the latest image. If you find a bug, report it! The wiki has links to help. Testing doesn't get anymore fun than this; flash your phone and try to break it! Go wild!
And if you don't own a device? You can still help! As bugs are found and fixed, the second part of the process is to create automated tests for them so they don't occur again. Any bug you see on the list is a potential candidate, but we'll be marking those we especially think would be useful to write an autopilot tests for with a " touch-needs-autopilot" tag.
Join us in testing, confirming bugs, or testwriting autopilot tests. We want the ubuntu touch images to be the best they can be in 1 month's time. Happy Testing!
I've now had the pleasure of writing autopilot tests for about 9 months, and along the way I've learned or been taught some of the things that are important to remember.
The eventually matcher provided by autopilot is your best friend. Use it liberally to ensure your code doesn't fail because of a millisecond difference during your runtime. Eventually will retry your assert until it's true or it times out. When combined with examining an object or selecting one, eventually will ensure your test failure is a true failure and not a timing issue. Also remember you can use lambda if you need to make your assert function worthy.
Every test can use more asserts -- even my own! Timing issues can rear there ugly head again when you fail to assert after performing an action.
This month has been all about dogfooding, and part of that has been the drive to automate the testing of the functionality found in the core applications. Nothing will replace true user testing, but we can certainly do a lot to help prevent bugs and regressions. So, during my "test all the things" series we went through each of the core applications and highlighted areas for testing. So the question is, how are we doing as this month draws to a close?
Well, fortunately, I've setup this wiki page as a reflection of where we stand.
As you can see, we've come a long way on many of the applications, but I wanted to especially point out where we still need help.
Adrian has been working on Dropping Letters and could use some help in finishing the testcases. Leo has proposed a merge that brings the testsuite up to date with the new sdk, so it's ready for you to add tests!
RSS Reader, aka shorts, just went through some large UI changes and is now ready for us to add tests again. Carla has begun working on updating her old tests for adding and editing a feed, but there's more work to do!
Calendar is currently undergoing some drastic changes, and it's testcases will need to be re-written once that's complete. Hang tight if your keen to help in this area!
Music has come a long way in a couple weeks and we have tests to prove it! Thanks to Daniel and Victor, we can test play, pause and library load. But there are still more tests needed. John is also volunteering on this application to help get the tests in shape. Thanks guys!
The following applications are very close to completion:
Nekhelesh has hacked together most of the tests, but I know he won't complain if you help him finish. Review the list and help bring it over the line!
As of this morning, the final calculator testcase has been written, testing tear-off. Pending a bugfix in the sdk, the testcase will get merged. Fingers crossed. Thanks Riccardo!
Congratulations to the following teams and folks who helped us reach 100% status on the following core apps:
terminal, weather, sudoku, stock ticker, file manager
Remember though, as we move forward it's important to keep the tests aligned with new features :-)
Still reading this? What are you waiting for? Jump in and help! Happy Automated Testing!
A little over a month ago, I posted about creating an autopilot emulator for
ubuntu sdk applications. Thanks to some hard work by Leo Arias and the ubuntu sdk team, I'm happy to announce the little side project from my +junk branch is all grown up and even more ready for your consumption. Leo has made the emulator a proper part of the sdk, complete with tests for all of it's functions! You can now easily install it and incorporate it into your project. Here's how to snag your copy.
1) Add the ubuntu sdk team ppa if you haven't already
sudo add-apt-repository ppa:ubuntu-sdk-team/ppa
2) Install the ubuntu-ui-toolkit-autopilot package
sudo apt-get update && sudo apt-get install ubuntu-ui-toolkit-autopilot
This will install the emulator as a python module, ready for you to import. If you want to checkout what the module can do, have a look at the documentation.
Incorporating the module might seem a little tricky, so Leo has also put together an example of one of the ubuntu touch core apps using the new module. Check out it. Here's a branch showing off the work done for ubuntu-filemanager-app. And here's one for dropping-letters.
Please do check out the module and incorporate it into your ubuntu sdk project. Feedback is encouraged, and bug reports too! Please file any issues you find against the ubuntu-ui-toolkit project. Happy Automated Testing!
The music app. A vital and necessary piece of my running setup. I've run in silence before, and while it's nice, sometimes you need some jams to keep you motivated on longer runs.
Recently I've been running in silence but not by choice, definitely needing a way to play music via my phone. Thanks to the hack day for music last week and the ongoing work on the music app developers that's all changing. Today the app is feature complete enough to begin work on some autopilot tests. It can for instance, play music now!
Today I thought we'd take a break from the testing all the things series to take a look at what we're doing, how far we've come, and most importantly, where we need to get to.
The core apps project has come a long way since it's inception at the beginning of this cycle. So as the development started maturing, testing was the natural step for me to get involved and check out what the apps had to offer. With that in mind, let's look at the burndown chart for the core apps shall we?
It's almost FRIDAY! To celebrate, let's look at the second game that's become part of the core apps, dropping letters.
Today we look at another important piece of the puzzle for the ubuntu touch platform; the file manager. File managers provide a way for the end users to access there files in an arbitrary manner. Without one on the device you would be stuck viewing your data only through specific applications intended to utilize that data (photos, music, videos) or the terminal. Despite being the bane of simple computing advocates at times, file managers serve an important purpose.
So, let's look at the file manager app for ubuntu touch. Glancing at the dogfooding page you can see many of the basics are already in place. We can browse files and folders, make new folders and view file information.
The calculator app is probably the biggest fashion saving application (for me!) on the platform. You might be scratching your heads, so let me share a quick story.
You see not unlike the crazy pebble watch idea, there was a time when I wore a crazy watch on my wrist.
Ahh, Monday morning! It's time to look at the news headlines and check the stock market. As usual, your ubuntu phone has you covered for this part of your morning routine! Enter stock ticker!
As one of the newest core apps, Stock Ticker has done it's part in catching up quickly. The interface already sports working charting, news and detail views for stock symbols of your choosing via the management page.
"What's the weather like? Got a window? Open it!"
Or I suppose I could swipe my gorgeous ubuntu phone and see my weather with lovely icons. I may just decide to stay inside and watch my virtual sun bask it's artificial light upon my skin for hours. I'm only (half) kidding.
Still it's not hard to admit the weather app is shaping up to be one of the sharpest looking applications for ubuntu touch. Martin Borho has already gotten through many of the bugs listed so there is an excellent structure in place for you to add some further tests.
A big thank you to everyone who made it to the first workshop for automated testing! For those of you who couldn't make it, let me remind you we have 3 more scheduled.
Friday July 5th at 1300 UTC
Tuesday July 9th at 1800 UTC
Thursday July 11th at 2200 UTC
But I also wanted to leave you with a quick log from the session introduction for you to peruse, as well as this list of questions from the session. I hope this helps those of you trying to learn the ropes and contribute new tests!
I'm including some of the questions below as a bit of an FAQ for those starting the journey. Read the workshop introduction and the FAQ below and go automate all the things! I'll see you at the next workshop.
What do I need to develop tests?
A raring or saucy installation (VM or real) and the autopilot and ubuntu-sdk packages installed. You will also need an understanding of how autopilot works (or be willing to learn :-) )
How do I run/install a core app?
Once you've branched your core app source code, you don't need to install it in order to run it. However there is a ppa with all the core apps you can install. To run the core app from source, run it like so in the root directory:
qmlscene APPNAME.qml, ie qmlscene dropping-letters.qml
In order to install from the ppa, follow the info here: https://wiki.ubuntu.com/Touch/CoreApps/PPA
You can install all the core apps run them as you would any other application. After installing from the ppa, simply run the application name, ie "dropping-letters"
I recieved an error installing from the ppa; qtdeclarative5-* missing, etc
You are missing the ubuntu-sdk and related packages. Install them using
sudo add-apt-repository ppa:canonical-qt5-edgers/qt5-proper && sudo add-apt-repository ppa:ubuntu-sdk-team/ppa && sudo apt-get update && sudo apt-get install ubuntu-sdk
Can we write tests for qml apps using autopilot using precise or quantal?
No, the ubuntu-sdk and the new version of autopilot we're utilizing both require raring and preferably saucy to work.
Where can I see an example of autopilot tests?
The tutorial on developer.ubuntu.com is an excellent first step for seeing an autopilot test in action and seeing an explanation of the test and how it works. In addition, the file manager, clock, weather, calendar and weather core apps already have some autopilot tests written as of this writing.
How much python does one need to know in order to write autopilot tests?
Not as much as you think :-) If you are familiar with programming and can understand and use the basic autopilot functions and the ubuntu sdk emulator, writing a test won't require you to learn any fancy python.
When your like me and work with wonderful people across the world, knowing what time it is turns out to be really important. It's important to know what time it is in my timezone and the timezone of others I work with around the world.
Even sticking with my own timezone, I use my my clock to wake me each morning, keep track of my running, and let me time things while cooking (proper eggs anyone?) :-) Nekhelesh Ramananthan and the other clock developers are tackling all of these problems with the clock app. Sadly unlike sudoku, I haven't found a way how to cheat time.
Recently we've been on a campaign to help increase the amount of automated tests we have for ubuntu. Specifically the effort is focused around helping out our community developers on the core apps project. The core apps project is building the core applications for ubuntu touch. Excellent stuff, all being done by the community!
The "testing all the things" blog series is currently covering each of these core applications and ends with a call to help the development teams. I've linked to tutorials like this and this on autopilot providing what you need to know. But sometimes seeing is understanding, and a helping hand can go a long way.
With that in mind, I am announcing a series of workshops to help you gather the skills needed to write automated tests. You can help contribute with just your ubuntu pc, writing and running tests without needing phone hardware! We're going to focus on autopilot, and for the moment the ubuntu core apps. I'll try and alternate to host them at timezone friendly times for everyone (granted I do have to sleep at some point too!). Here's the schedule, with links to the event on G+ page.
Tomorrow!, Wednesday July 3rd at 1800 UTC
Friday July 5th at 1300 UTC
Tuesday July 9th at 1800 UTC
Thursday July 11th at 2200 UTC
The workshops will take place in #ubuntu-quality and will all last an hour (but I won't leave you hanging if we need more time!). I'll host g+ hangouts and provide one on one help as needed to anyone writing tests. See you at the workshops!
In honor of the closing of google reader, I thought I would highlight another core application that needs some attention; namely the RSS Reader, with a proper name Shorts. If your already bored and yawning (RSS is dead, long live RSS), have a look at the design's recently shared by the design team as well as the original post with the user stories. Seems like RSS might not be so dead (or look it!)!
Yes, I still use RSS feeds, mainly as a news aggregator. In many ways honestly RSS feeds have long replaced my idea of bookmarking things. Bookmarks are general stale old content that never updates, is never refreshed and is eventually just purged. The ideas shown in the design of Shorts are great and the development team has a wonderful task ahead of them of implementing them.
© 2010 Canonical Ltd. Ubuntu and Canonical are registered trademarks of Canonical Ltd.