Continuing our discussion of testing within ubuntu, today's post will talk about how you can help the community core apps stay healthy.
As you recall the core apps go through a series of QA before being released to the store. However bugs in the application, or in the platform itself can still be exposed. The end result is that the dashboard contains tests failures for that application. To release a new stable image, we need a green dashboard, and more importantly we need to make sure the applications work properly.
Getting plugged in
So to help out, it's important to first plug into the communication stream. After all, we're building these applications and images as a community! First, join the ubuntu phone group on launchpad and sign up for the phone mailing list. The list is active and discussing all issues pertaining to the ubuntu phone. Most importantly, you will see landing team emails that summarize and coordinate issues with the phone images.
From there you can choose a community core app to help improve from a quality perspective. These applications all have development teams and it's helpful to stay in contact with them. Your merge proposal can serve as an introduction!
Finding something to work on
So what needs fixing? A landing team email might point out a failing test. You might notice a test failure on the dashboard yourself. In addition each application keeps a list of bugs reported against it, including bugs that point out failing tests or testing needs. For example here's the list of all new autopilot tests that need to be written for all of the core apps. Pick an app, browse the buglist, assign a bug to yourself, and fix it.
For example, here's the list of bugs for music app. As of this writing you can see several tests that need written, as well as a bug for a test improvement.
You can also simply enhance the app's existing testsuite by fixing a flaky test, or improving the test to use best practices, etc. As a bonus for those reading this near it's original publication date, we just had a session @ vUDS covering the core apps and the testing needs we have. Watch the session / browse the pad and pick something to work on.
Look into any failures you find and have a look at the tests. Often the tests can use a little improvement (or maybe an additional test), and you can help out here! Sometimes failures won't happen every run -- this is the sign of a weird bug, or more likely a flaky test. Fix the test(s), improve them, or add to them. Then commit your work and submit a merge proposal. Follow the guide on the wiki if you need help with doing this.
Remember, you can iteratively run the tests on your device as you work. Read my post on click-buddy for help with this. If you are lacking a device, run the tests on your desktop instead and a reviewer can test against a real device before merging.
For realtime help, check out #ubuntu-quality and #ubuntu-autopilot on freenode. You'll find a group of folks like yourself working on tests, hacking on autopilot and sharing advice. If IRC isn't your thing, feel free to contact us through another method instead. Happy hacking!
Continuing our discussion of the testing within ubuntu, today's post will talk about how you can help ubuntu stay healthy by manually testing the images produced. No amount of robots or automated testing in the world can replace you (well, at least not yet, heh), and more specifically your workflow and usage patterns.
As discussed, everyday new images are produced for ubuntu for all supported architecture types. I would encourage you to follow along and watch the progression of the OS through these images and your testing. Every data point matters and testing on a regular basis is helpful. So how to get started?
|Settle in with a nice cup of tea while testing!|
Since just before the last LTS, quality has been a buzzword within the ubuntu community. We've come a long way since precise and I wanted to provide some help and prospective on what ubuntu's process for quality looks like this cycle. In simple terms. Or as reddit would say, "explain to me like I'm 5".
I'll try and define terms as we go. First let me define CI, which is perhaps the buzzword of this cycle, lest I lose all of you! CI stands for continuous integration, and it means we are testing ubuntu. All the time. Non-stop. Every change we make, we test. The goal behind this idea is to find and fix problems, before well, they become problems on your device!
The CI dashboard then is a way to visually see the results of this testing. It acts as a representation of the health of ubuntu as a distribution. At least once a day runs are executed, apps and images are tested and benchmarked, and the results are populated on ci.ubuntu.com. This is perhaps the most visible part of the CI (our continuous testing efforts) that is happening within ubuntu. But let's step back a minute and look at how the overall CI process works within ubuntu.
App developers hack on a bit of code, fixing bugs or adding new features to the codebase. Once the code is ready, a merge proposal1 is created by the developer and feedback is sought. If the code passes the peer review and the application's tests, it will then become part of the image after a journey through the CI train.
|Though menacing, Alan assures me he doesn't bite|
|Adopted images are healthy images!|
Recently the ubuntu core app developers and myself have been on an adventure towards adopting cmake for the all the core applications. While some of the applications are pure qml it's still been useful to embark on adopting a singular build system for all of the projects. Now that (most) of the pain of transitioning is gone, I'm going to talk about one of the useful features of setting up cmake for your project; click-buddy!
Click-buddy is an evolving tool that helps you build and deploy click packages to your phablet device. In addition, it has the ability to setup the device to run your autopilot test suite. So, rather than writing anything further, let's cover an example. You are going to need phablet-tools installed for this to work. I'm going to branch the clock app, build the click package, install it on my device, and finally run the tests.
bzr branch lp:ubuntu-clock-app
click-buddy --dir . --provision
phablet-test-run -v ubuntu_clock_app
Click-buddy is also gaining the ability to build your project, even it involves a plugin and you are interested in building for your phablet device (armhf). Once landed you will be able to run something like this for non-qml applications.
sudo click chroot -a armhf create
click-buddy --arch armhf --provision
This will setup a chroot automagically for you and compile and build your application. Give it a try!
Note, as of this writing, emulator support for the ubuntu-ui-toolkit emulator is not yet built in. If your tests fail with a module import, run this line from your connected pc. It will copy over the ubuntu-ui-toolkit emulator (provided you have it installed on your pc :-) ) and your tests should now properly run.
adb push /usr/lib/python2.7/dist-packages/ubuntuuitoolkit /home/phablet/autopilot/ubuntuuitoolkit
First things first, we have our burndown up for the trusty cycle! If your name appears on the list next to a work item, congratulations! Let's work through the items and make it happen ;-) If your name doesn't appear, don't worry! Feel free to help complete the tasks anyway, or jump in an get started on something you love. The mailing list is always open and we love to hear new ideas. See a blueprint that looks interesting and want to help? Get in touch!
What's happening now?
Now, for an update as to what is going on in quality. This week starts Alpha 1 for the flavors that are participating. In addition, the first work items are starting to see completion.
Ali has been working on getting the wiki pages and workflow ready for our community calls for testing. You can check out the page on the wiki here, and schedule testing events. The page explains the basic workflow -- I'd encourage you to schedule events for your favorite packages and let others know about the event on the mailing list. Be sure and report the results once you've completed.
Alberto has been working on getting the papercuts project running for the trusty cycle and ensuring the buglist is maintained and accessible.
Dan has been working on getting our automated testing in place and we're meeting with the CI team this week to get this implemented as part of the release process. Excellent work Dan!
Dan has also been working on the autopilot gtk emulator, which seeks to do the same thing as we've done with the autopilot ubuntu sdk emulator. Both of these frameworks help make writing tests easier.
Speaking of tests, Carla and others have been busy adding and fixing tests to make sure our dashboard stays green. And look at it! We're over 99% passing tests for the ubuntu touch platform. I want to personally thank everyone who's helped write and maintain these tests. Kudos to you all!
On the desktop side, Dan is helping new folks land tests, and the ubuntu desktop default apps continue to run and provide feedback on the desktop.
We also saw several new members join the ubuntu community via ubuntu quality recently. Congrats to Jean-Baptiste, Dan Chapman and Daniel Kessel (my apologies if I missed you!).
Finally we have Thibaut and other folks looking at refactoring and improving ubuntu startup disk creator. Checkout the full blueprint for more information.
So, what's next?
Well for most of us, it's the Holiday season. Looking beyond the Holidays, , there's several big items in my list to work on in January:
Since I last posted on the topic of our communities future, we've helped push out a new release of ubuntu. Saucy and fresh, the little salamander is now out in the wild. With the release out it's time to move forward with these changes.
|(c) Carla Sella|
We're still here and we're still testing!
As I eluded to in my previous post, it's time for us as a community to fully embrace the place of automated testing in our workflows. Over the past year we've learned how to write testcases and to then apply that knowledge towards writing autopilot tests. At the same time ubuntu has been building testing infrastructure and launching a CI team to help run and maintain it.
With these changes and our acquired knowledge we can construct a sustainable vision for testing. So just what is that vision?
Let's make manual testing fun again. As we've ramped up our workloads, we find ourselves repetitively testing the same packages again and again (and running the same tests!). We'd call this regression testing and it's a perfect candidate for automated testing. Let's automate these tests and keep our focus on where we as humans excel.
In addition, we as a community participate in "calls for testing". These calls have become more and more exploratory in nature. In general, we have moved further away from a defined testcase ("perform steps 1, 2, and 3"). Instead, we encourage testers to adopt a try and break it attitude. This is where we as humans can excel, and you know what, have some fun at trying! Remember QA is the only place we encourage you to break things!
So let's get manual testing back to the exploratory testing we love.
|Thank you little testing robot!|
|All test and no play makes for a sad image|
With all the buzz and excitement around quality over the last few cycles, we've been busy. We've built tools, started processes, wrote testcases and committed ourselves to testing. When the time came for us to write new applications for the ubuntu touch platform, we took our quality aspirations with us and tested all through development. The "core apps" for the phone is a perfect example of this; community developed software including tests. Kudos to everyone who is taking part in these efforts.
However, as more and more tests came online, new problems developed. How can we run these tests? When do we want to run them? How can we develop more? What tools do we use? The quality team met all of these challenges, but the weight started to grow heavier.
|Rob Macklem Victoria BC derivative: Plasticspork [CC-BY-SA-3.0]|
|Ivanko Barbell Company|
Autopilot has continued to change with the times, and the pending release of 1.4 brings even more goodies; including some performance fixes. But today I wanted to cover a newly landed feature from the minds of Martin and Jean-Baptiste (thanks guys!).
If you've developed autopilot tests in the past you will have noticed how cumbersome it can be to run the tests. If you run a test on your desktop, you lose control of your mouse and keyboard for the duration, and you might even accidentally cause a test to fail. This can be especially noticeable when you are iterating over getting your test to run "just right" while wanting to keep your introspection tree in vis open, or reviewing someone else's code while wanting to verify the tests run. Enter sandbox mode.
A new command called autopilot-sandbox-run lets you easily run a testsuite inside your choice of two sandboxes; Xvfb by default, or if you want to visually see the output, Xephyr. Have a quick look at the command options below as of this writing:
Usage: autopilot-sandbox-run [OPTIONS...] TEST [TEST...]
Runs autopilot tests in a 'fake' Xserver with Xvfb or Xephyr. autopilot runs
in Xvfb by default.
TEST: autopilot tests to run
-h, --help This help
-d, --debug Enable debug mode
-a, --autopilot ARG Pass arguments ARG to 'autopilot run'
-X, --xephyr Run in nested mode with Xephyr
-s, --screen WxHxD Sets screen width, height, and depth to W, H, and D respectively (default: 1024x768x24)
The next time you want to get your hands dirty with some autopilot tests, try out the sandbox. I'm sure you'll find a very nice use for it in your workflow; after all wouldn't it be handy to run multiple testsuites at once?
As of today, we are exactly one month away from the release of Saucy Salamander. As part of that release, ubuntu is committed to delivering an image of ubuntu-touch, ready to install on supported devices.
And while folks have been dogfooding the images since May, many changes have continued to land as the images mature. As such, the qa team is committing to test each of the stable images released, and do exploratory testing against new features and specific packagesets.
If you have a device, I would encourage you to join this effort! Everything you need to know can be found upon this wiki page. You'll need a nexus device and a little time to spend with the latest image. If you find a bug, report it! The wiki has links to help. Testing doesn't get anymore fun than this; flash your phone and try to break it! Go wild!
And if you don't own a device? You can still help! As bugs are found and fixed, the second part of the process is to create automated tests for them so they don't occur again. Any bug you see on the list is a potential candidate, but we'll be marking those we especially think would be useful to write an autopilot tests for with a " touch-needs-autopilot" tag.
Join us in testing, confirming bugs, or testwriting autopilot tests. We want the ubuntu touch images to be the best they can be in 1 month's time. Happy Testing!
As mentioned in my last post, Mir is one of the biggest changes coming in 13.10. With feature freeze now happening this week, it's time to amp up our testing engines once more to test the final features and help land Mir into the archive.
The Mir team has put together both a ppa and wiki page that contains all the information you need to help with testing. The testing window closes in 2 days on August 28th, just in time for feature freeze. The biggest changes for Mir are the inclusion of multi-monitor support and thus are a focus for this testing. So here's the details you need to know.
Help test Mir using your current system, ubuntu saucy and the Mir team ppa.
Now through August 28th.
The full instructions for installing the ppa, running the tests, and reporting the results can be found on this wiki page. Results are reported on this page or via the package tracker testing page.
Thank you for your contributions! Good luck and Happy Testing Everyone!
So with all the automated testing buzz occurring in the quality world of ubuntu this cycle, I wanted to speak a little about why we're doing the work, and what the output of the work looks like.
So, why? Why go through the trouble of writing tests for the code you write? I'll save everyone a novel and avoid re-hashing the debate about whether testing is a proper use of development time or not. Simply put, for developers, it will prevent bugs from reaching your codebase, alleviate support and maintenance burdens, and will give you more confidence and feedback during releases. If you want your application to have that extra polish and completeness, testing needs to be a part. For users, the positives are similar and simple. Using well tested software prevents regressions and critical bugs from affecting you. Every bug found during testing is a bug you don't have to deal with as an end user. Coincidentally, if you like finding bugs in software, we'd love to have you on the team :-)
In general, three technologies are being used for automated testing.
Autopilot, Autopkg, and QML tests. Click to learn more about writing tests for each respectively.
|Any color is good, as long as it's green|
|Sorry robot, all our tests are belong to us|
We're already approaching feature freeze at a quickening pace, and thus the next few weeks are rather important to us as a testing community. 13.10 is landing in October, which is now rapidly approaching (where did the summer go?!).
Let's run through some manual tests for ubuntu and flavors. I'd like to ask for a special focus to be given to Mir/xMir. We plan to have a rigorous test of the package again in about a week once all features have landed. In the interim, let's try and catch any additional bugs.
This week Saturday August 17th through Saturday August 24th. It's week 5 for our cadence tests.
Ok, I'm sold, what do I need to do?
Execute some testcases against the latest version of saucy; in particular the xMir test.
Got any instructions?
You bet, have a look at the Cadence Week testing walkthrough on the wiki, or watch it on youtube. If you get stuck, contact us.
Where are the tests?
You can find the Mir test in it's own milestone here. Remember to read and follow the installation instructions link at the top of the page!
The rest of the applications and packages can be found here.
I don't want to run/install the unstable version of ubuntu, can I still help?
YES! Boot up the livecd on your hardware and run the live session, or use a virtual machine to test (install ubuntu or use a live session). The video demonstrates using a virtual machine booting into a live session to run through the tests. For the Mir/xMir tests, however, we'd really like results from real hardware.
But, virtual machines are scary; I don't know how to set one up!
There's a tool called testdrive that makes setting up a vm with ubuntu development a point and click operation. You can then use it to test. Seriously, check out the video and the walkthrough for more details.
Thank you for your contributions! Good luck and Happy Testing Everyone!
I've now had the pleasure of writing autopilot tests for about 9 months, and along the way I've learned or been taught some of the things that are important to remember.
The eventually matcher provided by autopilot is your best friend. Use it liberally to ensure your code doesn't fail because of a millisecond difference during your runtime. Eventually will retry your assert until it's true or it times out. When combined with examining an object or selecting one, eventually will ensure your test failure is a true failure and not a timing issue. Also remember you can use lambda if you need to make your assert function worthy.
Every test can use more asserts -- even my own! Timing issues can rear there ugly head again when you fail to assert after performing an action.
This month has been all about dogfooding, and part of that has been the drive to automate the testing of the functionality found in the core applications. Nothing will replace true user testing, but we can certainly do a lot to help prevent bugs and regressions. So, during my "test all the things" series we went through each of the core applications and highlighted areas for testing. So the question is, how are we doing as this month draws to a close?
Well, fortunately, I've setup this wiki page as a reflection of where we stand.
As you can see, we've come a long way on many of the applications, but I wanted to especially point out where we still need help.
Adrian has been working on Dropping Letters and could use some help in finishing the testcases. Leo has proposed a merge that brings the testsuite up to date with the new sdk, so it's ready for you to add tests!
RSS Reader, aka shorts, just went through some large UI changes and is now ready for us to add tests again. Carla has begun working on updating her old tests for adding and editing a feed, but there's more work to do!
Calendar is currently undergoing some drastic changes, and it's testcases will need to be re-written once that's complete. Hang tight if your keen to help in this area!
Music has come a long way in a couple weeks and we have tests to prove it! Thanks to Daniel and Victor, we can test play, pause and library load. But there are still more tests needed. John is also volunteering on this application to help get the tests in shape. Thanks guys!
The following applications are very close to completion:
Nekhelesh has hacked together most of the tests, but I know he won't complain if you help him finish. Review the list and help bring it over the line!
As of this morning, the final calculator testcase has been written, testing tear-off. Pending a bugfix in the sdk, the testcase will get merged. Fingers crossed. Thanks Riccardo!
Congratulations to the following teams and folks who helped us reach 100% status on the following core apps:
terminal, weather, sudoku, stock ticker, file manager
Remember though, as we move forward it's important to keep the tests aligned with new features :-)
Still reading this? What are you waiting for? Jump in and help! Happy Automated Testing!
A little over a month ago, I posted about creating an autopilot emulator for
ubuntu sdk applications. Thanks to some hard work by Leo Arias and the ubuntu sdk team, I'm happy to announce the little side project from my +junk branch is all grown up and even more ready for your consumption. Leo has made the emulator a proper part of the sdk, complete with tests for all of it's functions! You can now easily install it and incorporate it into your project. Here's how to snag your copy.
1) Add the ubuntu sdk team ppa if you haven't already
sudo add-apt-repository ppa:ubuntu-sdk-team/ppa
2) Install the ubuntu-ui-toolkit-autopilot package
sudo apt-get update && sudo apt-get install ubuntu-ui-toolkit-autopilot
This will install the emulator as a python module, ready for you to import. If you want to checkout what the module can do, have a look at the documentation.
Incorporating the module might seem a little tricky, so Leo has also put together an example of one of the ubuntu touch core apps using the new module. Check out it. Here's a branch showing off the work done for ubuntu-filemanager-app. And here's one for dropping-letters.
Please do check out the module and incorporate it into your ubuntu sdk project. Feedback is encouraged, and bug reports too! Please file any issues you find against the ubuntu-ui-toolkit project. Happy Automated Testing!
© 2010 Canonical Ltd. Ubuntu and Canonical are registered trademarks of Canonical Ltd.