While much of the excitement around unity8 and the next generation of ubuntu has revolved around mobile, again I'd like to point your attention to the desktop. The unity8 desktop is starting to evolve and gain more "desktopy" features. This includes things like window management and keyboard shortcuts for unity8, and MIR enhancements with things like native library support for rendering and support for X11 applications.
I hosted a session with Stephen Webb at UOS last year where we discussed the status of running unity8 on the desktop. During the session I mentioned my own personal goal of having some brave community members running unity8 as there default desktop this cycle. Now, it's still a bit early to realize that goal, but it is getting much closer! To help get there, I would encourage you to have a look at unity8 on your desktop and start running it. The development teams are ready for feedback and anxious to get it in shape on the desktop.
So how do you get it? Check out the unity8 desktop wiki page which explains how you can run unity8, even if you are on a stable version of ubuntu like the LTS. Install it locally in an lxc container and you can login to a unity8 desktop on your current pc. Check it out! After you finish playing, please don't forget to file bugs for anything you might find. The wiki page has you covered there as well. Enjoy unity8!
I thought I would add a little festivity to the holiday season, quality style. In case your holidays just are not the same without a little quality in your life, allow me to share how you can get involved.
There are opportunities for every role listed on the QA wiki. Testers and test writers are both needed. Testing and writing manual tests can be learned by anyone, no coding required. That said if you have skills or interest in technical work, I would encourage you help out. You will learn by doing and get help from others while you do it.
Now onto the good stuff! What can you do to help ubuntu this cycle from a quality perspective?
There is an ever present need for brave folks willing to simply run the development version of ubuntu and use it as a daily machine throughout the cycle. It's one of the best ways for us as a community to uncover bugs and issues, in particular things that regress from the previous release. Upgrade to vivid today and see what you can break!
This tool is written in drupal7 and runs the iso.qa.ubuntu.com and packages.qa.ubuntu.com sites. These sites are used to record and view the results of all of our manual testing efforts. Currently dkessel is leading the effort on implementing some needed UI changes. The code and more information about the project can be found on launchpad. The tracker is one of our primary tools and needs your help to become friendly for everyone to use.
In addition a charm would be useful to simplify setting up a development environment. The charm can be based upon the existing drupal charm. At the moment this work is ready for someone to jump in.
Running unity8 as a full-time desktop is a personal goal I have for this cycle. I hope some others might also want to be early adopters and join me in this goal. For now you can help by testing the unity8 desktop. Have a look at running unity in lxc for an easy way to run unity8 today on your machine. Use it, test it, and offer feedback. I'll be talking more about unity8 as the cycle progresses and opportunities to test new features aimed at the desktop appear.
The core apps project is an excellent way to get involved. These applications have been lovingly developed by community members just like you. Many of the teams are looking for help in writing tests and for someone who can help bring a testing mindset and eye to the work. As of this writing specifically the docviewer, terminal and calculator teams would love your help. The core apps hackdays are happening this week, drop by and introduce yourself to get started!
Like the sound of writing tests but the idea of writing code turns you off? Manual tests are needed as well! They are written in English and are easy to understand and write. Manual tests include everything you see on the qatracker and are managed as a launchpad project. This means you can pick a bug and "fix it" by submitting a merge request. The bugs involve both fixing existing tests as well as requests for new testcases.
As always there are images that need testing. Testing milestones occur later in the cycle which involve everyone helping to test a specific set of images. In the meantime, daily images are generated that have made it through the automated tests and are ready for manual testing. Booting an image in a live session is a great way to check for regressions on your machine. Doing this early in the cycle can help make sure your hardware and others like it experience a regression free upgrade when the time comes.
After subjecting software to testing, bugs are naturally found. These bugs then need to be verified and triaged. The bugsquadders, as they are called, would be happy to help you learn to categorize or triage bugs and do other tasks.
No matter how you choose to get involved, feel free to contact me for help if needed. Most of all, Happy Testing!
The final images of what will become utopic are here! Yes, in just one short week utopic unicorn will be released into the world. Celebrate this exciting release and be among the first to run utopic by helping us test!
We need your help and test results, both positive and negative. Please head over to the milestone on the isotracker, select your favorite flavor, and perform the needed tests against the images.
If you've never submitted test results for the iso tracker, check out the handy links on top of the isotracker page detailing how to perform an image test, as well as a little about how the qatracker itself works. If you still aren't sure or get stuck, feel free to contact the qa community or myself for help.
Thank you for helping to make ubuntu better! Happy Testing!
In my last next post, I discussed will discuss notable autopilot features and talk about how autopilot has matured since it became an independent project.
In the meantime I would be remiss if I didn't also talk about the different test runners commonly used with autopilot tests. In addition to the autopilot binary which can be executed to run the tests, different tools have cropped up to make running tests easier.
This tool ships with autopilot itself and was developed as a way to run autopilot test suites on your desktop in a sane manner. Run the autopilot3-sandbox-run command with --help to see all the options available. By default, the tests will run in an Xvfb server, all completely behind the scenes with the results being reported to you upon completion. This is a great way to run tests with no interference on your desktop. If you are a visual person like me, you may instead wish to pass -X to enable the test runs to occur in a Xephyr window allowing you to see what's happening, but still retaining control of your mouse and keyboard.
I need this tool!
sudo apt-get install python3-autopilot
I want to run tests on my desktop without losing control of my mouse!
I want to run tests on my desktop without losing control of my mouse, but I still want to see what's happening!
autopilot3-sandbox-run -X my_testsuite_name
Autopkgtest was developed as a means to automatically test Debian packages, "as-installed". Recently support was added to also test click packages and to run on phablet devices. Autopkgtest will take care of dependencies, setting up autopilot, and unlocking the device. You can literally plug in a device and wait for the results. You should really checkout the README pages, including those on running tests. That said, here's a quick primer on running tests using autopkgtest.
I need this tool!
sudo apt-get install autopkgtest
If you are on trusty, grab and install the utopic deb from here.
I want to run tests for a click package installed on my device!
Awesome. This one is simple. Connect the device and then run:
adt-run --click my.click.name --- ssh -s adb
adt-run --click com.ubuntu.music --- ssh -s adb
will run the tests for the installed version of the music app on your device. You don't need to do anything else. For the curious, this works by reading the manifest file all click packages have. Read more here.
I want to run the tests I wrote/modified against an installed click package!
For this you need to also pass your local folder containing the tests. You will also want to make sure you installed the new version of the click package if needed.
adt-run my-folder/ --click my.click.name --- ssh -s adb
Autopkgtest can also run in a lxc container, QEMU, a chroot, and other fun targets. In the examples above, I passed --- ssh -s adb as the target, instructing autopkgtest to use ssh and adb and thus run the tests on a connected phablet device. If you want to run autopilot tests on a phablet device, I recommend using autopkgtest as it handles everything for you.
This tool is part of the greater phablet-tools package. It was originally developed as an easy way to execute tests on your phablet device. Note however that copying the tests and any dependencies to the phablet device is left to you. The phablet-tools package provides some other useful utilities to help you with this (checkout phablet-click-test-setup for example).
I need this tool!
sudo apt-get install phablet-tools
I want to run the tests I wrote/modified against an installed click package!
First copy the tests to the device. You can use the ubuntu sdk or click-buddy for this, or even do it manually via adb. Then run phablet-test-run. It takes the same arguments as autopilot itself.
phablet-test-run -v my_testsuite
Note the tools looks for the testsuite and any dependencies of the testsuite inside the /home/phablet/autopilot folder. It's up to you to make sure everything that is needed to run your tests are located there or else it will fail.
There are of course other possible test runners that wrap around autopilot to make executing tests easier. Perhaps you've written a script yourself. Just remember at the end of the day the autopilot binary will be running the tests. It simply needs to be able to find the testsuite and all of it's dependencies in order to run. For this reason, don't be afraid to execute autopilot3 and run the tests yourself. Happy test runs!
How acceptance tests are packaged and run has morphed over time. When autopilot was originally conceived the largest user was the unity project and debian packaging was the norm. Now that autopilot has moved well beyond that simple view to support many types of applications running across different form factors, it was time to address the issue of how to run and package these high-level tests.
While helping develop testsuites for the core apps targeting ubuntu touch, it became increasingly difficult for developers to run their application's testsuites. This gave rise to further integration points inside qtcreator, enhancements to click and its manifest files, and tools like the phablet-tools suite and click-buddy. All of these tools operate well within the confines they are intended, but none truly meets the needs for test provisioning and execution.
With these thoughts in mind I opened the floor for discussion a couple months ago detailing the need for a proper tool that could meet all of my needs, as well as those of the application developer, test author and CI folks. In a nutshell, a workflow to setup a device as well as properly manage dependencies and resolve them was needed.
Autopkg tests all the things
I'm happy to report that as of a couple weeks ago such a tool now exists in autopkgtest. If the name sounds familar, that's because it is. Autopkgtest already runs all of our automated testing at the archive level. New package uploads are tested utilizing its toolset.
So what does this mean? Utilizing the format laid out by autopkgtest, you can now run your autopilot testsuite on a phablet device in a sane manner. If you have test dependencies, they can be defined and added to the click manifest as specified. If you don't have any test dependencies, then you can run your testsuite today without any modifications to the click manifest.
Yes, but what does this really mean?
This means you can now run a testsuite with adt-run in a similar manner to how debian packages are tested. The runner will setup the device, copy the tests, resolve any dependencies, run them, and report the results back to you.
Support for running tests this way is still new. If you do find a bug, please file it!
To use the tool first install autopkgtest. If you are running trusty, the version in the archive is old. For now download the utopic deb file and install it manually. A proper backport still needs to be done.
Also as of this writing, I must caution you that you may run into this bug. If the application fails to download dependencies (you see 404 errors during setup), update your device to the latest image and try again. Note, the latest devel image might be too old if a new image hasn't been promoted in a long time.
I want to see it!
Go ahead, give it a whirl with the calendar application (or your favorite core app). Plug in a device, then run the following on your pc.
bzr branch lp:ubuntu-calendar-app
adt-run ubuntu-calendar-app --click=com.ubuntu.calendar --- ssh -s /usr/share/autopkgtest/ssh-setup/adb
Autopkgtest will give you some output along the way about what is happening. The tests will be copied, and since --click= was specified, the runner will use the click from the device, install the click in our temporary environment, and read the click manifest file for dependencies and install those too. Finally, the tests will be executed with the results returned to you.
Please try running your autopilot testsuites this way and give feedback! Feel free to contact myself, the upstream authors (thanks Martin Pitt for adding support for this!), or simply file a bug. If you run into trouble, utilize the -d and the --shell switches to get more insight into what is happening while running.
We're having our first hackfest of the utopic cycle this week on Tuesday, July 15th. You can catch us live in a hangout on ubuntuonair.com starting at 1900 UTC. Everything you need to know can be found on the wiki page for the event.
During the hangout, we'll be demonstrating writing a new manual testcase, as well as reviewing writing automated testcases. We'll be answering any questions you have as well about contributing a testcase.
We need your help to write some new testcases! We're targeting both manual and automated testcase, so everyone is welcome to pitch in.
We are looking at writing and finishing some testcases for ubuntu studio and some other flavors. All you need is some basic tester knowledge and the ability to write in English.
If you know python, we are also going to be hacking on the toolkit helper for autopilot for the ubuntu sdk. That's a mouthful! Specifically it's the helpers that we use for writing autopilot tests against ubuntu-sdk applications. All app developers make use of these helpers, and we need more of them to ensure we have good coverage for all components developers use.
Don't worry about getting stuck, we'll be around to help, and there's guides to well, guide you!
Hope to see everyone there!
The first testing day of the utopic cycle is coming this week on Thursday, July 10th. You can catch us live in a hangout on ubuntuonair.com starting at 1900 UTC. We'll be demonstrating running and testing the development release of ubuntu, reporting test results, reporting bugs, and doing triage work. We'll also be availible to answer your questions and help you get started testing as well.
Please join us in testing utopic and helping the next release of ubuntu become the best it can be. Hope to see everyone there!
P.S. We have a team calendar that can help you keep track of the release schedule along with this and other events. Check it out!
Building click packages should be easy. And to a reasonable extent, qtcreator and click-buddy do make it easy. Things however can get a bit more complicated when you need to build a package that needs to run on an armhf device (you know like your phone!). Since your pc is almost certainly based on x86, you need to use, create or fake an armhf environment for building the package.
So then what options exist for getting a proper build of a project that will install properly on your device?
A phone can be more than a phone
It can also be a development environment!? Although it's not my recommendation, you can always use the source device to compile the package with. The downsides of this is namely speed and storage space. Nevertheless, it will build a click.
As promised, here is your reminder that we are indeed fast approaching the final image for trusty. It's release week, which means it's time to put your energy and focus into finding and getting the remaining bugs documented or fixed in time for the release.
We need you!
The images are a culmination of effort from everyone. I know many have already tested and installed trusty and reported any issues encountered. Thank you! If you haven't yet tested, we need to hear from you!
How to help
The final milestone and images are ready; click here to have a look.
Execute the testcases for ubuntu and your favorite flavor images. Install or upgrade your machine and keep on the lookout for any issues you might find, however small.
I need a guide!
Sound scary? It's simpler than you might think. Checkout the guide and other links at the top of the tracker for help.
I got stuck!
Help is a simple email away, or for real-time help try #ubuntu-quality on freenode. Here are all the ways of getting ahold of the quality team who would love to help you.
Plan to help test and verify the images for trusty and take part in making ubuntu! You'll join a community of people who do there best everyday to ensure ubuntu is an amazing experience. Here's saying thanks, from me and everyone else in the community for your efforts. Happy testing!
|Say that three times fast. Time to test trusty,|
time to test trusty, time to test trusty!
|The release team is ready to pounce on untested images|
Continuing our discussion of the testing within ubuntu, today's post will talk about how you can help ubuntu stay healthy by manually testing the images produced. No amount of robots or automated testing in the world can replace you (well, at least not yet, heh), and more specifically your workflow and usage patterns.
As discussed, everyday new images are produced for ubuntu for all supported architecture types. I would encourage you to follow along and watch the progression of the OS through these images and your testing. Every data point matters and testing on a regular basis is helpful. So how to get started?
|Settle in with a nice cup of tea while testing!|
Recently the ubuntu core app developers and myself have been on an adventure towards adopting cmake for the all the core applications. While some of the applications are pure qml it's still been useful to embark on adopting a singular build system for all of the projects. Now that (most) of the pain of transitioning is gone, I'm going to talk about one of the useful features of setting up cmake for your project; click-buddy!
Click-buddy is an evolving tool that helps you build and deploy click packages to your phablet device. In addition, it has the ability to setup the device to run your autopilot test suite. So, rather than writing anything further, let's cover an example. You are going to need phablet-tools installed for this to work. I'm going to branch the clock app, build the click package, install it on my device, and finally run the tests.
bzr branch lp:ubuntu-clock-app
click-buddy --dir . --provision
phablet-test-run -v ubuntu_clock_app
Click-buddy is also gaining the ability to build your project, even it involves a plugin and you are interested in building for your phablet device (armhf). Once landed you will be able to run something like this for non-qml applications.
sudo click chroot -a armhf create
click-buddy --arch armhf --provision
This will setup a chroot automagically for you and compile and build your application. Give it a try!
Note, as of this writing, emulator support for the ubuntu-ui-toolkit emulator is not yet built in. If your tests fail with a module import, run this line from your connected pc. It will copy over the ubuntu-ui-toolkit emulator (provided you have it installed on your pc :-) ) and your tests should now properly run.
adb push /usr/lib/python2.7/dist-packages/ubuntuuitoolkit /home/phablet/autopilot/ubuntuuitoolkit
Since I last posted on the topic of our communities future, we've helped push out a new release of ubuntu. Saucy and fresh, the little salamander is now out in the wild. With the release out it's time to move forward with these changes.
|(c) Carla Sella|
As I eluded to in my previous post, it's time for us as a community to fully embrace the place of automated testing in our workflows. Over the past year we've learned how to write testcases and to then apply that knowledge towards writing autopilot tests. At the same time ubuntu has been building testing infrastructure and launching a CI team to help run and maintain it.
With these changes and our acquired knowledge we can construct a sustainable vision for testing. So just what is that vision?
Let's make manual testing fun again. As we've ramped up our workloads, we find ourselves repetitively testing the same packages again and again (and running the same tests!). We'd call this regression testing and it's a perfect candidate for automated testing. Let's automate these tests and keep our focus on where we as humans excel.
In addition, we as a community participate in "calls for testing". These calls have become more and more exploratory in nature. In general, we have moved further away from a defined testcase ("perform steps 1, 2, and 3"). Instead, we encourage testers to adopt a try and break it attitude. This is where we as humans can excel, and you know what, have some fun at trying! Remember QA is the only place we encourage you to break things!
So let's get manual testing back to the exploratory testing we love.
|Thank you little testing robot!|
|All test and no play makes for a sad image|
So with all the automated testing buzz occurring in the quality world of ubuntu this cycle, I wanted to speak a little about why we're doing the work, and what the output of the work looks like.
So, why? Why go through the trouble of writing tests for the code you write? I'll save everyone a novel and avoid re-hashing the debate about whether testing is a proper use of development time or not. Simply put, for developers, it will prevent bugs from reaching your codebase, alleviate support and maintenance burdens, and will give you more confidence and feedback during releases. If you want your application to have that extra polish and completeness, testing needs to be a part. For users, the positives are similar and simple. Using well tested software prevents regressions and critical bugs from affecting you. Every bug found during testing is a bug you don't have to deal with as an end user. Coincidentally, if you like finding bugs in software, we'd love to have you on the team :-)
In general, three technologies are being used for automated testing.
Autopilot, Autopkg, and QML tests. Click to learn more about writing tests for each respectively.
|Any color is good, as long as it's green|
|Sorry robot, all our tests are belong to us|
This month has been all about dogfooding, and part of that has been the drive to automate the testing of the functionality found in the core applications. Nothing will replace true user testing, but we can certainly do a lot to help prevent bugs and regressions. So, during my "test all the things" series we went through each of the core applications and highlighted areas for testing. So the question is, how are we doing as this month draws to a close?
Well, fortunately, I've setup this wiki page as a reflection of where we stand.
As you can see, we've come a long way on many of the applications, but I wanted to especially point out where we still need help.
Adrian has been working on Dropping Letters and could use some help in finishing the testcases. Leo has proposed a merge that brings the testsuite up to date with the new sdk, so it's ready for you to add tests!
RSS Reader, aka shorts, just went through some large UI changes and is now ready for us to add tests again. Carla has begun working on updating her old tests for adding and editing a feed, but there's more work to do!
Calendar is currently undergoing some drastic changes, and it's testcases will need to be re-written once that's complete. Hang tight if your keen to help in this area!
Music has come a long way in a couple weeks and we have tests to prove it! Thanks to Daniel and Victor, we can test play, pause and library load. But there are still more tests needed. John is also volunteering on this application to help get the tests in shape. Thanks guys!
The following applications are very close to completion:
Nekhelesh has hacked together most of the tests, but I know he won't complain if you help him finish. Review the list and help bring it over the line!
As of this morning, the final calculator testcase has been written, testing tear-off. Pending a bugfix in the sdk, the testcase will get merged. Fingers crossed. Thanks Riccardo!
Congratulations to the following teams and folks who helped us reach 100% status on the following core apps:
terminal, weather, sudoku, stock ticker, file manager
Remember though, as we move forward it's important to keep the tests aligned with new features :-)
Still reading this? What are you waiting for? Jump in and help! Happy Automated Testing!
A little over a month ago, I posted about creating an autopilot emulator for
ubuntu sdk applications. Thanks to some hard work by Leo Arias and the ubuntu sdk team, I'm happy to announce the little side project from my +junk branch is all grown up and even more ready for your consumption. Leo has made the emulator a proper part of the sdk, complete with tests for all of it's functions! You can now easily install it and incorporate it into your project. Here's how to snag your copy.
1) Add the ubuntu sdk team ppa if you haven't already
sudo add-apt-repository ppa:ubuntu-sdk-team/ppa
2) Install the ubuntu-ui-toolkit-autopilot package
sudo apt-get update && sudo apt-get install ubuntu-ui-toolkit-autopilot
This will install the emulator as a python module, ready for you to import. If you want to checkout what the module can do, have a look at the documentation.
Incorporating the module might seem a little tricky, so Leo has also put together an example of one of the ubuntu touch core apps using the new module. Check out it. Here's a branch showing off the work done for ubuntu-filemanager-app. And here's one for dropping-letters.
Please do check out the module and incorporate it into your ubuntu sdk project. Feedback is encouraged, and bug reports too! Please file any issues you find against the ubuntu-ui-toolkit project. Happy Automated Testing!
The music app. A vital and necessary piece of my running setup. I've run in silence before, and while it's nice, sometimes you need some jams to keep you motivated on longer runs.
Recently I've been running in silence but not by choice, definitely needing a way to play music via my phone. Thanks to the hack day for music last week and the ongoing work on the music app developers that's all changing. Today the app is feature complete enough to begin work on some autopilot tests. It can for instance, play music now!
© 2010 Canonical Ltd. Ubuntu and Canonical are registered trademarks of Canonical Ltd.