It's almost hard to believe, but the new cycle is starting to ramp up. In just over one week's time, we'll be putting out an alpha 1 iso for quantal! If you will remember last cycle, I began an adopt an iso campaign to help insure precise got the iso testing coverage it needed to be an awesome release. This cycle, that campaign will continue with an open call for folks to adopt an iso and help test it all cycle long. Instead of managing and updating all of our excellent testers myself via email, I am asking instead for you to subscribe to the iso your interested in adopting and helping to make sure it's in a ready state for each milestone release. Subscribing to the iso will alert you via email when there is a new build for you ready to test, so you don't need to watch the page or await an email from me. If you miss the email from me, you may of course contact me for personal interactions at anytime you wish :-)

Interested? Awesome! This wiki page should detail everything you need to get started. Specifically, you should ensure you subscribe to the testcases for the iso you wish. For example, if I am interested in Ubuntu Desktop i386, I would head over to this page. See that button at the bottom called subscribe? Hit it and you should be subscribed to new builds for that iso. Please note the subscription feature is a work in progress, and there is not (yet!) a management page for subscriptions. Additionally, there is no visual indication on the page that your subscription is active (contributions welcome, contact me if you know drupal and wish to help!). Please read the wiki page on ISO Testing for more information on confirming your subscription in the interim period.

Ok, great, so now your subscribed. There's just one piece left in making sure things go well for your iso. This cycle we are trying something new to help make the alpha, beta, final release process smoother for all of our iso's (and you!). We would like to have our adopters run the daily iso before we spin the first candidate for release. What this means for you is that the week before each milestone release date, go ahead and try testing out the daily version of the iso. Think of it as warm-up for the big day. After all, you don't want your iso to be the one causing the re-spin do you?

This schedule will come in handy. It shows the timeline for how and when we'll be undertaking this testing. I know it looks big and scary, but focus on just the first column called 'Community Testing'. See the week of May 31st with the 'Q-D' listed by it in the 'Community Testing' column? If you glance at the 'Legend' at the top you will notice that 'Q' stands for quantal, and 'D' stands for daily. You will also notice that the Alpha 1 release for quantal is schedule one week after. So our goal is to spend this week getting our iso's in shape for the first spins for Alpha 1. The good news is time spent now is respins saved later. Happier isos make happier users!

Before I close I do want to remind everyone that each cycle is a marathon. We spoke at UDS about burnout, and that is something to keep in mind. Pace yourself and share the work. We'll have a wonderful cycle together. As always, contact me if you need any help. Your response was overwhelming for ubuntu precise, let's keep going strong for quantal. Thanks for helping make ubuntu better for everyone!

Read more

I originally posted this wonderful wall of text on the ubuntu-qa mailing list. If you want to get invovled in QA on ubuntu this cycle, you should subscribe to that list. Additionally, sign-up for the ubuntu testing team. Monitoring this blog and @UbuntuTesting will also keep you informed.


ISOTesting
My goal is to help ensure things are smooth before milestones, and
before isotesting events. Before we spin an iso, we want to feel good
about what's going on that iso. And we as a community can help make that
happen. Overall, I want each individual to have a lighter workload than
last cycle, despite having a similar amount of overall work we need to
achieve. To do this I'd like to help enable more people to be testing,
and to expand the 'adopt an iso' program so that folks can focus on
testing things they like and are able to test without becoming
overwhelmed or burnt out. Additionally, respins will be a continuous
focus and communication of what has changed and what needs tested will
be a priority. As a community we want to avoid doing re-work/extra work
and dedicate ourselves to performing quality testing, not merely having
a large quantity of testing.

Application Testing
Last cycle we utilized checkbox to deliver manual application tests.
During UDS, we spoke of expanding the isotracker to do our testcase
management, and thus consolidate our application testing by using the
same tool used for the isotracker to create an application tracker. This
work is on-going, but should be finished at some point during the cycle
so we can adopt it and use it. In the interim period will be continue
utilizing checkbox or doing manual testing via blogs or mailing lists, etc.

SRU Verification
SRU verification is currently a manual process with a high learning
curve and little visibility for many people. During the cycle, we hope
to help change that but also utilizing a new tracker to do SRU testing.
This testing will involve running the stable version (currently precise)
of ubuntu, but testing fixes to individual packages. This makes it a
good fit for those who aren't living on the bleeding edge but wish to
help. When this process is ironed out (sometime during the cycle) I will
contact everyone again with information on howto get involved.

General Testing (eg, Day-to-Day running of the development version)
Some good feedback was given on how to help make this better. There are
a few things we would like to do to help improve this process. First,
day to day changes should be able to be followed easier with some
proposed changes to update-manager to better display changelogs for
updated packages. I'll be detailing some information about how
'whoopsie' works and what it means to you. In addition, keeping the
development release stable at all times will continue to be a priority
for the development teams.

Calls for testing (specific feature or new features of critical package
or focused testing on a specific package)
Last cycle this typically involved me posting and laying out a basic
testplan on my blog with instructions on how to help test. This cycle,
again we hope to consolidate this onto a tracker where the tests and
results can be recorded. I will still be utilizing my blog, the
@ubuntutesting twitter account, this mailing list and our IRC meeting to
publicize events like this for people to get involved and contribute.
It's always fun to see new features before they come to everyone else,
and the feedback loop with the developers was welcome on both sides.

QATracker Development
With these changes to the qatracker, there is room for some folks who
know python and django to get involved and help improve the qatracker
codebase to make testing and reporting easier  Contact me, or simply
have a look at the code on launchpad and start hacking.
lp:~ubuntu-qa-website-devel/ubuntu-qa-website/drupal7-rewrite
lp:~ubuntu-qa-website-devel/ubuntu-qa-website/drupal7-rewrite-testcase-management
lp:~ubuntu-qa-website-devel/ubuntu-qa-website/python-qatracker
lp:~ubuntu-qa-website-devel/ubuntu-qa-website/python-qatracker-testcase-management

Hardware Database
The idea for having a hardware database for testing is not a new one,
but work has begun anew. This is work that will go beyond this cycle,
but ideas are being explored at using ubuntu friendly and other tools to
make this a reality.

Testcases
As a testcase management system will soon be in place (hurray!), we'll
be migrating all of the testcases over to this system. That means will
have much better visibility and ease of maintenance for all of our
testcases. Cleanup and expansion of the number of testcases is
definitely a goal for the cycle, and expect to hear more about getting
involved in this area.

Whew, that's a wall of text, but I hope it helps outline what the plans
are for the cycle. Feedback appreciated and encouraged  Happy Testing!

Read more

As I wrote two weeks ago, I consider the QA related changes in Ubuntu 12.04 a great success. But while we will continue and even extend our efforts there, this is not where the ball stops: it’s great to have the feedback cycle between “break it” and “notice the bug” reduced from potentially a few months to one day in many cases, but wouldn’t it be cool to reduce that to a few minutes, and also put the machinery right at where stuff really happens — into the upstream trunks? If for every commit to PyGObject, GTK, NetworkManager, udisks, D-BUS, telepathy, gvfs, etc. we’d immediately build and test all reverse dependencies and the committer would be told about regressions?

I have had the desire to work on automated tests in Linux Plumbing and GNOME for quite a while now. Also, after 8 years of doing distribution work of packaging and processes (tech lead, release engineering/management, stable release updates, etc.) I wanted to shift my focus towards technology development. So I’ve been looking for a new role for some time now.

It seems that time is finally there: At the recent UDS, Mark announced that we will extend our QA efforts to upstream. I am very happy that in two weeks I can now move into a role to make this happen: Developing technology to make testing easier, work with our key upstreams to set up test suites and reporting, and I also can do some general development in areas that are near and dear to my heart, like udev/systemd, udisks, pygobject, etc. This work will be following the upstream conventions for infrastructure and development policies. In particular, it is not bound by Canonical copyright license agreements.

I have a bunch of random ideas what to work on, such as:

• Making it possible/easier to write tests with fake hardware. E. g. in the upower integration tests that I wrote a while ago there is some code to create a fake sysfs tree which should really go into udev itself, available from C and introspection and be greatly extended. Also, it’s currently not possible to simulate a uevent that way, that’s something I’d like to add. Right now you can only set up /sys, start your daemon, and check the state after the coldplugging phase.
• Interview some GNOME developers what kind of bugs/regressions/code they have most trouble with and what/how they would like to test. Then write a test suite with a few working and one non-working case (bugzilla should help with finding these), discuss the structure with the maintainer again, find some ways to make the tests radically simpler by adding/enhancing the API available from gudev/glib/gtk, etc. E. g. in the tests for apport-gtk I noticed that while it’s possible to do automatic testing of GUI applications it is still way harder than it should and needs to be. I guess that’s the main reason why there are hardly any GUI tests in GNOME?
• I’ve heard from several people that it would be nice to be able to generate some mock wifi/ethernet/modem adapters to be able to automatically test NetworkManager and the like. As network devices are handled specially in Linux, not in the usual /dev and sysfs manner, they are not easy to fake. It probably needs a kernel module similar to scsi_debug, which fakes enough of the properties and behaviour of particular nmetwork card devices to be useful for testing. One could certainly provide a pipe or a regular bridge device at the other end to actually talk to the application through the fake device. (NB this is just an idea, I haven’t looked into details at all yet).
• For some GUI tests it would be much easier if there was a very simple way of providing mocks/stubs for D-BUS services like udisks or NetworkManager than having to set up the actual daemons, coerce them into some corner-case behaviour, and needing root privileges for the test due to that. There seems to be some prior art in Ruby, but I’d really like to see this in D-BUS itself (perhaps a new library there?), and/or having this in GDBus where it would even be useful for Python or JavaScript tests through gobject-introspection.
• There are nice tools like the Clang static code analyzer out there. I’d like to play with those and see how we can use it without generating a lot of false positives.
• Robustify jhbuild to be able to keep up with building everything largely unattended. Right now you need to blow away and rebuild your tree way too often, due to brittle autotools or undeclared dependencies, and if we want to run this automatically in Jenkins it needs to be able to recover by itself. It should be able to keep up with the daily development, automatically starting build/test jobs for all reverse dependencies for a module that just has changed (and for basic libraries like GLib or GTK that’s going to be a lot), and perhaps send out mail notifications when a commit breaks something else. This also needs some discussion first, about how/where to do the notifications, etc.

Other ideas will emerge, and I hope lots of you have their own ideas what we can do. So please talk to me! We’ll also look for a second person to work in that area, so that we have some capacity and also the possibility to bounce ideas and code reviews between each other.

Read more

The first Beta of the upcoming PostgreSQL 9.2 was released yesterday (see announcement). Your humble maintainer has now created packages for you to test. Please give them a whirl, and report any problems/regressions that you may see to the PostgreSQL developers, so that we can have a rock solid 9.2 release.

Remember, with the postgresql-common infrastructure you can use pg_upgradecluster to create a 9.2 cluster from your existing 8.4/9.1 cluster and run them both in parallel without endangering your data.

For Debian the package is currently waiting in the NEW queue, I expect them to go into experimental in a day or two. For Ubuntu 12.04 LTS you can get packages from my usual PostgreSQL backports PPA. Note that you need at least postgresql-common version 0.130, which is available in Debian unstable and the PPA now.

I (or rather, the postgresql-common test suite) found one regression: Upgrades do not keep the current value of sequences, but reset them to their default value. I reported this upstream and will provide updated packages as soon as this is fixed.

Read more

Half a year ago I blogged about the changed expectancies and processes to improve quality of the development release which we discussed at the UDS in Orlando: A promise that we don’t break the development version, regressions are not to be tolerated, acceptance criteria for Canonical upstreams. For that we introduced the Stable+1 team, actually did some reversions of broken packages, our QA team set up rigorous daily installation image and upgrade tests, and the code development process for Unity and related project was changed to enforce buildability and passing automatic tests with each and every change to trunk.

To be honest I was still a tad sceptic back then when this was planned. These were a lot of changes for one cycle, the stable+1 team was a considerable resource investment (starting with three people fulltime in the first few months), and not to the least our friends in the DX team felt thwarted because they had to sit down for a long time developing tests, and then changing their habits and practices for development.

So was all that effort worth it?

One word: OMGCRYOUTLOUDYES!!!!

Just a random sample of goodness that this brought:

• It was nice to not have to sit down for an hour every cople of days to figure out how to get back my desktop after the daily dist-upgrade bricked it.
• Unity, compiz, and friends were remarkably stable. I still remember the previous cycles where every new version got differently crashy, broke virtual workspaces, and what not. The worst thing that happened this cycle is eternally breaking keybindings (or changing them around), but at least those usually had obvious workarounds.
• As a result of those, I think we had at least one, maybe two magnitudes more testers of the daily development release than in previous cycles. So we got a lot of good bug reports and also patch contributions for smaller issues in Precise which we otherwise would not have discovered.
• The daily dist-upgrade tests tremendously helped to uncover packaging problems which would break real-world upgrades out there by the dozens. It took months to fix the hardest one: upgrading 10.04 LTS to 12.04 LTS with all universe packages offered in software-center. This beast takes 13 hours to run, so nobody really did manual tests like that in the past cycles.
• Due to the daily automatic CD image builds we dramatically reduced both the cost of fixing regressions as well as the emergency hackathons during milestone preparations. It is a lot easier to unbreak e. g. LVM setup or OEM install modes on our images when the regression happened just a day before than discovering it two days before a milestone is due, as again nobody tests these less common modes very often.

So as a result, I really think the investments into QA and the stable+1 teams already paid off twofold by giving us more time to work on the less critical fixes, avoiding lots of user frustration about broken upgrades, and generally making the daily development a lot more enjoyable. Or, as Rick Spencer puts it: Velocity, velocity, velocity!

Despite these improvements, there are still some improvements I’m looking forward to in the next cycles: Thanks to Colin Watson we can now use -proposed as a proper staging area, and used this feature rather extensively in the past month. From my point of view, 90% of the remaining daily dist-upgrade failures were due to packages building on different architectures at vastly different times, or failing on some, but not all architectures (“arch skew”). This is something you cannot really predict or guard against as a developer when you upload large and potentially harmful packages directly to the development release, so uploading them to the staging area and letting everything build there will reduce the breakage to zero. This was successfully demonstrated with Unity, GTK, and other packages where arch skew pretty much always causes people to hose their desktop, as well as daily CD images not working.

I’m also looking forward to combining the staging area with lots of automatic tests against reverse dependencies (e. g. testing the installer against a new GTK or pygobject before it lands), something we just barely tipped our toes in.

I can’t imagine how we were ever able to develop our new releases the old way.

Precise Pangolin^W^WUbuntu 12.04, I’m proud of you! Go out and amaze people!

Read more

Your commitment to quality and excellence has shown itself in this release. People love numbers, so let me spill some!

We had 13 calls for testing this cycle, with 6 iso testing milestones, and lots of bug reports, support and users using the development version of precise. Additionally, we set a new record for iso testing the precise final isos! Have a look yourself!

https://wiki.ubuntu.com/QATeam/ReleaseReports/PreciseFinalTestReport

That's 114 people who helped make sure your ubuntu precise experience was going to be a good one out of the box. Thank you to all of our testers this cycle!

https://wiki.ubuntu.com/PrecisePangolin/ReleaseNotes/Credits/Testers

Read more

Ubuntu Desktop i386

Yesterday, ~70 people (and counting! you guys are rockstars, thank you so much!) answered the call to adopt, and are now sponsoring over 2 dozen isos for ubuntu, and it's flavors like xubuntu, kubuntu, edubuntu, lubuntu and ubuntu studio. That is wonderful news.


However, there are a couple sad iso's still out there who are wanting to be adopted. They are the more troublesome ones to adopt if you will. Everyone loves and wants to help Ubuntu Desktop iso, but little pay attention to our mac and wubi specific testing. If you have access         to a windows machine and can help test wubi, please let me know! If you have access to a macbook or other mac hardware please also let me know! I can help you adopt those troublesome iso's and make ubuntu precise a better experience for those with similar hardware.

Ubuntu Desktop amd64+mac

Look at his face and then hit your compose button to email me. You'll be glad you did. Contact me at nicholas.skaggs at the canonical.com domain and I will make sure to get you connected to one of these little guys!

Read more

Would you adopt an ISO? A poor ISO, who cannot otherwise fend for itself in the cruel world of digital information. He's here one day and gone the next. Replaced by newer and better. Would you be willing to love, test, and use him while he is in existence? May his death not be in vain, but rather for the greater good of advancing us closer to that golden iso release to which we will all turn. It's powers we hope will transform our computers into the goodness that is ubuntu precise!

I trust my story has stirred your heart to action! In all seriousness, the ubuntu community is looking for a few brave volunteers to help shepherd our iso's thru the remaining week before release. If you volunteer, we are asking you to run through all of the testcases each day until next week when we release, for a particular iso and report your results to the isotracker. You can see what's required by having a look at this page. That lists the builds for the isos that were created today, each having some tests that help ensure the isos don't contain bugs. It's worth budgeting an hour or two to complete these tests, not including the time you need to download the iso. On Thursday of this week (April 19th 2012), we are scheduled to start the RC image testing which will be a dedicated milestone iso testing, utilizing the same tests.

As an added bonus to sponsoring, I am committing to personally emailing you and following up with help, tips and status updates on iso testing as we go throughout the week. Hurry, this offer won't last forever! To get started, email me personally with your the following info:

• Your testing hardware (real or virtual machine; amd64 or i386)
• Your name and email address
• Optionally, a specific iso(s) you wish to sponsor

I can be reached at nicholas.skaggs at the canonical.com domain. You may also leave a comment on this post, or send a message to the ubuntu-qa mailing list and I will followup with you. After you contact me, I will help you through the adoption process and get you and your new iso off to a wonderful start. Thanks so much for considering making a commitment to ubuntu!

 (If this reads like a "Help save X, donate now!" campaign, then you've read it correctly :-) )

Read more

As of last Friday, the precise beta2 release went to final and was published. As with beta1, a ppa containing manual test cases via checkbox has been made available. For those who helped test or contribute, perhaps you saw your name in lights? If not go check out those release notes! We appreciate your good work; kudos to our testing community. (BTW, if we missed you, we apologize! Shoot me an email and I will get you added to the list!)

Thanks for everyone who contributed testcases for this testing cycle. We got more comprehensive coverage than beta1. We're always happy to get more tests :-) Take a look at the wiki page for contributing tests. If you are unable to use launchpad for whatever reason, feel free to send a message to the ubuntu-qa mailing list. We can help you get your test cases added to checkbox even if your not a developer.

Now onto the testing! First you need to download the precise beta2 iso. You will find the iso's available here. Pick one that will work on your hardware.

Next, follow this wiki page to get checkbox and the application tests installed, run through the test cases and report your results. Thanks so much for helping test! If you find a bug, Jono has a great tutorial on how to file a bug. Make sure you mention your bug report in your comment if a test fails!

Thanks for testing and helping make ubuntu rock!

Read more

With the renewed focus on quality this cycle, things might start to look a bit confusing for everyone as to who is doing what in regards to quality in ubuntu. If your interested in helping out, it's certainly easy to get lost. Let me do my best to clear the waters and shed some light on what teams exist in the quality realm and what they are doing. First, a list of teams:

• Ubuntu QA Team
• Canonical Platform QA Team
• Ubuntu Testing Team
• Ubuntu Laptop Testing Team
• SRU verification Team
• Product Strategy Quality Team
• Ubuntu Bug Squad
• Ubuntu Friendly Squad
• Ubuntu+1 Testing Team(s)
• Ubuntu+1 Maintenance Team
• Ubuntu Localized Image Testing Teams
• Ubuntu Flavors QA Teams
• Edubuntu
• Lubuntu
• Xubuntu
• Kubuntu
• Ubuntu Studio
• Mythbuntu

Let's march down the list one at a time and discuss what each team does. If any of the descriptions sound interesting, be sure and follow the links to find out more information and/or to join the team.

Ubuntu QA Team
Mailing List: https://lists.ubuntu.com/mailman/listinfo/ubuntu-qa
IRC Channel: #ubuntu-testing
Launchpad page: https://launchpad.net/~ubuntu-qa
Wiki: https://wiki.ubuntu.com/QATeam
Blog: http://qa.ubuntu.com
This team’s mission is to help test ubuntu. The testing takes many forms and the team helps maintain a set of manual test cases usable for many different types of testing.

Canonical Platform QA Team
Launchpad page: https://launchpad.net/~canonical-platform-qa
This team is made up of Canonical employees who are performing qa for ubuntu. They are responsible for helping keep the testing infrastructure going, as well as coordinating and performing daily smoke testing, SRU's, and iso testing.

Ubuntu Testing Team
Launchpad page: https://launchpad.net/~ubuntu-testing
This team is a focus group of the ubuntu qa team specializing in performing iso testing, SRU testing, as well as manual application testing.

Ubuntu Laptop Testing Team
Launchpad page: https://launchpad.net/~ubuntu-laptop-testing
Wiki page: https://wiki.ubuntu.com/Testing/Laptop
This team is a focus group of the ubuntu qa team specializing in ensuring laptops work properly with each ubuntu release by testing isos for basic functionality across a wide range of laptops.

SRU Verification Team
Launchpad page: https://launchpad.net/~sru-verification
Wiki page: https://wiki.ubuntu.com/StableReleaseUpdates
People in this team perform the verification of packages that are candidates for a Stable Release Update. The candidate packages exist in the -proposed repository and need testing before they are moved to -updates.

Product Strategy Quality Team
Blog: http://qualityhour.wordpress.com/
IRC channel: #ubuntu-testing
This team is made up of Canonical employees from the product strategy team. They are responsible for things like unity and enhancing the end user experience inside ubuntu. The quality team is specifically focused on making sure all of those ideas and features are put to the test before being deployed to the greater community of end users. I am personally excited to see this team focus on quality and solicit feedback and help from the community. If your interested in helping improve the ubuntu user experience, this team's mailing lists and IRC meetings are a great place to start.

Ubuntu Bug Squad
Launchpad page: https://launchpad.net/~bugsquad
Wiki: https://wiki.ubuntu.com/BugSquad
Mailing List: https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugsquad
The bug squad! These wonderful folks pore over, triage, and assign bug reports. They act as the critical piece of communication and help between developers and users.

Ubuntu Friendly Squad
Launchpad page: https://launchpad.net/~ubuntu-friendly-squad
Website: https://friendly.ubuntu.com/
Wiki: https://wiki.ubuntu.com/UbuntuFriendly
This team is focused on making it easier to find and use laptops and computers that work "out of the box" with ubuntu. If you have ever had the experience of having ubuntu not boot on that old laptop, or boot to a black screen even though your desktop pc works marvelously, this is the site for you. I recently made use of the site to find out just how well supported the laptop I wanted to purchase was, in addition to which components I should look for (and avoid!) in order to have ubuntu work with my hardware.

Ubuntu+1 Testing Team(s)
IRC Channel: #ubuntu+1
Forum: http://ubuntuforums.org/forumdisplay.php?f=412
These team(s) are composed of people willing to be on the front lines and always running the next version of ubuntu. They provide help and support amongst each other, and valuable bug reports and first responder feedback to the development teams. The structure is rather informal, and the requirements are low -- you simply need to commit to running the development release and learn how to submit good bug reports.

Ubuntu+1 Maintenance Team
IRC Channel: #ubuntu+1-maint
Wiki: https://wiki.ubuntu.com/PlusOneMaintenanceTeam
This team is brand new for this cycle, and they have a very specific goal that is key to delivering quality in precise. They're goal is to have the archive be ALWAYS installable and usable. For anyone who has ever run the development version of ubuntu in the past, you know how difficult a task this is! Thus far the team has done an excellent job, and it's been noticeably easy to run precise throughout the cycle.

Ubuntu Localized Image Testing Teams
Wiki: https://wiki.ubuntu.com/PrecisePangolin/LocalizedImageContacts
Ubuntu's wonderful LoCo teams showoff the true meaning of ubuntu and the spirit of FOSS. In addition to the many other activities LoCo's do, some are providing localized iso's for there users. If your a non-English speaker, you understand what a blessing this can be when your having to install ubuntu. Since these images contain additions to the standard iso image, they must also be tested for bugs.

Ubuntu Italian Testing Team
Launchpad page: https://launchpad.net/~ubuntu-it-testing
This team is a wonderful example of a loco team doing testing for ubuntu. They organize goals each cycle typically surrounding creating a localized Italian iso, as well as doing laptop testing to ensure ubuntu works on common laptop configurations.



Ubuntu Flavors QA Teams
Do you like using ubuntu, but find yourself using one of the official flavors of ubuntu instead? These teams produce there own isos and packages, in addition to supporting different subsets of software. Each flavor has there own method of doing QA, but the goal is the same -- to deliver quality releases for there users.

Edubuntu
IRC Channel: #edubuntu
Webpage: http://www.edubuntu.org/

Lubuntu
IRC Channel: #lubuntu-qa
Launchpad Page: https://launchpad.net/~lubuntu-qa
Wiki: https://wiki.ubuntu.com/Lubuntu/Testing
Mailing List: https://lists.launchpad.net/lubuntu-qa/

Xubuntu
IRC Channel: #xubuntu-devel
Webpage: http://www.xubuntu.org/contribute/qa_bugs_testing
Mailing List: https://lists.ubuntu.com/mailman/listinfo/xubuntu-devel

Kubuntu
IRC Channel: #kubuntu-devel
Webpage: https://wiki.ubuntu.com/Kubuntu/QA
Mailing List: https://lists.ubuntu.com/mailman/listinfo/kubuntu-devel

Ubuntu Studio
IRC Channel: #ubuntustudio
Webpage: http://ubuntustudio.org/

Mythbuntu
IRC Channel: #ubuntu-mythtv
Webpage: http://www.mythbuntu.org/testingandreporting

Whew, all done :-) It is worth noting that although I attempted to be comprehensive with this list, it is not exhaustive. Are you doing QA work in ubuntu that I didn't mention? Let me know about it! Or, perhaps you have an idea for some work that doesn't fit into one of the above. Let me know about your ideas as well! I have already found this community to be full of interesting people and ideas, and likely you will be able to find someone to help you move forward with your plan.

If any of this work interests you, please do contact the appropriate team via the links above or simply contact me directly and I can help get you connected. A big thank you to all of these people for their commitment to helping make ubuntu great!

Read more

The Ubuntu Kernel Team has uploaded a new kernel (3.2.0-17.27) which contains an additional fix to resolve the remaining issues seen with the RC6 power saving enabled. For users with Sandy Bridge based hardware we would appreciate them to run the tests described on https://wiki.ubuntu.com/Kernel/PowerManagementRC6 and add their results to that page.

Read more

The precise beta1 release just dropped! As a follow-up to my earlier post about participating in testing with this release, now is your opportunity to do some manual application testing for ubuntu. This is a perfect ubuntu global jam activity!

Thanks for everyone who contributed testcases for this testing cycle. If you missed getting your merge in on time, don't worry! Beta2 is coming up and I hope we can add even more tests. Take a look at the wiki page for contributing tests. If you are unable to use launchpad for whatever reason, feel free to send a message to the ubuntu-qa mailing list. We can help you get your test cases added to checkbox for beta2!

Now onto the testing! First you need to download the precise beta1 iso. You will find the iso's available here. Pick one that will work on your hardware.

Next, follow this wiki page to get checkbox and the application tests installed, run through the test cases and report your results. Thanks so much for helping test! If you find a bug, Jono has a great tutorial on how to file a bug. Make sure you mention your bug report in your comment if a test fails!

Go forth and test!

Read more

The precise beta1 release is happening this week (I know, I know beta already!). Despite my short tenure in the ubuntu QA community, we've already seen a huge increase in the amount of manual application testing. I thank everyone who helped test when I and others have put out calls for testing this cycle. As part of our desire to continually improve our processes, I'm am adopting the checkbox tool to help do manual testing for the beta1 and beta2 releases during the precise cycle.

For anyone who participated in the Unity testing, you will remember how much nicer it was to use checkbox to deliver the tests and be able to submit feedback directly from your desktop. Running these tests was a huge part getting Unity 5.2 and Unity 5.4 out in precise. Didrocks posted about the response and subsequent lessons learned. For beta1 and beta2, I'd like to move this method of testing onto the default desktop applications.

There are two opportunities here to get involved. First of all, if your interested in helping test our default desktop applications, you will be able to do so directly with checkbox in a similar vein as unity. I will post instructions on testing this week. It will require you to install from a ppa, but I hope to remove the requirement for beta2 :-)

Secondly, if your an application developer who wants to get some testing done you can contribute tests directly to my bazaar branch on launchpad. I've written up instructions for doing so on the wiki. Currently the following testcases are being targeted. If you responsible for any of these applications and haven't spoken to me, I'd love to talk to you about increasing test coverage!

firefox
rythmnbox
empathy
thunderbird
nautilus
libreoffice
software-center
system-settings
deja-dup
totem
evince
file-roller
gedit
eog
gwibber
seahorse
ubuntuone
update-manager
shotwell

If you are a passionate user of any of these applications (or any other applications), get in touch with me as well! You can help contribute tests even if your not capable of submitting a merge request via launchpad :-) EDIT: Since so many have asked, if you can't contribute by submitting a merge request, update the ubuntu testcase wiki and then send me an email.  You can find my address here on my launchpad profile. I will incorporate your tests into the tool. Read the guidelines on the format, and then add the testcase to the applications page. It's important to get the format right so doublecheck your work! That said, please try going through the normal route of writing a checkbox test if you can -- they are really easy to do!

In closing, I would like to thank everyone who helped rewrite testcases on the ubuntu testcase wiki. These testcases were used as a leg up on getting tests for these applications, and have us off to a good start on getting more testing coverage for ubuntu's default applications. I would also like to briefly mention the future of testing for next cycle. My goal is to gather feedback from doing manual testing this cycle with checkbox and use it to blueprint the tools and parameters for how we plan on testing during the Q cycle. I hope to have some sessions around testing during UDS, and look forward to hearing from you about your experiences!

Read more

The Ubuntu Kernel Team has released a call for testing for a set of RC6 power saving patches for Ubuntu 12.04 Precise Pangolin LTS. Quoting Leann Ogasawara's email to the ubuntu kernel team and ubuntu-devel mailing lists:

"Hi All,

RC6 is a technology which allows the GPU to go into a very low power consumption state when the GPU is idle (down to 0V). It results in considerable power savings when this stage is activated. When comparing under idle loads with machine state where RC6 is disabled, improved power usage of around 40-60% has been witnessed [1].

Up until recently, RC6 was disabled by default for Sandy Bridge systems due to reports of hangs and graphics corruption issues when RC6 was enabled. Intel has now asserted that RC6p (deep RC6) is responsible for the RC6 related issues on Sandy Bridge. As a result, a patch has recently been submitted upstream to disable RC6p for Sandy Bridge [2].

In an effort to provide more exposure and testing for this proposed patch, the Ubuntu Kernel Team has applied this patch to 3.2.0-17.26 and newer Ubuntu 12.04 Precise Pangolin kernels. We have additionally enabled plain RC6 by default on Sandy Bridge systems so that users can benefit from the improved power savings by default.

We have decided to post a widespread call for testing from Sandy Bridge owners running Ubuntu 12.04. We hope to capture data which supports the the claims of power saving improvements and therefore justify keeping these patches in the Ubuntu 12.04 kernel. We also want to ensure we do not trigger any issues due to plain RC6 being enabled by default for Sandy Bridge.

If you are running Ubuntu 12.04 (Precise Pangolin) and willing to test and provide feedback, please refer to our PowerManagementRC6 wiki for detailed instructions [3]. Additionally, instructions for reporting any issues with RC6 enabled are also noted on the wiki. We would really appreciate any testing and feedback users are able to provide.

Thanks in advance,
The Ubuntu Kernel Team"

So please contribute to this call for testing by visiting https://wiki.ubuntu.com/Kernel/PowerManagementRC6 and follow the instructions.  Thank you!

Read more

Testing is hugely important to us at Canonical. We all strive to have Ubuntu reliable, consistent and fast. But we’re human, and we make mistakes. Sometimes a bugfix will break something else, and for something as complex as a desktop shell, it’s easy to miss these breakages. While manual tests can help reduce these regressions, realistically we need an automated system to emulate the users inputs and verify our software works as it should – and scream bloody murder if it doesn’t!

In Unity 2D (my project!), we have just introduced an automated User Experience testing system, based on a test framework called Testability Driver (I’ll just call it ‘Testability’ from now on). First off, a clarification:

Testability is for Qt-based applications only!

A limitation: yes, but this requirement comes with a great reward: Testability allows inspection of the tree of QObjects in a Qt application while it is running!

It can read and write object properties, call methods and slots, verify signals are emitted, as well as fake mouse/keyboard/gesture inputs, grab visual outputs, and measure performance.

And best of all, Testability is open source and maintained by Nokia! That means everyone can run and contribute tests!

To show it off, here is a screengrab of the Testability Visualizer application which allows you to connect to a running application, dig into the QObject tree and investigate what’s going on (here, the Unity 2D Launcher – it will be good to zoom in):

On the left is an interactive snapshot of the UI, in the centre is the QObject tree which you can navigate, and on the right is the list of the selected QObject’s properties, methods and signals you can interact with.

On display in that screengrab are some of the attributes of the QDeclarativeImage object which draws the Terminal icon in the launcher (this is defined in QML!) – you can read off the source of the image, the x,y coordinates, width & height, and a whole lot more.

All these properties, methods, signals, etc, are scriptable via Ruby. Thus we can emulate every interaction that a user can make with the application, and determine the reaction matches our expectation.

And all this power comes with almost no changes* to the source code! (* = just need to add object names, see later)

This forms the foundation for the Unity 2D User Experience testing suite.

How Testability Works

First some definitions:
“target” = the machine with the software being tested
“host” = the machine running the test suite, and controlling the target

Testability works as follows

• any Qt application using Qt4.6+ can be tested by executing it with the “-testability” switch.
• a standalone server “qttasserver” runs in the background.
• With the -testability switch, the Qt library tries to load a “libtestability.so” plugin which establishes a connection between the application and qttasserver, giving qttasserver access to the root node of the QObject tree.
• qttasserver then climbs the QObject tree, reads all the info it can and converts it to an XML format. It also can receive commands and cause the application to react upon them (click here, type, etc..).
• A series of Ruby scripts connect to qttasserver, receive and parse this XML and allow us to script tests and interactions with the application.

Note that there is a clear divide between the machine of the target and of the host.

This means Testability is great for testing software on embedded devices too. (Indeed Meego has been using it for that exact task!). It also reduces the risk that packages used to run the test suite could interfere with the software being tested.

However there’s nothing stopping you having the target and host the same machine.

[The Unity 2D test framework also includes an extra helper library to control the X server, to control the mouse and keyboard, and to manipulate windows on the desktop. As far as X is concerned, a human is controlling it.]

With the ability to fake any form of user input, and directly read the output from the shell applications
themselves, we can test almost every behaviour of the desktop shell. And as a result, the quality of the user’s experience will only go up.

How to get Testability and start writing tests

Unfortunately Testability is not available in Ubuntu’s repositories just yet, but I have it packaged in a PPA. Installation takes a few steps, so I suggest that interested people consult this wiki page.

Tests are just Ruby scripts, so running tests is just a matter of running a Ruby script!

I think it easiest to show how to use Testability with an example:

# Example test for Unity 2D with Testability Driver.
#
# Note: this probably won't succeed on your installation. Only works
# on unity-2d trunk *right now*, something that will change soon.# This is only a taster!require 'tdriver'
include TDriverVerify

# Establish connection to 'qttasserver' on target
@sut = TDriver.connect_sut(:Id => 'sut_qt')

# Execute the application to test, supplying '-testability' flag
@app = @sut.run( :name => 'unity-2d-launcher',
                 :arguments => '-testability' )
# ------------- Start of Testing Code ----------------

# Check Launcher is 65 pixels wide
verify_true(1, 'Launcher is not 65 pixels wide') {
  @app.LauncherView()['width'] == '65'
}

# Check that Terminal tile is using correct icon source
verify_true(1, 'Terminal icon in Launcher is wrong') {
  @app.LauncherList( :name => 'main' ) \
      .QDeclarativeItem( :name => 'Terminal' ) \
      .QDeclarativeImage( :name => 'icon' )['source']     == 'image://icons/utilities-terminal'
}
# this is a bad test, as the name "Terminal" can be # localised, meaning a fail if you're using a non-English # installation. Choosing good objectNames is important!

# ------------------- Clean up -----------------------
# This closes (kills actually) the launcher when we're done.
@app.kill

There is some boiler-plate code there, but the bits I want to point out are:

@app.LauncherView()['width']

This causes Testability to search for an object of type “LauncherView()” in the QObject tree, and if found reads its “width” property and returns it. Then in Ruby, we can check this value matches what we expect (65).

The second test does a similar operation, but needs a little more help. As there are many tiles, we need to help Testability to find the exact tile we want. We do this by adding some object names (“main”, “Terminal”) to the C++/QML.

By consulting with the Visualizer image above, maybe you can see how Testability navigates the tree to find the icon source for the Terminal tile!

Once you can track down objects uniquely, you can then start interacting with them, send mouse clicks, set properties, call methods, etc. This power gives you great flexibility in testing your application!

Demo

As a demo, here is a video of a part of the Unity 2D test suite in action. On the right is Ubuntu Oneiric running inside VirtualBox, where the Launcher will be tested. On the left is my host machine terminal running the test suite.

Various hide/show tests are being performed, with windows in the way, mouse being controlled, keyboard shortcuts being pressed etc.

Our policy is that every new feature and bug fix in Unity 2D will now be tested like this. You can see our growing test suite here. This will ensure Unity 2D remains reliable, consistent and fast. Thanks to Testability!

Tests will improve your project’s quality too. Will you give it a try?

- Gerry Boland

Read more

It's been a few weeks since the last drop of unity, and now the unity team has readied the new version of unity 5.2. Let's walk through how to preview the new features unity 5.2 is bringing, and help test those features using checkbox! Checkbox allows you to get your feedback straight into the hands of the unity developers and report any problems your system may have with the new version of unity. First let's talk a little bit about what's new. Note that these features only exist for right now in Unity 3D.

• Multi-monitor support
• You will now see launchers on each of your monitor, and when you scroll across a monitor, you should feel some resistance in order to allow for you to use the launcher on that screen.
• New screen edge detection
• To invoke the launcher, you now need to push (or "scroll into") against the left of the screen, rather than hover for X seconds. No more hitting the back button in firefox and having the launcher pop up in your way!

Feedback is appreciated on these features especially. Utilize #ubuntu-unity on freenode and checkbox feedback form to let the developers know how they work for you.

Installing

Prerequisites: Make sure you are running the latest version of precise, and all your packages are up to date. Unfortunately this cannot be installed on oneiric or any previous ubuntu release. 

Also, unity 5.2 did not ship with "the HUD" sadly. So if you have been testing the HUD you will need to use ppa-purge to remove and downgrade your packages. See this post for information on using ppa-purge if you need help doing so.

1) Add the unity ppa (https://launchpad.net/~unity-team/+archive/ppa). You can do this by issuing the following command:
2) Update apt and run a dist upgrade -- this should prompt you to upgrade unity and some indicators as well as install checkbox-unity.


3) Restart your unity session by logging out and logging back in again.

Ok, hopefully the upgrade went smooth for you, but if not, head over to freenode #ubuntu-unity channel and let folks know what went wrong.

Testing
So, now that your up and running you can run the through the manual tests the unity team has prepared. Open the dash and type 'unity testing'. The Checkbox Unity Tests should launch. Checkbox will gather some information on your system and then ask you which tests you wish to run. Once complete you will see a link containing your system report and an option to publish it to launchpad. Use the text box below the link to enter your launchpad email address and then hit submit. This will ensure your results and feedback go to the unity developers.

Please ensure you have finished and submitted your testing results ASAP. The testing window will be closed this Thursday at 8am UTC, in order to give the unity developers time to finish fixing the bugs found. Then unity 5.2 will be pushed to precise and coding on Unity 5.4 will begin.

Filing Bugs
Please file bugs against unity package in launchpad (https://bugs.launchpad.net/unity/+filebug). When filing, please make sure to tag your bug '5.2-rc1' and mention your running Unity 5.2-rc1 in your description.

Final Thoughts
Don't hesitate to reach out to the unity team on IRC #ubuntu-unity on freenode at any time or to follow the latest in unity development. Thanks for helping test ubuntu and unity!

Read more

I hope everyone has seen the announcement about the upcoming Heads Up Display feature hopefully landing in 12.04. If not, go read about it here: http://www.markshuttleworth.com/archives/939 I'll wait.

Great, now in order for this feature to show up in precise it needs some more work and refinement. Our focus on quality continues and this feature is not excluded -- in order to ensure its of release quality for precise, we're asking the community to help test and evaluate the feature. Here's everything you need to know to get started:

Prerequisites: Make sure you are running the latest version of precise, and all your packages are up to date. Unfortunately this cannot be installed on oneiric or any previous ubuntu release.

1) Add the HUD ppa (https://launchpad.net/~unity-team/+archive/hud). You can do this by issuing the following command:

sudo add-apt-repository ppa:unity-team/hud

2) Update apt and run a dist upgrade -- this should prompt you to upgrade unity and some indicators

sudo apt-get update && sudo apt-get dist-upgrade

3) Restart your unity session by logging out and logging back in again.

Now you should be up and running. Invoke the HUD using the 'alt' key. Go and try out your favorite apps and see how things work. When you find a bug, at this point please do not use the ubuntu-bug command or apport -- these tools are not setup to handle working within a ppa. Instead file a bug using launchpad against one of the following projects depending on the nature of the bug:

For anything related to the user interface, ie directly unity related, file a bug against unity: http://bugs.launchpad.net/unityWhen filing, tagging the bug with 'HUD' would be helpful to streamline visibility to the unity developers.

For any issues with matching or other issues core to the tool itself, file them against the appmenu indicator. http://bugs.launchpad.net/indicator-appmenu

Still stuck or have more questions? Visit the wonderful folks on #ubuntu-unity on Freenode. And remember, HUD will also land with unity 5.2 coming soon!

Read more

Part of my focus this cycle is to see where we can make power saving improvements for Ubuntu Precise 12.04 LTS. There has been a lot of anecdotal evidence of specific machines or power saving features behaving poorly over the past few cycles.   So, armed with a 6.5 digit precision multimeter from Fluke I've been measuring the power consumption on various laptops in different test scenarios to try and answer some outstanding questions:

* Is it safe to enable Matthew Garrett's PCIe ASPM fix?
* Are the power savings suggested by PowerTop useful and can we reliably enabled any of these in pm-utils?
* How accurate are the ACPI battery readings to estimate power consumption?
* Do the existing pm-utils power.d scripts still make sense?
* Which is better for power saving: i386, i386-pae or amd64?
* How much power does the laptop backlight really use?
* Does halving the mouse input rate really save that much more power?
* Should we re-enable Aggressive Link Power Management (ALPM)?
* Are there any misbehaving applications that are consuming too much power?
* What are the root causes of HDD wake-ups
* Which applications and daemons are creating unnecessary wake events?
* How much does the MSR_IA32_ENERGY_PERF_BIAS save us?

..and many more besides!

From some of the analysis and "crowd sourcing" tests it is clear that the PCIe ASPM fix works well, so we've already incorporated that into Precise.

Aggressive Link Power Management (ALPM) is a mechanism where a SATA AHCI controller can put the SATA link that connects to the disk into a very low power mode during periods of zero I/O activity and into an active power state when work needs to be done. Tests show that this can save around 0.5-1.5 Watts of power on a typical system. However, it has been known in the past to not work on some devices, so I've put a call for testing of ALPM out to the community so we can get a better understanding of the power savings vs reliability.

Some of the PowerTop analysis has shown we can save another 1-2 Watts of power by putting USB and PCI controllers of devices like Webcams, SD card controllers, Wireless, Ethernet and Bluetooth  into a lower power state.  Again, we would like to understand the range of power savings across a large set of hardware and to see how reliable this is, so another crowd sourced call for testing has been also set up.

So, if you want to contribute to the testing, please visit the above links and spend just a few tens of minutes to see we can extend the battery life of your laptop or netbook.  And periodically visit https://wiki.ubuntu.com/Kernel/PowerManagement to see if there any new tests you can participate in.

[UPDATE]

I've written some brief notes on power saving tweaks and also some simple recommendations for application developers to follow too.

The thread continues here (part 2)

Read more

I’m the release engineer in charge for Precise Alpha 1 which is currently being prepared. I must say, this has been a real joy! The fruits of the new QA paradigm and strategy and the new Stable+1 maintenance team have already achieved remarkable results:

• The archive consistency reports like component-mismatches, uninstallability, etc. now appear about 20 minutes earlier than in oneiric.
• CD image builds can now happen 30 minutes earlier after the publisher start, and are much quicker now due to moving to newer machines. We can now build an i386 or amd64 CD image in 8 minutes! Currently they still need to wait for the slow powerpc buildd, but moving to a faster machine there is in progress. These improvements lead to much faster image rebuild turnarounds.
• Candidate CDs now get automatically posted to the new ISO tracker as soon as they appear.
• Whenever a new Ubuntu image is built (daily or candidate), they automatically get smoke-tested, so we know that the installer works under some standard scenarios and produces an install which actually boots.
• Due to the new discipline and the stable+1 team, we had working daily ISOs pretty much every day. In previous Alphas, the release engineer(s) pretty much had to work fulltime for a day or two to fix the worst uninstallability etc., all of this now went away.

All this meant that as a release engineer almost all of the hectic and rather dull work like watching for finished ISO builds and posting them or getting the archive into a releasable state completely went away. We only had to decide when it was a good time for building a set of candidate images, and trigger them, which is just copy&pasting some standard commands.

So I could fully concentrate on the interesting bits like actually investigating and debugging bug reports and regressions. As the Law of Conservation of Breakage dictates, taking away work from the button pushing side just caused the actual bugs to be much harder and earned us e. g. this little gem which took Jean-Baptiste, Andy, and me days to even reproduce properly, and will take much more to debug and fix.

In summary, I want to say a huge “Thank you!” to the Canonical QA team, in particular Jean-Baptiste Lallement for setting up the auto-testing and Jenkins integration, and the stable+1 team (Colin Watson, Mike Terry, and Mathieu Trudel-Lapierre in November) for keeping the archive in such excellent shape and improving our tools!

Read more

I arrived back home in Augsburg, from last week’s Ubuntu Developer Summit in Orlando, FL. As this is a quality/LTS cycle, we pretty much already knew in advance what to do (bug fixing, bug fixing, some boot speed, and did I mention bug fixing?), but still we had many highly interesting and exciting sessions this time, not so much about what we are going to do, but how we are going to build 12.04.

So far our common practice has been to toss everything new into the development release until Feature Freeze and then try and clean up most of the fallout. Me and many other developers have always cried for having more time for fixing long-standing bugs and not introducing breakage in the first place. It seems that now with 12.04, Ubuntu/Canonical are actually getting serious about it.

(Any resemblance to that postcard from the Kennedy Space Center which I went to last Sunday is of course absolutely unintended and purely coincidental ).

The mission statement is now to have working ISOs, stable ? development, and daily intra-development upgrades every day, quick and regular cleanup of uninstallable packages, component-mismatches, NBS etc., backed by a new “stable +1″ team backed by three people on a rotational shift.

QA team is now setting up daily automatic smoketesting of the installer and other packages which have tests. For the latter we’ll convert some packages to the DEP-8, the proposed format for running autopkgtest on (I’ll do udisks, postgresql-common, pygobject, apport, and jockey soon).

We’ll try do put uploads which might break something (like new libraries) to a staging area first, against which we can run test suites of reverse dependencies before it lands in the new release. As doing this on a large scale still requires infrastructure to be created, we’ll only exercise it for a few packages by uploading to precise-proposed first, but this has a high potential for extension.

We want to commit to fixing major breakage within 3 hours of development time, or otherwise revert the faulty package to the previous version (unless that aggravates problems, such as file conflicts).

Finally, for Canonical upstreams we are introducing “acceptance criteria”, which will hopefully significantly raise the quality and lower the regressions of each Unity etc. release.

So, the mission is clear. In practice we’ll probably have to make some real-life concessions, and Murphy’s law dictates that there still will be some breakage, but we can learn from that as we go.

Let’s build 12.04 LTS!

Read more