I blogged a couple of weeks ago about the addition of the key performance indicators to the Ubuntu QA Dashboard. Since that post the QA team has been hard at work. We have added a bootspeed KPI to the dashboard, giving you a quick look at today vs yesterday. Another cool feature that has been added to the dashboard is the addition of bug information. Previously the dashboard just provided a link to the bug in Launchpad. The dashboard now fetches the bug data from Launchpad and displays it for you when the mouse hovers over the link. No more having to click through to see what the bug is that is causing issues!
The other big things that we have added goes back to one of the most basic types of testing that we do, the smoke test. We have added two new big things to smoke testing in the past few weeks. The first is that you are now able to see, from the dashboard, which tests pass and which fail. The new test case results page shows you quite a number of things about the testing that was done. You get the basics that you see on the other smoke testing pages (total tests, pass/fail/error count, pass rate, as well as image and machine information) but you now see a list of test cases and their return codes. As you see on the results page I linked to, there were 19 test cases that ran plus four setup type commands. The setup commands are shown since it is possible for them to error. Clicking on the individual command types will give you more details about the specific test to include the test suite and the command that was run. This is all valuable data in determining the quality of Ubuntu each day and easily pinpointing any problems that there are.
The final thing that has changed with smoke testing isn’t so much a change in the dashboard but an addition of a type of test. The QA team is now running daily Ubuntu Touch testing with autopilot. Adding the autopilot testing provides us with a new and much better grasp of the daily quality of Ubuntu. Previously, we relied on manual testing for much of the functional testing of Ubuntu to determine if things where the way that they were supposed to be. While the manual testing is still an important part of the overall indication of quality, the addition of the autopilot tests provides us with the ability to test many more things at a much higher frequency than relying completely on manual testing. The autopilot testing results show up in the smoke testing results each day after they run.Read more