Right now as I type we have two jams going on! Last week Jono posted about enhancing the ubuntu.com/community page. If your a part of the community, join in raising the banner for your specific focus area. The fun is happening now on #ubuntu-docs. For the full details, see Jono's post. For us quality folks, the pad is here: http://pad.ubuntu.com/communitywebsite-contribute-quality. Feel free to type and edit away!

In addition, as Daniel Holbach mentioned, there is a hackathon for automated testing. Come hang out with us on #ubuntu-quality, learn, ask and write some tests. Again, the full details can be found on Daniel's post.

Come join us!

Read more

Software quality has received a lot of attention recently. There have been tons of books, blog posts, conferences and the like on improving quality. Tools and practices such as TDD, automatic builds, agile methods, pair programming and static code analysers are praised for improving code quality.

And, indeed, that is what they have done.

But one should never mix the tool with the person using it. All these wonderful tools are just that: tools. They are not the source of quality, only facilitators of it. The true essence of quality does not flow from them. It comes from somewhere else entirely. When distilled down to its core, there is only one source of true quality.

Caring.

The only way to get consistently high quality code is that the people who generate it care about it. This means that they have a personal interest in their code tree. They want it to succeed and flourish. In the best case they are even proud of it. This is the foundation all quality tools lie on.

If caring does not exist, even the best of tools can not help. This is due to the fact that human beings are very, very good at avoiding work they don’t want to do. As an example, let’s look at code review. A caring person will review code to the best of their abilities because he wants the end result be the best it can be. A non-caring one will shrug, think “yeah, sure, fine, whatever” and push the accept button, because it’s less work for him and he knows that his merge requests will go in easier if there is a general (though unspoken) consensus of doing things half-assed.

Unfortunately caring is not something you can buy, it is something you must birth. Free food and other services provided by companies such as Valve and Google can be seen as one way of achieving quality. If a company sincerely cares about its employees, they will in return care about the quality of their work.

All that said, here is my proposal for a coder’s mascot:

Read more

So last time we learned some basics for autopilot testcases. We're going to use the same code branch we pulled now to cover writing an actual testcase.

bzr branch lp:~nskaggs/+junk/autopilot-walkthrough

As a practical example, I'm going to convert our (rather simple and sparse) firefox manual testsuite into an automated test using autopilot. Here's a link to the testcase in question.

If you take a look at the included firefox/test_firefox.py file you should recognize it's basic layout. We have a setup step that launches firefox before each test, and then there are the 3 testcases corresponding to each of the manual tests. The file is commented, so please do have a look through it. We utilize everything we learned last time to emulate the keyboard and mouse to perform the steps mentioned in the manual testcases. Enough code reading for a moment, let's run this thing.

autopilot run firefox

Ok, so hopefully you had firefox launch and run through all the testcases -- and they all, fingers-crossed, passed. So, how did we do it? Let's step through the code and talk about some of the challenges faced in doing this conversion.

Since we want to test firefox in each testcase, our setUp method is simple. Launch firefox and set the focus to the application. Each testcase then starts with that assumption. Inside test_browse_planet_ubuntu we simply attempt to load a webpage. Our assertion for this is to check that the application title changes to "Planet Ubuntu" - - in other words that the page loaded. The other two testcases expand upon this idea by searching wikipedia and checking for search suggestions.

The test_search_wikipedia method uses the keyboard shortcut to open the searchbar, select wikipedia and then search for linux. Again, our only assertion for success here is that the page with a title of Linux and wikipedia loaded. We are unable to confirm for instance, that we properly selected wikipedia as the search engine (although the final assertion would likely fail if this was not the case).

Finally, the test_google_search_suggestions method is attempting to test that the "search suggestions" feature of firefox is performing properly. You'll notice that we are missing the assertion for checking for search suggestions while searching. With the knowledge we're gained up till now, we don't have a way of knowing if the list is generated or not. In actuality, this test cannot be completed as the primary assertion cannot be verified without some way of "seeing" what's happening on the screen.

In my next post, I'll talk about what we can do to overcome the limitations we faced in doing this conversion by using "introspection". In a nutshell by using introspection, autopilot will allow us to "see" what's happening on the screen by interacting with the applications data. It's a much more robust way of "seeing" what we see as a user, rather than reading individual screen pixels. With any luck, we'll be able to finish our conversion and look at accomplishing bigger tasks and tackling larger manual testsuites.

I trust you were able to follow along and run the final example. Until the next blog post, might I also recommend having a look through the documentation and try writing and converting some tests of your own -- or simply extend and play around with what you pulled from the example branch. Do let me know about your success or failure. Happy Testing!

Read more

If you caught the last post, you'll have some background on autopilot and what it can do. Start there if you haven't already read the post.

So, now that we've seen what autopilot can do, let's dig in to making this work for our testing efforts. A fair warning, there is some python code ahead, but I would encourage even the non-programmers among you to have a glance at what is below. It's not exotic programming (after all, I did it!). Before we start, let's make sure you have autopilot itself installed. Note, you'll need to get the version from this ppa in order for things to work properly:

sudo add-apt-repository ppa:autopilot/ppa
sudo apt-get update && sudo apt-get install python-autopilot

Ok, so first things first. Let's create a basic shell that we can use for any testcase that we want to write. To make things a bit easier, there's a lovely bazaar branch you can pull from that has everything you need to follow along.

bzr branch lp:~nskaggs/+junk/autopilot-walkthrough
cd autopilot-walkthrough

You'll find two folders. Let's start with the helloworld folder. We're going to verify autopilot can see the testcases, and then run and look at the 'helloworld' tests first. (Note, in order for autopilot to see the testcases, you need to be in the root directory, not inside the helloworld directory)

$ autopilot list helloworld
Loading tests from: /home/nskaggs/projects/

    helloworld.test_example.ExampleFunctions.test_keyboard
    helloworld.test_example.ExampleFunctions.test_mouse
    helloworld.test_hello.HelloWorld.test_type_hello_world

 3 total tests.

Go ahead and execute the first helloworld test.

autopilot run helloworld.test_hello.HelloWorld.test_type_hello_world
 
A gedit window will spawn, and type hello world to you ;-) Go ahead and close the window afterwards. So, let's take a look at this basic testcase and talk about how it works.

from autopilot.testcase import AutopilotTestCase

class HelloWorld(AutopilotTestCase):

    def setUp(self):
        super(HelloWorld, self).setUp()
        self.app = self.start_app("Text Editor")

    def test_type_hello_world(self):
        self.keyboard.type("Hello World")

If you've used other testing frameworks that follow in the line of xUnit, you will notice the similarities. We implement an AutopilotTestCase object (class HelloWorld(AutopilotTestCase)), and define a new method for each test (ie, test_type_hello_world). You will also notice the setUp method. This is called before each test is run by the testrunner. In this case, we're launching the "Text Editor" application before we run each test (self.start_app("Text Editor")). Finally our test (test_type_hello_world) is simply sending keystrokes to type out "Hello World".

From this basic shell we can add more testcases to the helloworld testsuite easily by adding a new method. Let's add some simple ones now to show off some other capabilities of autopilot to control the mouse and keyboard. If you branched the bzr branch, there is a few more tests in the test_example.py file. These demonstrate some of the utility methods AutopilotTestCase makes available to us. Try running them now. The comments inside the file also explain briefly what each method does.

autopilot run helloworld.test_example.ExampleFunctions.test_keyboard
autopilot run helloworld.test_example.ExampleFunctions.test_mouse

Now there is more that autopilot can do, but armed with this basic knowledge we can put the final piece of the puzzle together. Let's create some assertions, or things that must be true in order for the test to pass. Here's a testcase showing some basic assertions.

autopilot run helloworld.test_example.ExampleFunctions.test_assert
  
Finally, there's some standards that are important to know when using autopilot. You'll notice a few things about each testsuite.

• We have a folder named testsuite.
• Inside the folder, we have a file named test_testsuite.py
• Inside the file, we have TestSuite class, with test_testcase_name
• Finally, in order for autopilot to see our testsuite we need to let python know there is a submodule in the directory. Ignoring the geekspeak, we need an __init__.py file (this can be blank if not otherwise needed)

Given the knowledge we've just acquired, we can tackle our first testcase conversion! For those of you who like to work ahead, you can already see the conversion inside the "firefox" folder. But the details, my dear Watson, will be revealed in due time. Until the next post, cheerio!

Read more

So, as has been already mentioned, automated testing is going to come into focus this cycle. To that end, I'd like to talk about some of the tools and methods for automated testing that exist and are being utilized inside ubuntu.

I'm sure everyone has used unity at some point, and you will be happy to know that there is an automated testsuite for unity. Perhaps you've even heard the name autopilot. The unity team has built autopilot as a testing tool for unity. However, autopilot has broader applications beyond unity to help us do automated testing on a grander scale. So, to introduce you to the tool, let's check out a quick demo of autopilot in action shall we? Run the following command to install the packages needed (you'll need quantal or raring in order for this to work):

sudo apt-get install python-autopilot unity-autopilot

Excellent, let's check this out. A word of caution here, running autopilot tests on your default desktop will cause your computer to send mouse and keyboard commands all by itself ;-) So, before we go any further, let's hop over into a 'Guest Session'. You should be able to use the system indicator in the top right to select 'Guest Session'. Once you are there, you'll be in a new desktop session, so head back over to this page. Without further ado, open a terminal and type:

autopilot run unity.tests.test_showdesktop.ShowDesktopTests.test_showdesktop_hides_apps

This is a simple test to check and see if the "Show Desktop" button works. The test will spawn a couple of applications, click the show desktop button and verify clicking on it will hide your applications. It'll clean up after itself as well, so no worries. Neat eh?

You'll notice there's quite a few unity testcases, and you've installed them all on your machine now.

autopilot list unity

As of this writing, I get 461 tests returned. Feel free to try and run them. Pick one from the list and see what happens. For example,

autopilot run unity.tests.test_dash.DashRevealTests.test_alt_f4_close_dash

Just make sure you run them in a guest session -- I don't want anyone's default desktop to get hammered by the tests!

If you are feeling adventurous, you can actually run all the unity testcases like this (this will take a LONG TIME!).

autopilot run unity

As a sidenote, you are likely to find some of the testcases fail on your machine. The testsuite is run constantly by the unity developers, and the live results of commit by commit success or failure is actually available on jenkins. Check it out.

So in closing, this cycle we as a community have some goals surrounding easing the burden for ourselves in testing, freeing our resources and minds towards the deeper and more thorough testing that automation cannot handle. To help encourage this move of our basic testcases towards automation, the next series of blog posts will be a walkthrough on how to write Autopilot testcases. I hope to learn, explore and discover along with all of you. Autopilot tests themselves are written in python, but don't let that scare you off! If you are able to understand how to test, writing a testcase that autopilot can run is simply a matter of learning syntax -- non-programmers are welcome here!

Read more

It all started innocently enough. A simple idea, turned into a simple post on the mailing list. This idea eventually led the ubuntu QA community to perform an experiment for this cycle, which has been dubbed "cadence testing".

Now, before I manage to confuse everyone with this "cadence testing" term, let's define cadence testing. Scratch that, let's just give a simple definition of what was intended by original idea. If you want the whole story read the thread. heh. I'll be waiting (hint it's LONG!).

Cadence testing was intended to introduce regularity into testing. If the development release could be "stable" everyday (which was the grand experiment during the precise cycle), could we not also test to ensure that things were good all throughout the release? If the everyday images and archive were now to the quality of previous release's milestones, could we just eliminate the milestone idea and go with a calendar schedule for testing? Thus, a proposal was made test every 2 weeks, whether or not a milestone had been planned, and report the results.

Fast forward 2 months to today. So what happened? Well, I'm happy to report that the QA community despite the confusion more or less met the goal of testing the desktop images every 2 weeks (milestone or not). But what did this achieve? And where are the results?

Let's step back a moment and talk about what we learned by doing this. My comments are specific to the non-milestone cadence testing weeks. First, the development process inside ubuntu is still built around milestones. The daily images during cadence testing weeks were sometimes stable, and sometimes flat out broken by a new change landing from a development team. Second, the tools we used are built around milestone testing as well. The qatracker as well as any qa dashboard or report available doesn't have a good way to track and image health across the cadence week. This meant it was both difficult to test and difficult to see the results of the testing. Finally, the development teams were not expecting results against the daily images, and couldn't follow-up well on any bugs we reported, nor where we able to coordinate well with the release team, as the bugs reported were not available in a summarized or meaningful way.

Now, I'll save the discussion on my ideas of a healthy QA workflow for a later post, but I think we can agree that testing without good result reporting, and without developer follow-up has a limited impact. So does this mean "cadence testing" was a bad idea? No, it was simply poorly executed. The trouble comes in the assumptions listed above.

The archive has not been "stable" everyday, and development teams have have continued development, pausing only as required by the current milestones. In addition, changes, even major ones (like the ubiquity changes landing a few weeks ago, or the nvidia change just this past weekend), are not well communicated. Since they land without little or no warning, we as a QA community are left to react to them, instead of planning and executing them. In this environment, cadence testing makes little sense.

So was the experiment a failure then? In my mind, not at all! In fact, I think the future of ubuntu and QA is to push for complete adoption of this idea, and this experiment confirms the obstacles we will face in getting there. I'll be posting more about what this vision for QA looks like, but I'll leave you with a few thoughts until then.

In my mind, QA should enhance and improve developers, testers, and users lives and workflows. Our work is critical to the success of ubuntu. I would like to see a future where users receive regular, timely scheduled updates that the folks in the QA community have vetted by working with the development and release teams to deliver focused quality updates. The ideal workflow is more fluid, more agile and yes, it has a cadence.

Read more

A couple Fridays ago I asked for feedback on how quality and the ubuntu QA team this cycle. That survey has now been completed and I have some results to share with everyone. Before I dive into the numbers, let me take a moment to say thank you to all of you who responded. Thank you! I read all of the comments left as well, and all were helpful feedback. Remember the survey was anonymous, so I cannot respond individually to anything written. Feel free to contact me if you wish to discuss anything further or to receive a response.

The second question on the survey asked rather simply, "What does quality mean to you?".

As it turns out, the largest answers mirrored those of a later question, in which I asked "What's the biggest problem with quality in ubuntu right now?".

Note, I read all of the "other" responses and categorized them into some new categories to display here for general consumption.

So there is some agreement amongst those who were polled both about what quality means, and about where ubuntu's biggest problems lie. The respondents indicated the largest issue with quality in ubuntu, according to them, was also the definition of what "quality" is!

Now I asked this question for a specific reason. "Quality" is a subjective term! Perhaps I'll get some disagreement on this, but hear me out. All of the answers for the question in my mind are valid with respect to quality. As an example, let's say, I asked you to recommend software to balance my checkbook. If I specified I wanted a quality piece of software, would you not recommend to me a stable (works without crashing), mature (good development/bug workflow), and easy to use (just works) piece of software that has a nice feature set (latest and greatest)? It's easy to see that "quality" can refer to all of this and more.

Still, in my mind, when I speak to wanting a "quality" release of ubuntu, I tend to focus on the stability and ease of use aspects. As the graphs indicate, the respondents seemed to echo this idea. In other words, it's really important to the idea of quality that things "just work". In the context of ubuntu this means applications run without crashing, and the operating system runs on your hardware. If things don't "just work", even if all the other indications of quality are true, you aren't likely to describe or perceive the product as having "good quality".

Let's ponder that thought for a moment and look at some more results. The survey captured about a 50/50 split of folks who run the development release, and over 70% run it or intend to run it before the final release.
So among those 50-70% who run or will run the development release, how many have participated in ubuntu qa?
Yikes! Only about a third. Just under half have no idea a ubuntu QA team existed. There's some clear evangelizing work to be done here. Let me take pause here just for a moment to say the team does exist, and would love to have you!

Ok, now onto the last multiple-guess multiple-choice question.
I'm happy to see people desire to help! That's wonderful. The responses regrading time, being technically able, or where to start are all very solvable. I would encourage you to watch this space for invitations to help test. QA work comes in all shapes and sizes, sometimes it's as little as 15 minutes, and the ability to install/uninstall a package and reboot a machine. If this sounds like something you would be able to do, please start by having a look at our wiki page. Send us an email and introduce yourself. There's no requirements or forced participation and we welcome everyone. And who knows, you might even learn something about ubuntu :-)

Ok, so I've shared the hard numbers from the survey, but I'd like to leave you with a few takeaway thoughts. First, while quality is subjective, our focus in ubuntu QA should be to have things "just work". That doesn't mean we shouldn't also help improve our development and bug processes, or continue to push for new applications and features, but rather that we ensure that our efforts help forward this cause.

I've said it before, but I want to help deliver good computing experiences. That story I shared when I introduced myself was close to home. My first interaction with the ubuntu community came via the forums, and yes, getting a printer to work. The community undertook work to change what was once a nightmare not for the feint of heart to child's play. The execution of this work is what defines the experience. This is where QA fits. We aren't just testing; we're delivering the result of the entirety of the ubuntu community's labor.

Judging from the survey results, many of you share this same vision. So won't you join us? QA transcends across teams and the ubuntu community. I would encourage you to get involved and be a part of making it happen. The list of "problems with quality" reach many areas. Would you be part of the solution?

Read more

About 2.5 months ago I wrote about the plans for the ubuntu QA community for the quantal cycle. We were building off of lots of buzz from the precise release and we planned to undertake lots of new work, while being very careful to avoid burnout. Our focus was to take QA to the next level and help us communicate and grow as a team to take on the opportunities we have.

So, how are we doing? Let's go over each of the points noted in the original post and talk about the progress and plans.

ISOTesting
Our alpha1 testing went very well, but the alpha2 and alpha3 have seen less participation. In addition we were asked and responded to a plan to test our iso's every 2 weeks as part of a more cadenced testing. Overall, isotesting continues to be a weak spot for us as a community. ISO Testing is perhaps the most important piece of testing for us as a greater ubuntu community. The image we produce is literally the first experience many folks have with ubuntu. If it fails to install, well, there went our chance for a positive first impression :-( I would be happy to hear ideas or comments on isotesting in particular.

Application Testing
This work has been mostly completed. The package tracker now allows us to perform work that was done via checkbox or manual testing last cycle. We can now manage results, tests and reporting all in one tool -- and it's all publicly available. For more information about the qatracker, see this wiki page.

SRU Verification
This work is still on paper, awaiting for the 12.04.1 release before further discussions and work will begin.

General Testing (eg, Day to Day running of the development version)
I am still experimenting with understanding how to enable better reporting and more focused testing on this. The current plan is to track specific packages that are critical to the desktop, and allow those run the development version the ability to report how the application is working for each specific upload during the development release. This is done with the qatracker. I'll blog more about this and the results in a later post. Contact me as always if your interested in helping.

Calls for Testing
This has been a wonderful success. There have been several calls for testing and the response has been wonderful. A big thank you to all of you who have helped test this. We've had over 50 people invovled in testing, and 41 bugs reported. Myself and the development teams thank you! But we're not done yet, unity testing among other things are still coming!

QATracker development
There is still room for more developers on the qatracker project. It's written in drupal, and I would happy to help you get started. As we grow, there will continue to be a need for people who want to build awesome tools to help us as a community test. If you have ideas for a tool (or know of a tool) that would help us test, please feel free to share with me.

Hardware Database
Work has been completed to spec out the design, and is scheduled now to land this cycle not in a future cycle. Fingers crossed we'll sneak this in before we release quantal :-) I'm very excitied to share this new tool with you; as soon as it's complete we'll be able to incorporate it into our workflow on the qatracker.

Testcases
Done, and for the most part our testcases have been migrated over. In addition, there is now a special team of folks who help to manage and maintain our testcases. If you have a passion for this work, contact me and I can help get you involved with the team.

Overall, I am happy to see signs of growth and newcomers to the community. If your on the fence about getting more involved with ubuntu, I would encourage you to check out QA. We collaborate with almost every area of ubuntu in some way, and no two days are the same :-) Send an email to the ubuntu-qa mailing list and introduce yourself.

So what's your opinion? Feel free to respond here with your thoughts and/or fill out the quality survey to give feedback.

Read more

What's your perception of quality this cycle? Are things working well for you? It's been several months now since precise landed, and ubuntu development for the next version has been ongoing. The ubuntu QA team has had a busy summer putting into place the new tools we spoke about at UDS. The qatracker has been revamped to allow us to consolidate our testcases and test reporting across all of our activities. In addition, we've been helping in the release of 3 alpha milestones, and 3 testing campaigns. To all those who have helped in this testing, a very big thank you!

I have my own thoughts about the impact to the ubuntu project this testing has had, and I will continue to share my thoughts to point out the progress we make in this regard. But now, I want your input. I have created a survey to understand the community perspective on how we as a ubuntu project are doing on quality. If you have a few moments, please fill out the survey and let your thoughts and perspective be known. The survey will be anonymous, but I will share an aggregation and summary of the results.

My hope is to help gain an understanding of how we can focus our efforts on what's important to ubuntu as a project in terms of quality, as well as how we can help you (yes you!) become a more active part of QA if your interested.

Here's a link to survey. I'll leave it open until next Friday August 10th. Thanks in advance for your participation.

Read more

Notice I'm not mentioning a good video walk-through, but it is in HD! It can be viewed on youtube here. I had a lot of trouble doing my first screencast, but I think I have a promising setup now for the future. I promise HD, lag-free, and better editing in the future. For reference, I highly recommend flowblade to edit and kazam to record. Kudos to both developers for nice pieces of software!

The text version is available here and I encourage you to follow along with the video. You don't need anything more than a reasonably modern pc that can run a VM and you can help iso test. Take 20 minutes to watch and follow along and take the fear out of iso testing. Then join us next week as we continue our testing cadence and verify our daily isos are in good shape.

And don't worry, no one will know it's running the installation on a separate workspace while you watch youtube videos.. It'll be our secret.

Read more

Quality has been a buzzword for a couple releases now. Certainly, it's safe to say we saw many more people talking about quality last cycle than any previous cycle I can remember. Ubuntu of course has done LTS's in the past, but something about this past release was different. People yearned again for the perceived quality of the past LTS release and wanted to see ubuntu succeed.

Ahh yes, the good ole days of 8.04 or 10.04 LTS releases when everything was right in the world and ubuntu just worked, etc. Heh, it's easy for us to remember the past and the good things we enjoyed. But to the extent those releases served me well, I have the community to thank. Ubuntu is community, and it's success or failure is determined by all of us.

Open source projects are not typically known for there quality. In the same way ubuntu has innovated and paved the ground for a consumer focused desktop linux, I believe we as a community are uniquely positioned to show how open source can be better quality than competing ideological offerings. In the same way a minuteman can win a battle against a better equipped mercenary, so too can a dedicated community provide better quality software than a commercial offering. This only makes sense. I would rather work with a group of passionate people than with folks who don't care about there work.

One of the best parts about getting to work in QA is seeing the end result of the entire communities work. Those who perform QA represent the last pair of eyes for the work our community does. The developers, translator, doc writers, bug triagers, forums and IRC admins can all do marvellous work in support of ubuntu. But if it doesn't install/work on your pc, then sadly you won't be able to enjoy any of that work.

Recently, those running the development version of ubuntu noticed a bug that caused there webcam to no longer work. The community responded by helping figure out exactly what commit caused the regression. Many folks were involved here! People with the issue tested different kernel versions to narrow down when the regression occurred, while the kernel team made these kernels available and provided insight. Ultimately with everyone's help the problem code was identified and confirmed by testing a custom built kernel. Armed with this knowledge an upstream bug was created.

Now as I sit here today, my webcam works again! And not just for me, or other ubuntu users, but for everyone who uses the linux 3.5 kernel. After upstreaming the bug,  the original developer made a fix available which we tested. Ultimately, this fix made it into the 3.5 kernel. This prevented the 3.5 kernel from shipping with broken logitech webcams for everyone. This is the power of our community!

The story is but one of many examples of problems the ubuntu community has solved. If you enjoyed or remember the good ole days in ubuntu, I would encourage you to get involved. If your running ubuntu on your desktop now and want to continue doing so, consider donating your skills to help. A healthy QA community is vital for ubuntu to continue to grow and get better. We, as a community, can be the standard for quality in open source.  So how can we take ubuntu, and specifically quality in ubuntu, to the next level? It starts with you.

Read more

EDIT: This has been migrated to the production instance. Check out the new post here to get invovled.

The first calls for testing for this cycle are happening! I am excited to not only announce the opportunity to help test this new kernel, but also to unveil some of the new features to the qatracker to allow us to better serve our needs for calls for testing.

Last cycle, calls for testing was a manual thing -- I asked, and the community responded, following along using instructions from a blog or mailing list post. Now we're going to put some more structure around this as I spoke about earlier.

The kernel team is committing to keeping precise up-to-date by providing kernels from the future releases in precise. The first one of these will be the 12.10 kernel which will land in precise as part of the normal 12.04.X update. In order for that to happen, the team is making available kernels to test on 12.04. If you need a newer kernel for hardware enablement, this is the kernel you are encouraged to run and report on. For those running the mainline kernel, this differs in that the ubuntu patchset and official support for this kernel running on 12.04 will happen when it is pushed via update.

So we're really testing a couple things here -- the 12.10 kernel on 12.04, but also the new qatracker. Feedback is encouraged on the qatracker also! Ok, so how does this work?

First, you'll need to use the staging site for the qatracker, were a package tracker has been setup. If you click on the 'Quantal kernel for precise LTS' link, you will wind up on a page showcasing the tests and instructions for this call for testing. If you click the 'Link to the installation information' link you will get information on installing and uninstalling the package and filing bugs against the package. Especially note the instructions for filing a bug properly; additional information is requested to help make your bug report more helpful to the development team. If you click the 'Kernel Smoke Tests' link you'll arrive on the page with the instructions for the testcase. If you login to the tracker using your ubuntu SSO credentials, you will be able to report results as well. This should look very familiar to those of you who have used the isotracker in the past. Neat eh? If you have any issues in using the tracker, feel free to get in touch with me.

I'm asking those folks willing to help test please head over to the qatracker and submit results. Note that the qatracker emails are turned off, but otherwise everything should function as expected for you. To leave feedback on the new site, file a bug and mention your using the new staging qatracker. Contributions to the qatracker are welcome and encouraged, contact me if your interested in helping out.

Read more

It's almost hard to believe, but the new cycle is starting to ramp up. In just over one week's time, we'll be putting out an alpha 1 iso for quantal! If you will remember last cycle, I began an adopt an iso campaign to help insure precise got the iso testing coverage it needed to be an awesome release. This cycle, that campaign will continue with an open call for folks to adopt an iso and help test it all cycle long. Instead of managing and updating all of our excellent testers myself via email, I am asking instead for you to subscribe to the iso your interested in adopting and helping to make sure it's in a ready state for each milestone release. Subscribing to the iso will alert you via email when there is a new build for you ready to test, so you don't need to watch the page or await an email from me. If you miss the email from me, you may of course contact me for personal interactions at anytime you wish :-)

Interested? Awesome! This wiki page should detail everything you need to get started. Specifically, you should ensure you subscribe to the testcases for the iso you wish. For example, if I am interested in Ubuntu Desktop i386, I would head over to this page. See that button at the bottom called subscribe? Hit it and you should be subscribed to new builds for that iso. Please note the subscription feature is a work in progress, and there is not (yet!) a management page for subscriptions. Additionally, there is no visual indication on the page that your subscription is active (contributions welcome, contact me if you know drupal and wish to help!). Please read the wiki page on ISO Testing for more information on confirming your subscription in the interim period.

Ok, great, so now your subscribed. There's just one piece left in making sure things go well for your iso. This cycle we are trying something new to help make the alpha, beta, final release process smoother for all of our iso's (and you!). We would like to have our adopters run the daily iso before we spin the first candidate for release. What this means for you is that the week before each milestone release date, go ahead and try testing out the daily version of the iso. Think of it as warm-up for the big day. After all, you don't want your iso to be the one causing the re-spin do you?

This schedule will come in handy. It shows the timeline for how and when we'll be undertaking this testing. I know it looks big and scary, but focus on just the first column called 'Community Testing'. See the week of May 31st with the 'Q-D' listed by it in the 'Community Testing' column? If you glance at the 'Legend' at the top you will notice that 'Q' stands for quantal, and 'D' stands for daily. You will also notice that the Alpha 1 release for quantal is schedule one week after. So our goal is to spend this week getting our iso's in shape for the first spins for Alpha 1. The good news is time spent now is respins saved later. Happier isos make happier users!

Before I close I do want to remind everyone that each cycle is a marathon. We spoke at UDS about burnout, and that is something to keep in mind. Pace yourself and share the work. We'll have a wonderful cycle together. As always, contact me if you need any help. Your response was overwhelming for ubuntu precise, let's keep going strong for quantal. Thanks for helping make ubuntu better for everyone!

Read more

I originally posted this wonderful wall of text on the ubuntu-qa mailing list. If you want to get invovled in QA on ubuntu this cycle, you should subscribe to that list. Additionally, sign-up for the ubuntu testing team. Monitoring this blog and @UbuntuTesting will also keep you informed.


ISOTesting
My goal is to help ensure things are smooth before milestones, and
before isotesting events. Before we spin an iso, we want to feel good
about what's going on that iso. And we as a community can help make that
happen. Overall, I want each individual to have a lighter workload than
last cycle, despite having a similar amount of overall work we need to
achieve. To do this I'd like to help enable more people to be testing,
and to expand the 'adopt an iso' program so that folks can focus on
testing things they like and are able to test without becoming
overwhelmed or burnt out. Additionally, respins will be a continuous
focus and communication of what has changed and what needs tested will
be a priority. As a community we want to avoid doing re-work/extra work
and dedicate ourselves to performing quality testing, not merely having
a large quantity of testing.

Application Testing
Last cycle we utilized checkbox to deliver manual application tests.
During UDS, we spoke of expanding the isotracker to do our testcase
management, and thus consolidate our application testing by using the
same tool used for the isotracker to create an application tracker. This
work is on-going, but should be finished at some point during the cycle
so we can adopt it and use it. In the interim period will be continue
utilizing checkbox or doing manual testing via blogs or mailing lists, etc.

SRU Verification
SRU verification is currently a manual process with a high learning
curve and little visibility for many people. During the cycle, we hope
to help change that but also utilizing a new tracker to do SRU testing.
This testing will involve running the stable version (currently precise)
of ubuntu, but testing fixes to individual packages. This makes it a
good fit for those who aren't living on the bleeding edge but wish to
help. When this process is ironed out (sometime during the cycle) I will
contact everyone again with information on howto get involved.

General Testing (eg, Day-to-Day running of the development version)
Some good feedback was given on how to help make this better. There are
a few things we would like to do to help improve this process. First,
day to day changes should be able to be followed easier with some
proposed changes to update-manager to better display changelogs for
updated packages. I'll be detailing some information about how
'whoopsie' works and what it means to you. In addition, keeping the
development release stable at all times will continue to be a priority
for the development teams.

Calls for testing (specific feature or new features of critical package
or focused testing on a specific package)
Last cycle this typically involved me posting and laying out a basic
testplan on my blog with instructions on how to help test. This cycle,
again we hope to consolidate this onto a tracker where the tests and
results can be recorded. I will still be utilizing my blog, the
@ubuntutesting twitter account, this mailing list and our IRC meeting to
publicize events like this for people to get involved and contribute.
It's always fun to see new features before they come to everyone else,
and the feedback loop with the developers was welcome on both sides.

QATracker Development
With these changes to the qatracker, there is room for some folks who
know python and django to get involved and help improve the qatracker
codebase to make testing and reporting easier  Contact me, or simply
have a look at the code on launchpad and start hacking.
lp:~ubuntu-qa-website-devel/ubuntu-qa-website/drupal7-rewrite
lp:~ubuntu-qa-website-devel/ubuntu-qa-website/drupal7-rewrite-testcase-management
lp:~ubuntu-qa-website-devel/ubuntu-qa-website/python-qatracker
lp:~ubuntu-qa-website-devel/ubuntu-qa-website/python-qatracker-testcase-management

Hardware Database
The idea for having a hardware database for testing is not a new one,
but work has begun anew. This is work that will go beyond this cycle,
but ideas are being explored at using ubuntu friendly and other tools to
make this a reality.

Testcases
As a testcase management system will soon be in place (hurray!), we'll
be migrating all of the testcases over to this system. That means will
have much better visibility and ease of maintenance for all of our
testcases. Cleanup and expansion of the number of testcases is
definitely a goal for the cycle, and expect to hear more about getting
involved in this area.

Whew, that's a wall of text, but I hope it helps outline what the plans
are for the cycle. Feedback appreciated and encouraged  Happy Testing!

Read more

UDS is now behind us, and the excitement of the work that lies before us for the next cycle is fresh on our minds and hearts. Last cycle I solicited and received some amazing ideas for improving how we as a community do QA inside of ubuntu. As UDS neared I encouraged many of those with ideas to participate in UDS by attending, signing up for work items, and advocating their ideas.
This is a key portion of being a part of the community -- you must be willing to act. If you are unwilling to act upon your own idea, why would anyone else? If you don't believe in it, no one else will. Own the problem you wish to solve and you will find others who share your passion along the way to help you achieve your goals. This is the heart of open source.
But how? How can I act? What if the problem is outside of my skillset? Because of the greater community and the nature of open source, you don't have to solve all of the problem by yourself. As you undertake work to execute your idea, you will find it attracts those who are of like-mind and similar persuasion to you. The best part is that they will have different skillsets to bring to the problem and can help you accomplish more than you could alone.
In a previous job, I was given the freedom to spend a percentage of my time on anything I chose; provided I could convince two of my workmates to help out. The idea behind the requirement was a litmus test for my idea. If the idea has merit, I should be able to convince my colleagues to work on it with me. Ubuntu is one of several open source projects to operate on this idea of 'meritocracy'. The basic premise is to have the best people making the most informed decisions possible about problems specific to there expertise. This is achieved by granting authority to make decisions to anyone who demonstrates there ability to do so by contributing to the project.
So, returning to UDS I would like to tell you a small story of just one example of executing on an idea. Let me introduce Paolo Sammicheli to you. Paolo is from the Italian Loco team, and has been active in driving growth in the localized iso community. He began his work by starting an "Italian Testing Team" several UDS's ago, and has been advocating greater testing and community participation for several cycles now. This past UDS, Paolo wanted to help kickstart a localized iso community beyond just his Italian loco iso. Before UDS, he had already produced a set of wiki pages documenting how to use the isotracker admin features with a bent towards running your own localized iso tracker. Additionally, the Italian loco team planned and tested during the 12.04 cycle to create a localized ubuntu 12.04 image for release. Finally, Paolo came to UDS and created a blueprint so he could share his idea with others. Have a look at it yourself:

https://blueprints.launchpad.net/ubuntu/+spec/community-q-localized-iso-community-growth

Paolo was able to generate good ideas, and see other people attempt to replicate his work within their own locos. Plans were made to have two other loco teams produce localized isos this cycle, and ultimately use there findings as a model for future loco teams. Although the work is on-going this cycle, Paolo, I think, has been successful at bringing his idea to life.
How can you replicate Paolo's example? A couple key points I see in what happened.

• Lay the groundwork
• Start proving the idea out as best you can. Perhap's it's a demo or prototype -- maybe even just a specification or a storyboard. You need to convince yourself (and others!) your idea makes sense and can be done
• Tell others
• Let others know about your work. Blog about it, come to UDS, present it at a Ubuntu user days event, post it to the forums, talk to people on IRC about it, etc
• Do it
• This is key. You need to start executing your idea as best you can. People are not going to make your idea a reality without you! (and why would you want them to? It's your idea! Own it :-) )
• Share your work
• Invite others to work with you on your idea. It's helpful to have specific and easy ways to get involved, but don't limit people. You want to work openly in a way that anyone can participate at any level.

Read more

Your commitment to quality and excellence has shown itself in this release. People love numbers, so let me spill some!

We had 13 calls for testing this cycle, with 6 iso testing milestones, and lots of bug reports, support and users using the development version of precise. Additionally, we set a new record for iso testing the precise final isos! Have a look yourself!

https://wiki.ubuntu.com/QATeam/ReleaseReports/PreciseFinalTestReport

That's 114 people who helped make sure your ubuntu precise experience was going to be a good one out of the box. Thank you to all of our testers this cycle!

https://wiki.ubuntu.com/PrecisePangolin/ReleaseNotes/Credits/Testers

Read more

With the renewed focus on quality this cycle, things might start to look a bit confusing for everyone as to who is doing what in regards to quality in ubuntu. If your interested in helping out, it's certainly easy to get lost. Let me do my best to clear the waters and shed some light on what teams exist in the quality realm and what they are doing. First, a list of teams:

• Ubuntu QA Team
• Canonical Platform QA Team
• Ubuntu Testing Team
• Ubuntu Laptop Testing Team
• SRU verification Team
• Product Strategy Quality Team
• Ubuntu Bug Squad
• Ubuntu Friendly Squad
• Ubuntu+1 Testing Team(s)
• Ubuntu+1 Maintenance Team
• Ubuntu Localized Image Testing Teams
• Ubuntu Flavors QA Teams
• Edubuntu
• Lubuntu
• Xubuntu
• Kubuntu
• Ubuntu Studio
• Mythbuntu

Let's march down the list one at a time and discuss what each team does. If any of the descriptions sound interesting, be sure and follow the links to find out more information and/or to join the team.

Ubuntu QA Team
Mailing List: https://lists.ubuntu.com/mailman/listinfo/ubuntu-qa
IRC Channel: #ubuntu-testing
Launchpad page: https://launchpad.net/~ubuntu-qa
Wiki: https://wiki.ubuntu.com/QATeam
Blog: http://qa.ubuntu.com
This team’s mission is to help test ubuntu. The testing takes many forms and the team helps maintain a set of manual test cases usable for many different types of testing.

Canonical Platform QA Team
Launchpad page: https://launchpad.net/~canonical-platform-qa
This team is made up of Canonical employees who are performing qa for ubuntu. They are responsible for helping keep the testing infrastructure going, as well as coordinating and performing daily smoke testing, SRU's, and iso testing.

Ubuntu Testing Team
Launchpad page: https://launchpad.net/~ubuntu-testing
This team is a focus group of the ubuntu qa team specializing in performing iso testing, SRU testing, as well as manual application testing.

Ubuntu Laptop Testing Team
Launchpad page: https://launchpad.net/~ubuntu-laptop-testing
Wiki page: https://wiki.ubuntu.com/Testing/Laptop
This team is a focus group of the ubuntu qa team specializing in ensuring laptops work properly with each ubuntu release by testing isos for basic functionality across a wide range of laptops.

SRU Verification Team
Launchpad page: https://launchpad.net/~sru-verification
Wiki page: https://wiki.ubuntu.com/StableReleaseUpdates
People in this team perform the verification of packages that are candidates for a Stable Release Update. The candidate packages exist in the -proposed repository and need testing before they are moved to -updates.

Product Strategy Quality Team
Blog: http://qualityhour.wordpress.com/
IRC channel: #ubuntu-testing
This team is made up of Canonical employees from the product strategy team. They are responsible for things like unity and enhancing the end user experience inside ubuntu. The quality team is specifically focused on making sure all of those ideas and features are put to the test before being deployed to the greater community of end users. I am personally excited to see this team focus on quality and solicit feedback and help from the community. If your interested in helping improve the ubuntu user experience, this team's mailing lists and IRC meetings are a great place to start.

Ubuntu Bug Squad
Launchpad page: https://launchpad.net/~bugsquad
Wiki: https://wiki.ubuntu.com/BugSquad
Mailing List: https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugsquad
The bug squad! These wonderful folks pore over, triage, and assign bug reports. They act as the critical piece of communication and help between developers and users.

Ubuntu Friendly Squad
Launchpad page: https://launchpad.net/~ubuntu-friendly-squad
Website: https://friendly.ubuntu.com/
Wiki: https://wiki.ubuntu.com/UbuntuFriendly
This team is focused on making it easier to find and use laptops and computers that work "out of the box" with ubuntu. If you have ever had the experience of having ubuntu not boot on that old laptop, or boot to a black screen even though your desktop pc works marvelously, this is the site for you. I recently made use of the site to find out just how well supported the laptop I wanted to purchase was, in addition to which components I should look for (and avoid!) in order to have ubuntu work with my hardware.

Ubuntu+1 Testing Team(s)
IRC Channel: #ubuntu+1
Forum: http://ubuntuforums.org/forumdisplay.php?f=412
These team(s) are composed of people willing to be on the front lines and always running the next version of ubuntu. They provide help and support amongst each other, and valuable bug reports and first responder feedback to the development teams. The structure is rather informal, and the requirements are low -- you simply need to commit to running the development release and learn how to submit good bug reports.

Ubuntu+1 Maintenance Team
IRC Channel: #ubuntu+1-maint
Wiki: https://wiki.ubuntu.com/PlusOneMaintenanceTeam
This team is brand new for this cycle, and they have a very specific goal that is key to delivering quality in precise. They're goal is to have the archive be ALWAYS installable and usable. For anyone who has ever run the development version of ubuntu in the past, you know how difficult a task this is! Thus far the team has done an excellent job, and it's been noticeably easy to run precise throughout the cycle.

Ubuntu Localized Image Testing Teams
Wiki: https://wiki.ubuntu.com/PrecisePangolin/LocalizedImageContacts
Ubuntu's wonderful LoCo teams showoff the true meaning of ubuntu and the spirit of FOSS. In addition to the many other activities LoCo's do, some are providing localized iso's for there users. If your a non-English speaker, you understand what a blessing this can be when your having to install ubuntu. Since these images contain additions to the standard iso image, they must also be tested for bugs.

Ubuntu Italian Testing Team
Launchpad page: https://launchpad.net/~ubuntu-it-testing
This team is a wonderful example of a loco team doing testing for ubuntu. They organize goals each cycle typically surrounding creating a localized Italian iso, as well as doing laptop testing to ensure ubuntu works on common laptop configurations.



Ubuntu Flavors QA Teams
Do you like using ubuntu, but find yourself using one of the official flavors of ubuntu instead? These teams produce there own isos and packages, in addition to supporting different subsets of software. Each flavor has there own method of doing QA, but the goal is the same -- to deliver quality releases for there users.

Edubuntu
IRC Channel: #edubuntu
Webpage: http://www.edubuntu.org/

Lubuntu
IRC Channel: #lubuntu-qa
Launchpad Page: https://launchpad.net/~lubuntu-qa
Wiki: https://wiki.ubuntu.com/Lubuntu/Testing
Mailing List: https://lists.launchpad.net/lubuntu-qa/

Xubuntu
IRC Channel: #xubuntu-devel
Webpage: http://www.xubuntu.org/contribute/qa_bugs_testing
Mailing List: https://lists.ubuntu.com/mailman/listinfo/xubuntu-devel

Kubuntu
IRC Channel: #kubuntu-devel
Webpage: https://wiki.ubuntu.com/Kubuntu/QA
Mailing List: https://lists.ubuntu.com/mailman/listinfo/kubuntu-devel

Ubuntu Studio
IRC Channel: #ubuntustudio
Webpage: http://ubuntustudio.org/

Mythbuntu
IRC Channel: #ubuntu-mythtv
Webpage: http://www.mythbuntu.org/testingandreporting

Whew, all done :-) It is worth noting that although I attempted to be comprehensive with this list, it is not exhaustive. Are you doing QA work in ubuntu that I didn't mention? Let me know about it! Or, perhaps you have an idea for some work that doesn't fit into one of the above. Let me know about your ideas as well! I have already found this community to be full of interesting people and ideas, and likely you will be able to find someone to help you move forward with your plan.

If any of this work interests you, please do contact the appropriate team via the links above or simply contact me directly and I can help get you connected. A big thank you to all of these people for their commitment to helping make ubuntu great!

Read more

As beta1 for precise draws closer and closer, it's time for testing some new stuff. On the docket is wubi, everyone's favorite way to get ubuntu without having to worry about disk partitioning or losing there current windows installation. Wubi was created out of a desire to lower the barrier of entry for newcomers to ubuntu. So, if you want to help create a wonderful experience for those new users, I would encourage you to help test wubi. The testing window for this goes through the weekend, as the first iso spins will be happening on Monday for iso testing :-)

Setup
Prerequisites: You will need access to a computer that has windows xp or higher installed on it. In addition, the computer will need at least 5 gb of free disk space. As always, make sure you have backups and can restore the computer if required.

1) Download a current daily image (http://cdimage.ubuntu.com/daily-live/current/). You can chose a 32-bit or 64-bit desktop cd.

2) Burn the iso you downloaded. If you need help burning the iso, check out this handy wiki article: https://help.ubuntu.com/community/BurningIsoHowto

3) Boot up the windows computer.

Testing
Once you have the cd and the windows computer is booted you can run through the testcases. For this round, we're focusing on 2 tests;  wdi-001 and wdi-002. Check out the wiki page and follow the instructions for each case. http://testcases.qa.ubuntu.com/Install/DesktopWubi
Once you've gone thru both testcases you should have a working ubuntu install alongside your windows installation.
Did everything work? Leave me a comment below and let me know about it. Failed? Check out how to file a bug below.

Filing Bugs
Please file bugs against the wubi package in launchpad (https://bugs.launchpad.net/wubi/+filebug). When filing, please make sure to tag your bug 'beta1'.

Giving Feedback
If you'd like to give feedback, comments or even patches, get in touch with the team on launchpad at https://launchpad.net/wubi.

Getting Help
Don't hesitate to reach out to the wonderful folks on launchpad on the wubi team, the ubuntu+1 forums, or myself if you have questions. Thanks for helping test ubuntu!

Read more

This tutorial describes how to produce test-coverage metrics for C/C++ projects. We start by instrumenting an autotools build to produce gcov output with gcc; we then generate test coverage artifacts by running our test suite, and finally we explore our results using lcov.

Introduction

We’d like to assess the quality of the existing test suite for each Product Strategy project. A measurement of test coverage will tell us what part of the project’s code is “covered” or exercised by its tests. 50% is a start; 80% would be a good goal for a neglected project; one rarely encounters 100% test coverage, but we’d like to get as close as we can. Initially we’ll use these findings to gain an overview of the test-quality of each project; ultimately these metrics will guide the improvement of our codebase, and enable us to monitor our progress using Jenkins and associated open-source tools.

A Three-Part Process in Several Steps

We’ll enable test-coverage of a particular C/C++ project in a three-part process:

• enabling a special build
• running the tests
• studying the output

Our first step will be to enable a special build. Ultimately this just means adding a few flags to our gcc invocation, but it never seems so straightforward with autotools projects . The Product Strategy Quality team has made available a set of files to facilitate this build–please check these out now if you’d like to follow along.

bzr co lp:coverage-tutorial

Inspecting this archive you’ll find

• gcov.m4: an autoconf macro which will check for relevant tools
• Makefile.am.coverage, which includes our coverage-enabled automake targets
• an old revision of dbus-test-runner
• a copy of this tutorial

Before we start let’s make sure we have lcov installed. lcov incoporates the GNU tool gcov and produces pretty HTML reports of our coverage results.

sudo apt-get lcov

(Note that the Debian lcov package includes genhtml.) We’re ready to begin.

1. Branch a project of interest

I heard you like testing, so for this example we’ll test dbus-test-runner which is a runner which runs tests against the dbus messaging system. An old revision is included in our coverage-tutorial archive; if you want to start fresh (or on a different project), you would

bzr branch lp:dbus-test-runner

2. Install the gcov.m4 autoconf macro and invoke it

Open gcov.m4: here’s where we’re defining the compiler flags and checking for necessary tools. Let’s put this file where autoconf can find it.

Your project may already have a directory for m4 macros:

$ grep AC_CONFIG_MACRO_DIR *
configure.ac:AC_CONFIG_MACRO_DIR([m4])

If you find such a directive, simply copy the gcov.m4 file into the named directory. If not, create an m4 directory and copy the gcov.m4 file there–and don’t forget to include a reference to the directory in the body of configure.ac (it’ll look like the grep result above).

3. Massage configure.ac to include our necessary coverage flags.

This is the essential move of our special build. gcc supports coverage reporting with the addition of a few compiler flags. Here they are in the gcov.m4 file:

# Remove all optimization flags from CFLAGS
changequote({,})
CFLAGS=`echo "$CFLAGS" | $SED -e 's/-O[0-9]*//g'`
changequote([,])
# Add the special gcc flags
COVERAGE_CFLAGS="-O0 -fprofile-arcs -ftest-coverage"
COVERAGE_CXXFLAGS="-O0 -fprofile-arcs -ftest-coverage"
COVERAGE_LDFLAGS="-lgcov"

When the tests are run, --fprofile-arcs produces a tally of the execution of each arc of the code–one .gcda file for each source file. The --ftest-coverage flag produces .gcno files, which link an arc to a source line so that we can see which lines were touched. We’ll watch these files being produced in a little bit; you can read about the flags at length in the GNU documentation. Note that we also remove optimization with the -O0 flag to get more precise results.

Now here’s the step which requires knowledge both of your code and little bit of autoconf. We’ve included the flags above (in step 3), but we’ll now need to edit our configure.ac to make the flags available to our build. For dbus-test-runner the diff looks like this:

=== modified file configure.ac
--- configure.ac 2010-12-08 02:35:12 +0000
+++ configure.ac 2011-12-06 21:42:04 +0000
@@ -45,6 +45,16 @@
 AM_GLIB_GNU_GETTEXT
 ###########################
+# gcov coverage reporting
+###########################
+
+m4_include([m4/gcov.m4])
+AC_TDD_GCOV
+AC_SUBST(COVERAGE_CFLAGS)
+AC_SUBST(COVERAGE_CXXFLAGS)
+AC_SUBST(COVERAGE_LDFLAGS)
+
+###########################
 # Files #
 ##########################

And then having added these flags to the build process, we need to actually actually add them to the build proper:

=== modified file src/Makefile.am
--- src/Makefile.am 2009-12-07 21:00:43 +0000
+++ src/Makefile.am 2011-12-06 21:42:04 +0000
@@ -3,6 +3,8 @@
 dbus_test_runner_SOURCES = dbus-test-runner.c
 dbus_test_runner_CFLAGS = $(DBUS_TEST_RUNNER_CFLAGS) \
+ $(COVERAGE_CFLAGS) \
-DDEFAULT_SESSION_CONF="\"$(datadir)/dbus-test-runner/session.conf\"" \
 -Wall -Werror
 dbus_test_runner_LDADD = $(DBUS_TEST_RUNNER_LIBS)
+dbus_test_runner_LDFLAGS = $(COVERAGE_LDFLAGS)

Here we risk running afoul of the autotools “ancient ones”. This is an uncomplicated project–if you’re not getting the results you want in the next step, I can recommend The Goat Book as a decent tutorial on how these macros work.

4. Verify that autoconf.sh --enable-gcov generates .gcno files

Having enabled our special build with the gcc compiler flags, let’s verify that autoconf is generating the .gcno files we’ve asked for:

$ ./autogen.sh --enable-gcov
...
$ find -name *.gcno
./src/dbus_test_runner-dbus-test-runner.gcno

If all goes well we’ll find a .gcno for each .o file compiled. If not, then our flags aren’t being respected–maybe something’s wrong with our configure.ac addition in step 3. It’s important to have these checkpoints along the way to divide the autotools process into testable pieces.

5. Install Makefile.am.coverage

So now that we have the instrumentation we need, we’ll use a special build to run the tests. Copy Makefile.am.coverage into the top-level directory.

cp ../Makefile.am.coverage .

Inspection shows that this file defines some extra make targets to generate coverage results using lcov. We’ll alert automake to this by simply including the file–here’s the diff for dbus-test-runner:

=== modified file Makefile.am
--- Makefile.am 2009-04-23 21:19:56 +0000
+++ Makefile.am 2011-12-19 18:00:38 +0000
@@ -1,1 +1,3 @@
 SUBDIRS = data src tests po
+
+include $(top_srcdir)/Makefile.am.coverage

6. Verify that make check generates .gcda files

Now we’ll actually run the tests.

$ make coverage-html
...
$ find -name *.gcda
./src/dbus_test_runner-dbus-test-runner.gcda

Again these .gcda files represent a tally of ‘arcs’ through the code. Note that we can generate these files not just while running tests, but also during normal execution–you can see the kinship with profiling. lcov (via gcov) has used these tallies with our .gcno files to produce line-by-line results which we’ll study in a moment. Here’s what the test coverage looks like for dbus-test-runner:

Overall coverage rate:
lines......: 78.8% (215 of 273 lines)
functions..: 86.4% (19 of 22 functions)
branches...: 61.2% (60 of 98 branches)

For a small project these figures show that we have a good test-suite to build on. We’re curious about what we’re missing, though, so we’ll investigate below. Before you register a comment about “no coverage results”, recognize that you may actually have zero code coverage. But at least you’re able to measure it, riiight?

7. Have a look at the lcov pages

lcov has produced a coverage-results directory at the top-level of our project; open index.html with a web browser and explore.

At a glance our line coverage is 78.8%, and our “test-coverage progress bar” is yellow. Be aware of the difference between the offered metrics:

• line coverage: how many lines have our tests touched
• function coverage: how many functions have our tests touched
• branch coverage: for the graph which describes all possible paths of control through this file–especially through conditionals, e.g.–what percentage has our tests touched

In our opinion the killer feature of the lcov output is the source-file display, which shows which lines weren’t touched by tests. Drill into the source directory to see the results for a particular file.

Here in dbus-test-runner.c line 93 we’ve hit the line that tests the status of G_IO_STATUS_NORMAL 61 times, but never taken the path in the line below, for which G_IO_STATUS_NORMAL is false.

Conclusion

We’ve presented an introduction to test coverage, featuring the gcc toolchain and autotools–as demonstrated with a set of utilities we’ve found useful here in the Quality group.

I hope it’s obvious that this tutorial is just the beginning–not just of applying these methods to our code (we and you), but also of understanding what test coverage means to the improvement of Quality. Our team has witnessed that having an observable measure like the lcov green bars encourages developers to increase test coverage. However while chasing into a particularly narrow corner of their code, our developers have produced some confounding results which have caused us to doubt the accuracy of the available tools–I hope that Thomas Voss will follow with a post on some of his findings in the future.

Regardless, test coverage will be an important metric for our group for the coming cycles, and the bars will continue to grow greener as Spring approaches. . . . Meanwhile I’ll be interested to learn about your autotools/gcc/coverage experiences:

• What lcov alternatives have you tried?
• Which hallowed autotools rituals have I profaned in preparing this tutorial?
• What’s the highest coverage percentage you’ve witnessed in a GPL project?

Read more

Everywhere. It's not a cliché. Being the new guy, one of my first objectives was to survey the landscape of ubuntu and meet all the great folks who are already doing QA work. While I found some interesting teams whose work I hope to spotlight in future posts on this blog, the biggest surprise was the commitment to quality on every team I met. Perhaps this is just because the precise release will be an LTS; and not just any LTS, but the first to be supported for 5 years on the desktop. In my opinion, this is the biggest release since dapper changed how we looked at ubuntu!

Delivering a release that will help define ubuntu is not an easy task -- we need everyone's help. Perhaps your like me and just getting started (or maybe a longtime lurker who has never contributed). Regardless, you can help make this release the best ubuntu release yet. Get involved! There is something for everyone to contribute, for many different skills and time commitments. The real answer to where quality is found in ubuntu is in you!

Read more