With python-dbusmock you can provide mocks for arbitrary D-BUS services for your test suites or if you want to reproduce a bug.

However, when writing actual tests for gnome-settings-daemon etc. I noticed that it is rather cumbersome to always have to set up the “skeleton” of common services such as UPower. python-dbusmock 0.2 now introduces the concept of “templates” which provide those skeletons for common standard services so that your code only needs to set up the particular properties and specific D-BUS objects that you need. These templates can be parameterized for common customizations, and they can provide additional convenience methods on the org.freedesktop.DBus.Mock interface to provide more abstract functionality like “add a battery”.

So if you want to pretend you have one AC and a half-charged battery, you can now simply do

  def setUp(self):
     (self.p_mock, self.obj_upower) = self.spawn_server_template('upower', {})

  def test_ac_bat(self):
     self.obj_upower.AddAC('mock_AC', 'Mock AC')
     self.obj_upower.AddChargingBattery('mock_BAT', 'Mock Battery', 50.0, 1200)

Or, if your code is not in Python, use the CLI/D-BUS interface, like in shell:

  # start a fake system bus
  eval `dbus-launch`
  export DBUS_SYSTEM_BUS_ADDRESS=$DBUS_SESSION_BUS_ADDRESS

  # start mock upower on the fake bus
  python3 -m dbusmock --template upower &

  # add devices
  gdbus call --system -d org.freedesktop.UPower -o /org/freedesktop/UPower \
      -m org.freedesktop.DBus.Mock.AddAC mock_ac 'Mock AC'
  gdbus call --system -d org.freedesktop.UPower -o /org/freedesktop/UPower \
      -m org.freedesktop.DBus.Mock.AddChargingBattery mock_bat 'Mock Bat' 50.0 1200

In both cases upower --dump or gnome-power-statistics will show you the expected devices (of course you need to run that within the environment of the fake $DBUS_SYSTEM_BUS_ADDRESS, or run the mock on the real system bus as root).

Iftikhar Ahmad contributed a template for NetworkManager, which allows you to easily set up ethernet and wifi devices and wifi access points. See pydoc3 dbusmock.templates.networkmanager for details and the test cases for how this looks like in practice.

I just released python-dbusmock 0.2.1 and uploaded the new version to Debian experimental. I will sync it into Ubuntu Raring in a few hours.

Read more

Greetings from Copenhagen! I thought I would give a mid-UDS checkup for the quality team community. You may have already heard some of the exciting stuff that is already been discussed at UDS. Automated testing is being pursued with full vigor, the release schedule has been changed, and cadence testing is in. In addition, ubuntu is being focused into getting into fighting shape by targeting the Nexus 7 as a reference platform for mobile.

I was honored enough to have a quick plenary where attendees here got to see and hear about the various automated testing efforts going on. Does that mean the machines have replaced us? Hardly! The goal with bringing automated testing online is to help us be more proactive with how and why we test. We've done an amazing job of reacting to changes and bugs, but now as a community I would like us to focus on being proactive with our testing. The changes below are all going to help set us firmly in this direction. By proactively testing things, we eliminate bugs, and repetitive or duplicated work for ourselves. This frees us to explore more focused, more interesting, and more in-depth testing. So without further ado, here's a quick rundown of the changes discussed here in Copenhagen -- hang on to your testing hats!

Release
The Release schedule has dropped all alphas, and the first beta, resulting in a beta and then final release milestone only. In addition, the freezes have been moved back a few weeks. The end result is the archive will not be frozen till late in the cycle, allowing development and testing to continue unencumbered. This of course is for ubuntu only. Which brings us to flavors!


Flavors
Flavors will now have complete control over there releases. They can chose to test, freeze, and re-spin according to there own schedule and timing. Some will adopt ubuntu's schedule, others may retain the old milestones or even do something completely different.


ISOs
Iso's will now be automatically 'smoke' tested before general release. No more completely broken installers on the published images! In addition, the iso's will be published daily as usual, but will not have typical milestones as mentioned above. Preference will be given to the daily iso -- the current one -- throughout the cycle. Testing will occur in a cadence instead of a milestone.

Cadence
Rather than milestones, a bi-weekly cadence of testing will occur with the goal of assuring good quality throughout the release cycle. The cadence weeks will be scheduled and feature testing different pieces of ubuntu in a more focused manner. This includes things like unity, the installer, and new features landing in ubuntu, but will also be the target of feedback from the state of ubuntu quality.

State of ubuntu Quality
A bold effort to generate a high level view of what needs testing and what is working well on a per image basis inside of ubuntu. This is an experimental idea whose implementation will garner feedback early in the cycle and will collect data and influence decisions for testing focus during the cycle. *fingers crossed*

AutoPilot
This tool will integrate xpresser to allow for a complete functional UI testing tool. One of the first focuses for testcases will be automating the installer from a UI perspective to free our manual testing resources from basic installer testing! From the community perspective, we can join in both the writing, and executing of automated, as well as the development of the tool itself.

Hardware Testing Database
This continuing experiment will become more of a reality. The primary focus of the work this cycle will be to bring the tool, HEXR, online and to do basic integration with the qatracker for linking your hardware profiles. In addition, focused hardware testing using the profiles will be explored.

I hope this gives you a nice preview of what's coming. I would encourage you to have a look a the blueprints and pads for the sessions, and ask questions or volunteer to help in places you are interested. I am excited about the opportunities to continue bringing testing to the next level inside of ubuntu. I  owe many thanks to the wonderful community that continues to grow around testing. Here's to a wonderful cycle.

Read more

I trust everyone is readying themselves -- don't blink! Ubuntu UDS-R is already upon us. Those of you who have been watching closely may have heard about some of the planned sessions for QA, but if not feel free to take a look. Don't worry, I'll wait.

But wait, there's more! In addition, there is going to be an evening event where testing is the focus. It's happening Tuesday evening. The goal is to learn about some of the testing efforts going on inside ubuntu, including automated testing; and more importantly, to write some testcases! Folks will be on hand to help talk you through and discuss writing both automated and manual test cases.

Looking through the tsessions, I hope you have the sense that testing is continuing to play a large role in ubuntu. And further, that you can be even more invovled! UI testing, automated testing, testcase writing -- all of these are focus points this cycle and have sessions. Get involved -- and if your at UDS, please do come to a session or two, add your voice, and grab some work items :-) Let's make it happen for next cycle.

Read more

For writing tests for GVFS (current tests, proposed improvements) I want to run Samba as normal user, so that we can test gvfs’ smb backend without root privileges and thus can run them safely and conveniently in a “make check” environment for developers and in JHBuild for continuous integration testing. Before these tests could only run under gvfs-testbed, which needs root.

Unlike other servers such as ssh or ftp, this turned out surprisingly non-obvious and hard, so I want to document it in this blog post for posterity’s benefit.

Running the server

Running smbd itself is mainly an exercise of figuring out all the options that you need to set; Alex Larsson and I had some fun figuring out all the quirks and hiccups that happen between Ubuntu’s and Fedora’s packaging and 3.6 vs. 4.0, but finally arrived at something working.

First, you need to create an empty directory where smbd can put all its databases and state files in. For tests you would use mkdtemp(), but for easier reading I just assume mkdir /tmp/samba here.

The main knowledge is in the Samba configuration file, let’s call it /tmp/smb.conf:

[global]
workgroup = TESTGROUP
interfaces = lo 127.0.0.0/8
smb ports = 1445
log level = 2
map to guest = Bad User
passdb backend = smbpasswd
smb passwd file = /tmp/smbpasswd
lock directory = /tmp/samba
state directory = /tmp/samba
cache directory = /tmp/samba
pid directory = /tmp/samba
private dir = /tmp/samba
ncalrpc dir = /tmp/samba

[public]
path = /tmp/public
guest ok = yes

[private]
path = /tmp/private
read only = no

For running this as a normal user you need to set a port > 1024, so this uses 1445 to resemble the original (privileged) port 445. The map to guest line makes anonymous logins work on Fedora/Samba 4.0 (I’m not sure whether it’s a distribution or a version issue). Don’t ask about “dir” vs. “directory”, that’s an inconsistency in Samba; with above names it works on both 3.6 and 4.0.

We use the old “smbpasswd” backend as shipping large tdb files is usually too inconvenient and brittle for test suites. I created an smbpasswd file by running smbpasswd on a “real” Samba installation, and then using pdbedit to convert it to a smbpasswd file:

sudo smbpasswd -a martin
sudo pdbedit -i tdbsam:/var/lib/samba/passdb.tdb -e smbpasswd:/tmp/smbpasswd

The result for password “foo” is

myuser:0:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX:AC8E657F83DF82BEEA5D43BDAF7800CC:[U          ]:LCT-507C14C7:

which you are welcome to copy&paste (you can replace “myuser” with any valid user name, of course).

This also defines two shares, one public, one authenticated. You need to create the directories and populate them a bit:

mkdir /tmp/public /tmp/private
echo hello > /tmp/public/hello.txt
echo secret > /tmp/private/myfile.txt

Now you can run the server with

smbd -iFS -s /tmp/smb.conf

The main problem with this approach is that smbd exits (“Server exit (failed to receive smb request)”) after a client terminates, so you need to write your tests in a way to only run one connection/request per test, or to start smbd in a loop.

Running the client

If you merely use the smbclient command line tool, this is rather simple: It has a -p option for specifying the port:

$ smbclient -p 1445 //localhost/private
Enter martin's password: [enter "foo" here]
Domain=[TESTGROUP] OS=[Unix] Server=[Samba 3.6.6]
smb: \> dir
  .                                   D        0  Wed Oct 17 08:28:23 2012
  ..                                  D        0  Wed Oct 17 08:31:24 2012
  myfile.txt                                   7  Wed Oct 17 08:28:23 2012

In the case of gvfs it wasn’t so simple, however. Surprisingly, libsmbclient does not have an API to set the port, it always assumes 445. smbclient itself uses some internal “libcli” API which does have a way to change the port, but it’s not exposed through libsmbclient. However, Alex and I found some mailing list posts ([1], [2]) that mention $LIBSMB_PROG, and it’s also mentioned in smbclient’s manpage. It doesn’t quite work as advertised in the second ML post (you can’t set it to smbd, smbd apparently doesn’t speak the socket protocol over stdin/stdout), and it’s not being used anywhere in the current Samba sources, but what does work is to use good old netcat:

export LIBSMB_PROG="nc localhost 1445"

with that, you can use smbclient or any program using libsmbclient to talk to our test smb server running as user.

Read more

I found it surprisingly hard to determine in tearDown() whether or not the test that currently ran succeeded or not. I am writing some tests for gnome-settings-daemon and want to show the log output of the daemon if a test failed.

I now cobbled together the following hack, but I wonder if there’s a more elegant way? The interwebs don’t seem to have a good solution for this either.

    def tearDown(self):
        [...]
        # collect log, run() shows it on failures
        with open(self.daemon_log.name) as f:
            self.log_output = f.read()

    def run(self, result=None):
        '''Show log output on failed tests'''

        if result:
            orig_err_fail = result.errors + result.failures
        super().run(result)
        if result and result.errors + result.failures > orig_err_fail:
            print('\n----- daemon log -----\n%s\n------\n' % self.log_output)

Read more

I was working on writing tests for gnome-settings-daemon a week or so ago, and finally got blocked on being unable to set up upower/ConsoleKit/etc. the way I need them. Also, doing so needs root privileges, I don’t want my test suite to actually suspend my machine, and using the real service is generally not suitable for test suites that are supposed to run during “make check”, in jhbuild, and the like — these do not have the polkit privileges to do all that, and may not even have a system D-Bus running in the first place.

So I wrote a little test_upower.py helper, then realized that I need another one for systemd/ConsoleKit (for the “system idle” property), also looked at the mock polkit in udisks and finally sat down for two days to generalize this and do this properly.

The result is python-dbusmock, I just released the first tarball. With this you can easily create mock objects on D-Bus from any programming language with a D-Bus binding, or even from the shell.

The mock objects look like the real API (or at least the parts that you actually need), but they do not actually do anything (or only some action that you specify yourself). You can configure their state, behaviour and responses as you like in your test, without making any assumptions about the real system status.

When using a local system/session bus, you can do unit or integration testing without needing root privileges or disturbing a running system. The Python API offers some convenience functions like “start_session_bus()“ and “start_system_bus()“ for this, in a “DBusTestCase“ class (subclass of the standard “unittest.TestCase“).

Surprisingly I found very little precedence here. There is a Perl module, but that’s not particuarly helpful for test suites in C/Vala/Python. And there is Phil’s excellent Bendy Bus, but this has a different goal: If you want to thoroughly test a particular D-BUS service, such as ensuring that it does the right thing, doesn’t crash on bad input, etc., then Bendy Bus is for you (and python-dbusmock isn’t). However, it is too much overhead and rather inconvenient if you want to test a client-side program and just need a few system services around it which you want to set up in different states for each test.

You can use python-dbusmock with any programming language, as you can run the mocker as a normal program. The actual setup of the mock (adding objects, methods, properties, etc.) all happen via D-Bus methods on the “org.freedesktop.DBus.Mock“ interface. You just don’t have the convenience D-Bus launch API.

The simplest possible example is to create a mock upower with a single Suspend() method, which you can set up like this from Python:

import dbus
import dbusmock

class TestMyProgram(dbusmock.DBusTestCase):
[...]
    def setUp(self):
        self.p_mock = self.spawn_server('org.freedesktop.UPower',
                                        '/org/freedesktop/UPower',
                                        'org.freedesktop.UPower',
                                        system_bus=True,
                                        stdout=subprocess.PIPE)

        # Get a proxy for the UPower object's Mock interface
        self.dbus_upower_mock = dbus.Interface(self.dbus_con.get_object(
            'org.freedesktop.UPower', '/org/freedesktop/UPower'),
            'org.freedesktop.DBus.Mock')

        self.dbus_upower_mock.AddMethod('', 'Suspend', '', '', '')

[...]

    def test_suspend_on_idle(self):
        # run your program in a way that should trigger one suspend call

        # now check the log that we got one Suspend() call
        self.assertRegex(self.p_mock.stdout.readline(), b'^[0-9.]+ Suspend$')

This doesn’t depend on Python, you can just as well run the mocker like this:

python3 -m dbusmock org.freedesktop.UPower /org/freedesktop/UPower org.freedesktop.UPower

and then set up the mocks through D-Bus like

gdbus call --system -d org.freedesktop.UPower -o /org/freedesktop/UPower \
      -m org.freedesktop.DBus.Mock.AddMethod '' Suspend '' '' ''

If you use it with Python, you get access to the dbusmock.DBusTestCase class which provides some convenience functions to set up and tear down local private session and system buses. If you use it from another language, you have to call dbus-launch yourself.

Please see the README for some more details, pointers to documentation and examples.

Update: You can now install this via pip from PyPI or from the daily builds PPA.

Update 2: Adjusted blog entry for version 0.0.3 API, to avoid spreading now false information too far.

Read more

As you may have read about this week here or here unity is getting some late additions to it's feature set, with much of it landing in the form of new lenses. Here's what's landed in Unity 6.6:

• Improved Window and workspace management
• When switching between multiple application windows, the windows scale
• The window decoration properly follows the selected theme
• All launcher icons can be reordered
• New shopping lens
• Video lens enhancements
• Improved lens previews
• Improved home dash

In support of this late landing code, the Unity development team is asking for some extra testing on these specific features. To that end, I've updated the testcases for Unity, and posted the testcases specific to these changes for testing. Pay attention to the cases marked 'mandatory' and 'run-once'. These testcases contain the changes mentioned above, but it doesn't hurt to run through the optional cases if you have time. We don't like regressions either :-) By the time this post hits you, unity 6.6 should be landing in quantal, and you'll simply need to do a dist-upgrade -- instructions here. Here's a link to the testcases themselves:

Unity 6.6 Testing

Before I forget, here's a link to help you navigate the tracker, if you've never used it before: Guide on using the qatracker.

I'll keep the call for testing open through beta2 next week. If you participate in testing the beta2 images, consider also running through these testcases to make sure Unity is in good shape as well.

Thank you in advance for your help, and happy testing everyone!

Read more

We've all had this experience. Upgrade our hardware to the new version of ubuntu (or perhaps installing ubuntu for the first time on some new hardware), and everything works perfectly. We quickly dive in and enjoy the new experience of our favorite OS. Well, it's almost perfect. There's just this one bug that is bothering you.

Perhaps instead you are a tester. Someone who lives and breathes for breakage and the latest and greatest raw code packaged from the hands of it's developers. There's a new upgrade out and everything is perfect; well, except for that one bug.

So what do you do in these situations? Sadly some people may chose to do nothing but hope and wait. "Next cycle, or next release I'm hoping my bug will be fixed". Odds are, no one knows about your bug, and it can't be fixed if it's unknown. And if you are using proprietary software, "wait and see" is about the limit of your options. As a part of ubuntu however, we as a community can do so much more!

With that let me present my patented wizzbang approach to a successful resolution of your bug within ubuntu!

  
First, let me clarify what a successful resolution is. If you have been around ubuntu, you may have seen a bug that expired as 'incomplete'. This is clearly not a successful resolution. In my eyes, a successful resolution to a bug sees the status changed to a 'won't fix, fix committed, triaged, etc'.

Ok, so here's the steps:

1. If you don’t know how to file a good bug, ask first! It's important to do your best to describe the problem you are experiencing, and if possible how to repeat the bug. Check out the post on askubuntu which has a nice summary of resources available to help you.
2. File a good bug, using your newly formed knowledge from above :-)
3. Get someone else to confirm it. This is important! If possible, have a friend confirm the bug on their system. Once they've confirmed it, have them mark the bug as affecting them as well on launchpad.
4. Answer questions promptly when asked by others. Make sure you are getting emails from launchpad, and when someone asks a question on your bug, respond promptly.
5. Get your bug triaged. If your bug is confirmed and filed correctly, the bug triagers should help triage the bug. If a long time has passed without this occurring, check to make sure you bug is in good order to be triaged. If so, asking politely for a triager to look at your bug on the #ubuntu-bugs channel is a good way to keep your bug moving.
6. Help debug, test, and confirm and fixes that are made for your issue. If the developer spends time working on your bug, do what you can to help confirm it fixes your issue.
7. Remember no one will care about your bug as much as you do! When you file a bug, commit to carrying it as far along in the process as you can.

That's it! There's no guarantee every bug you file now will receive your desired outcome, but you should see proper resolution, instead of your bugs expiring. By being a good participant you are ensuring you can feel good about the resolution of the bugs you file. Remember we are all human, and sometimes things get missed. Stick with your bug, and shepherd it through.

So is the process perfect? Not at all. We as a community still need to think more about improving our experience in dealing with problems. Not every "problem" encountered is a bug, and a process to better handle these problems is still worthy of thought. I invite those of you interested in this to look for a UDS session on the topic.

Special thanks to TheLordofTime and hggdh for their discussions surrounding bugs, and of course for our marvelous bugsquad without whom this would not be possible!

Read more

Global Jam is a unique time for folks to get together across the world and celebrate ubuntu. As part of the celebrations, there is an opportunity to download the latest beta (released yesterday!) and check out the next version of ubuntu. You can run it in a livecd or perhaps in a virtual machine. Ether way, there's opportunities for you to test the common applications and report your results.

The testcases are available here: Ubuntu Global Jam Testcases. For some of you, this page may seem a bit funny, but fortunately there is a handy walk-through for you to understand how to use the tracker and specifically, how to report results against these tests. Check out the guides below. If you follow the link, you'll even find some video of yours truly giving you a visual demonstration.

https://wiki.ubuntu.com/Testing/QATracker
https://wiki.ubuntu.com/Testing/CallforTesting/Walkthrough

If you get stuck, remember your friends in #ubuntu-testing on freenode are also happy to help. Have fun jamming, and if you do test, Happy Testing! Most of all, celebrate ubuntu!

Read more

The unstoppable PostgreSQL team just announced the first release candidate of 9.2, with several bug fixes since the Beta 4. If you haven’t tested 9.2 yet, now is the time! Remember that you can run a copy of your 8.4 or 9.2 cluster in parallel for testing with pg_upgradecluster.

If you use Debian, 9.2rc1 will be available in experimental in a few hours. For Ubuntu, you can get packages for all supported releases from my PostgreSQL backports PPA as usual.

Enjoy!

Read more

Over the past several months I've been occasionally back-porting a bunch of eCryptfs patches onto older Ubuntu releases.  Each back-ported fix needs to be properly sanity checked and so I've been writing test cases for each one and adding them to the eCryptfs test suite.

To get hold of the test suite, check it out using bzr:

 bzr checkout lp:ecryptfs  

and install the dependencies so one can build the test suite:

 sudo apt-get install debhelper autotools-dev autoconf automake \
 intltool libtool libgcrypt11-dev libglib2.0-dev libkeyutils-dev \
 libnss3-dev libpam0g-dev pkg-config python-dev swig acl \
 ecryptfs-utils  

If you want to test eCrytpfs with xfs and btrfs as the lower file system onto which eCryptfs is mounted, then one needs to also install the tools for these:

 sudo apt-get install xfsprogs btrfs-tools  

And then build the test programs:

 cd ecryptfs  
 autoreconf -ivf  
 intltoolize -c -f  
 ./configure --enable-tests --disable-pywrap  
 make  

To run the tests, one needs to create lower and upper mount points. The tests allow one to create ext2, ext3, ext4, xfs or btrfs loop-back mounted file systems on the lower mount point, and then eCryptfs is mounted on the upper mount point on top.   To create these, use something like:

 sudo mkdir /lower /upper  

The loop-back file system image needs to be placed somewhere too, I generally place mine in a directory /tmp/image, so this needs creating too:

 mkdir /tmp/image  

There are two categories of tests, "safe" and "destructive".  Safe tests should run in such a ways as to not lock up the machine.  Destructive tests try hard to force bugs that can cause kernel oopses or panics. One specifies the test category with the -c option.  Now to run the tests, use:

 sudo ./tests/run_tests.sh -K -c safe -b 1000000 -D /tmp/image -l /lower -u /upper  

The -K option tells the test suite to run the kernel specific tests. These are the ones I am generally interested in since I'm testing kernel patches.

The -b option specifies the size in 1K blocks of the loop-back mounted /lower file system size.  I generally use 1000000 blocks as a minimum.

The -D option specifies the path where the temporary loop-back mounted image is kept and the -l and -u options specified the paths of the lower and upper mount points.

By default the tests will use an ext4 lower filesystem. One can also run specify which file systems to run the tests on using the -f option, this can be a comma separated list of one or more file systems, for example:

 sudo ./tests/run_tests.sh -K -c safe -b 1000000 -D /tmp/image -l /lower -u /upper \
 -f ext2,ext3,ext4,xfs  

And also, instead of running a bunch of tests, one can just a particular test using the -t option:

 sudo ./tests/run_tests.sh -K -c safe -b 1000000 -D /tmp/image -l /lower -u /upper \
 -f ext2,ext3,ext4,xfs -t lp-926292.sh

..which tests the fix for LaunchPad bug 926292
 
We also run these tests regularly on new kernel images to ensure we don't introduce and regressions.   As it stands, I'm currently adding in tests for each bug fix that we back-port and for most new bugs that require a thorough test. I hope to expand the breadth of the tests to ensure we get better general test coverage.

And finally, thanks to Tyler Hicks for writing the test framework and for his valuable help in describing how to construct a bunch of these tests.

Read more

It's the testing event I've been waiting for! A new version unity has arrived, stock full of compiz fixes. The team has removed metacity completely, and also migrated to gsettings from gconf. The result is removal of unity2d, and the enablement of llvmpipe on unity for those running without hardware acceleration.

Now, of course this work, in particular the settings migration, needs to be tested. This is where all of you come in! The package tracker now contains an entry for unity, complete with testcases for this migration. For those who helped test last cycle, you will also notice all of the checkbox testcases have been ported over as well! Many thanks to the unity developers for there help in migrating these tests. For this call for testing, the 'Unity GSetting Migration' testcases have been marked as mandatory, meaning that testcase is the primary focus. However, if you are able, executing the other testcases also helps the unity team ensure there haven't been any regressions.

Please note the 'Unity GSetting Migration' has steps for you to complete BEFORE you install from the ppa. Please read it first before diving in. Here's a link to the testing on the tracker. And if you are new, check out our wiki for a guide on using the qatracker to help test.

Now, as an added bonus, the unity developers have also tagged several multi-monitor bugs and have asked users to go through the list and confirm any bugs that they can. Read the bug report, and if you have a multi-monitor unity setup, see if it's still affecting you. Leave a comment on the bug with your result. The unity team wants to make sure the bugs are able to be triaged and get proper attention if they are still valid.

List of unity bugs, tagged multi-monitor

Thank you in advance for your help, and happy testing everyone!

Read more

It all started innocently enough. A simple idea, turned into a simple post on the mailing list. This idea eventually led the ubuntu QA community to perform an experiment for this cycle, which has been dubbed "cadence testing".

Now, before I manage to confuse everyone with this "cadence testing" term, let's define cadence testing. Scratch that, let's just give a simple definition of what was intended by original idea. If you want the whole story read the thread. heh. I'll be waiting (hint it's LONG!).

Cadence testing was intended to introduce regularity into testing. If the development release could be "stable" everyday (which was the grand experiment during the precise cycle), could we not also test to ensure that things were good all throughout the release? If the everyday images and archive were now to the quality of previous release's milestones, could we just eliminate the milestone idea and go with a calendar schedule for testing? Thus, a proposal was made test every 2 weeks, whether or not a milestone had been planned, and report the results.

Fast forward 2 months to today. So what happened? Well, I'm happy to report that the QA community despite the confusion more or less met the goal of testing the desktop images every 2 weeks (milestone or not). But what did this achieve? And where are the results?

Let's step back a moment and talk about what we learned by doing this. My comments are specific to the non-milestone cadence testing weeks. First, the development process inside ubuntu is still built around milestones. The daily images during cadence testing weeks were sometimes stable, and sometimes flat out broken by a new change landing from a development team. Second, the tools we used are built around milestone testing as well. The qatracker as well as any qa dashboard or report available doesn't have a good way to track and image health across the cadence week. This meant it was both difficult to test and difficult to see the results of the testing. Finally, the development teams were not expecting results against the daily images, and couldn't follow-up well on any bugs we reported, nor where we able to coordinate well with the release team, as the bugs reported were not available in a summarized or meaningful way.

Now, I'll save the discussion on my ideas of a healthy QA workflow for a later post, but I think we can agree that testing without good result reporting, and without developer follow-up has a limited impact. So does this mean "cadence testing" was a bad idea? No, it was simply poorly executed. The trouble comes in the assumptions listed above.

The archive has not been "stable" everyday, and development teams have have continued development, pausing only as required by the current milestones. In addition, changes, even major ones (like the ubiquity changes landing a few weeks ago, or the nvidia change just this past weekend), are not well communicated. Since they land without little or no warning, we as a QA community are left to react to them, instead of planning and executing them. In this environment, cadence testing makes little sense.

So was the experiment a failure then? In my mind, not at all! In fact, I think the future of ubuntu and QA is to push for complete adoption of this idea, and this experiment confirms the obstacles we will face in getting there. I'll be posting more about what this vision for QA looks like, but I'll leave you with a few thoughts until then.

In my mind, QA should enhance and improve developers, testers, and users lives and workflows. Our work is critical to the success of ubuntu. I would like to see a future where users receive regular, timely scheduled updates that the folks in the QA community have vetted by working with the development and release teams to deliver focused quality updates. The ideal workflow is more fluid, more agile and yes, it has a cadence.

Read more

Yesterday I was playing around with determining webcam resolution by reverse engineering how fswebcam worked, using the ioctl VIDIOC_TRY_FMT. I keep working that evening on my code and was thinking of filing a bug against v4l2-ctl or the driver until I made a discovery: having fswebcam take a picture at a specified resolution actually changes what v4l2-ctl returns, here’s an example that’s been truncated to keep it short:

mfisch@caprica:/tmp/luvcview-0.2.6$ v4l2-ctl -V
Width/Height : 1280/720

mfisch@caprica:/tmp/luvcview-0.2.6$ fswebcam -d /dev/video0 -r640x480 /tmp/foo.jpg
Writing JPEG image to '/tmp/foo.jpg'.

mfisch@caprica:/tmp/luvcview-0.2.6$ v4l2-ctl -V
Width/Height : 640/480

Hey look, v4l2-ctl changed it’s answer! So is this a bug? I dug into the v4l2 docs and it turns out that you can make it change it’s answer again by picking a different resolution with fswebcam. This has to do with how fswebcam works.

fswebcam first calls VIDIOC_TRY_FMT and that ioctl returns and sets some parameters on the image. For example, if you request an image that’s 10000×10000 it will return back a success, but set the width and height to something like 1280×720 (depending on your hardware).

Next fswebcam calls VIDIOC_S_FMT. Unlike VIDIOC_TRY_FMT, S_FMT actually sets what the driver’s format is. So, if you take a picture at 640×480, the driver will keep that internal format as default until you call it again with a larger value. This is all described in the v4l2 spec.

So, I was wrong, the driver is fine and is in fact behaving as the spec requires. So given this information, I figured that to find the true max, I could just start with a huge value, like 10000×10000 and see what value TRY_FMT came back with. That’s where I left it last night, until a colleague, Josh Poulson, pointed me to luvcview.

luvcview can list all the supported formats and even the time intervals between frames, it prints a long list, so here’s a truncated version:

Device information:
Device path: /dev/video0
{ pixelformat = 'YUYV', description = 'YUV 4:2:2 (YUYV)' }
{ discrete: width = 640, height = 480 }
Time interval between frame: 1/30, 1/20, 1/15, 1/10, 1/5,
{ discrete: width = 640, height = 400 }
Time interval between frame: 1/30, 1/20, 1/15, 1/10, 1/5,
{ discrete: width = 352, height = 288 }
Time interval between frame: 1/30, 1/20, 1/15, 1/10, 1/5,
....


And that’s what I needed. It turns out that there’s an experimental ioctl called VIDIOC_ENUM_FRAMESIZES that luvcview is using. This information highlights the good and bad parts of the web. It was easy for me to take existing tools and dive into the problem I was trying to solve with virtually no knowledge of v4l2, but I made some bad assumptions. But it’s also thanks to the web, and my partially incorrect blog post that I know so much more now.

Read more

After you read this, please read my follow-up post here.

Update the bottom

I spent some time today trying to determine the answer to a seemingly simple question, “What resolution is my web cam?” There are a myriad of tools that display webcam info including v4l-info, v4l2-ctl, xawtv, fswebcam, etc. v4l-info and v4l2-ctl both claimed 1280×720, but I had some issues with the images before, so I wasn’t sure I trusted those values. So I decided to use fswebcam to take some stills. I was able to take 640×480 pictures with fswebcam just fine, so I tried 800×600, fswebcam balked at this value:


mfisch@caprica:~$ fswebcam -d /dev/video0 -r800x600 /tmp/foo.jpg

--- Opening /dev/video0...
Trying source module v4l2...
/dev/video0 opened.
No input was specified, using the first.
Adjusting resolution from 800x600 to 640x480.
--- Capturing frame...
Captured frame in 0.00 seconds.
--- Processing captured image...
Writing JPEG image to '/tmp/foo.jpg'.


Note that fswebcam adjusted the resolution down to 640×480. Maybe that’s my max resolution? That doesn’t match what the other tools report so I tried again with 1280×720:


mfisch@caprica:~$ fswebcam -d /dev/video0 -r1280x720 /tmp/foo.jpg
--- Opening /dev/video0...
Trying source module v4l2...
/dev/video0 opened.
No input was specified, using the first.
--- Capturing frame...
Captured frame in 0.00 seconds.
--- Processing captured image...
Writing JPEG image to '/tmp/foo.jpg'.


Note that it didn’t adjust the resolution this time. Let’s go for the limit, maybe it can do 10000p!


Writing JPEG image to '/tmp/foo.jpg'.
mfisch@caprica:~$ fswebcam -d /dev/video0 -r10000x10000 /tmp/foo.jpg
--- Opening /dev/video0...
Trying source module v4l2...
/dev/video0 opened.
No input was specified, using the first.
Adjusting resolution from 10000x10000 to 1280x720.
--- Capturing frame...
Captured frame in 0.00 seconds.
--- Processing captured image...
Writing JPEG image to '/tmp/foo.jpg'.


Nope, sadly it adjusted down again to 1280×720, so no 10000p for me. I dug into the code inside fswebcam to see how this works and it all boils down to an ioctl() called VIDIOC_TRY_FMT. What this ioctl does essentially is say “hey, can you give me this resolution and this format”? If it returns a non-negative that means “yes, I support that format”, and the driver modifies the fmt structure to report back what resolution it can support, generally (perhaps always?) adjusting the size downwards. You can see this being used in the file src_v4l2.c file in the fswebcam package. So when I requested a 10000×10000 image captured in MJPEG (the default) it downsized my request to the nearest supported size, 1280×720.

Note: VIDIOC_TRY_FMT is explained in detail in this lwn article.

1280×720 matches what v4l2-ctl -V showed me here:

mfisch@caprica:~$ v4l2-ctl -V
Format Video Capture:
Width/Height : 1280/720
Pixel Format : 'MJPG'
Field : None
Bytes per Line: 0
Size Image : 1843789
Colorspace : SRGB

What is more interesting is that I had a co-worker run this same set of tests and his results didn’t match. v4l2-ctl claimed his max was 640×480, but fswebcam’s calls to VIDIOC_TRY_FMT would allow 1280×1024, so his driver was caught in a lie!

My co-worker uses a System76 Lemur Ultra which has a 1.3MP camera. 1280×1024 = 1.3MB, so I think that it is taking pictures that size and not upscaling them (which was my earlier thought).

Given this info, I’m still planning on using v4l2-ctl -V, but I think if you want to be sure you should call VIDIOC_TRY_FMT and using fswebcam is a simple way to do so.

Update:

It seems that sometimes my driver is wrong as well. After taking a bunch of still images tonight during testing, look what mine now reports:

mfisch@caprica:~/qa/checkbox/scripts$ v4l2-ctl -V
Format Video Capture:
Width/Height : 640/480
Pixel Format : 'MJPG'
Field : None
Bytes per Line: 0
Size Image : 614400
Colorspace : SRGB

There is some kernel bug at work here, if I figure out more, I’ll file a bug and post an update here.

Read more

About 2.5 months ago I wrote about the plans for the ubuntu QA community for the quantal cycle. We were building off of lots of buzz from the precise release and we planned to undertake lots of new work, while being very careful to avoid burnout. Our focus was to take QA to the next level and help us communicate and grow as a team to take on the opportunities we have.

So, how are we doing? Let's go over each of the points noted in the original post and talk about the progress and plans.

ISOTesting
Our alpha1 testing went very well, but the alpha2 and alpha3 have seen less participation. In addition we were asked and responded to a plan to test our iso's every 2 weeks as part of a more cadenced testing. Overall, isotesting continues to be a weak spot for us as a community. ISO Testing is perhaps the most important piece of testing for us as a greater ubuntu community. The image we produce is literally the first experience many folks have with ubuntu. If it fails to install, well, there went our chance for a positive first impression :-( I would be happy to hear ideas or comments on isotesting in particular.

Application Testing
This work has been mostly completed. The package tracker now allows us to perform work that was done via checkbox or manual testing last cycle. We can now manage results, tests and reporting all in one tool -- and it's all publicly available. For more information about the qatracker, see this wiki page.

SRU Verification
This work is still on paper, awaiting for the 12.04.1 release before further discussions and work will begin.

General Testing (eg, Day to Day running of the development version)
I am still experimenting with understanding how to enable better reporting and more focused testing on this. The current plan is to track specific packages that are critical to the desktop, and allow those run the development version the ability to report how the application is working for each specific upload during the development release. This is done with the qatracker. I'll blog more about this and the results in a later post. Contact me as always if your interested in helping.

Calls for Testing
This has been a wonderful success. There have been several calls for testing and the response has been wonderful. A big thank you to all of you who have helped test this. We've had over 50 people invovled in testing, and 41 bugs reported. Myself and the development teams thank you! But we're not done yet, unity testing among other things are still coming!

QATracker development
There is still room for more developers on the qatracker project. It's written in drupal, and I would happy to help you get started. As we grow, there will continue to be a need for people who want to build awesome tools to help us as a community test. If you have ideas for a tool (or know of a tool) that would help us test, please feel free to share with me.

Hardware Database
Work has been completed to spec out the design, and is scheduled now to land this cycle not in a future cycle. Fingers crossed we'll sneak this in before we release quantal :-) I'm very excitied to share this new tool with you; as soon as it's complete we'll be able to incorporate it into our workflow on the qatracker.

Testcases
Done, and for the most part our testcases have been migrated over. In addition, there is now a special team of folks who help to manage and maintain our testcases. If you have a passion for this work, contact me and I can help get you involved with the team.

Overall, I am happy to see signs of growth and newcomers to the community. If your on the fence about getting more involved with ubuntu, I would encourage you to check out QA. We collaborate with almost every area of ubuntu in some way, and no two days are the same :-) Send an email to the ubuntu-qa mailing list and introduce yourself.

So what's your opinion? Feel free to respond here with your thoughts and/or fill out the quality survey to give feedback.

Read more

What's your perception of quality this cycle? Are things working well for you? It's been several months now since precise landed, and ubuntu development for the next version has been ongoing. The ubuntu QA team has had a busy summer putting into place the new tools we spoke about at UDS. The qatracker has been revamped to allow us to consolidate our testcases and test reporting across all of our activities. In addition, we've been helping in the release of 3 alpha milestones, and 3 testing campaigns. To all those who have helped in this testing, a very big thank you!

I have my own thoughts about the impact to the ubuntu project this testing has had, and I will continue to share my thoughts to point out the progress we make in this regard. But now, I want your input. I have created a survey to understand the community perspective on how we as a ubuntu project are doing on quality. If you have a few moments, please fill out the survey and let your thoughts and perspective be known. The survey will be anonymous, but I will share an aggregation and summary of the results.

My hope is to help gain an understanding of how we can focus our efforts on what's important to ubuntu as a project in terms of quality, as well as how we can help you (yes you!) become a more active part of QA if your interested.

Here's a link to survey. I'll leave it open until next Friday August 10th. Thanks in advance for your participation.

Read more

I've been using QEMU and KVM for quite a while now for general kernel testing, for example, sanity checking eCryptfs and Ceph.   It can be argued that the best kind of testing is performed on real hardware, however, there are times when it is much more convenient (and faster) to exercise kernel fixes on a virtual machine.

I used to use the command line incantations to run QEMU and KVM, but recently I've moved over to using virt-manager because it so much simpler to use and caters for most of my configuration needs.

Virt-manager provides a very usable GUI and allows one to create, manage, clone and destroy virtual machine instances with ease.

virt-manager view of virtual machines

Configuring a virtual machine

virt-manager viewing a 64 bit Natty server (for eCryptfs testing)

Read more

Anyone who has kids knows that three-year olds are starting to learn that the world is full of complex rules. Don’t go on red. Look before you cross the street. You can’t just watch TV all day. What every three year old wants to know is “why?”. I answer “why?” to my kid about twenty times a day. I’ve found that from all the teams I’ve worked on, coders are the same. Tell a coder that they need to fill out a new bug report form or write unit tests and they will ask “why”. Now, like a three year old you can threaten a coder (usually with a bad performance review rather than a timeout), you can bribe a coder (maybe a bonus or award instead of candy, or maybe just candy), or you can convince that it’s in their best interest to it. Like three year olds, it’s best to convince coders that doing something is in their best interest rather than threatening or bribing.

When it’s time to discuss unit testing, what most managers do is bring out some slides about quality and fill them with buzzwords about “quality metrics” or even worse, they bring up some industry standard that promises lots of paper work like CMM or ISO9000. I’ve got to be honest, while almost every coder I know cares about the product and the customer experience, most I know could care less about something like CMM or ISO9000.

So what is the right way to make coders care about unit tests? From my personal experience, unit tests give me four key benefits:

1. Prove that the API is not broken
2. Increase speed of execution
3. Catch issues before they get shipped
4. Allow me to really understand an API/project

Prove That the API is Not Broken

I really like writing unit tests that test an external API. I can use my tests to ensure that the API is staying stable. This is important not just to catch places where someone adds an argument to a function, but when someone makes a change that modifies the behavior of the API. Once an API ships, people start relying on the behavior of it, whether it was specified or not (wrongly or not). For example, some accomplishments-daemon code will throw a KeyError if you pass in an invalid key to a dictionary. So regardless as to whether this was specified in the documentation, someone may have code that relies on that, and so I test it. (I know that relying on implicit, undocumented behavior of an API is bad, but almost everyone does it.) If the code is changed next week to catch the KeyError instead and return None, the unit tests will at least tell us and we can decide whether at the very least the API version needs to roll.

Increase Speed of Execution

This is my favorite reason for writing unit tests. Well written tests let you isolate testing of one function all by itself and make it easier to debug an algorithm without all the setup and teardown. In a previous job I was writing software to get show listings from a TiVo and download shows. I could not make much progress unless I was in my office on the same subnet with my three TiVo devices. Also these devices were sometimes taken by the test group or marketing for various reasons, delaying my work. So what I ended up doing was writing unit tests and making a fake TiVo device, a mock, that behaved exactly like a real device (from an API stand-point). This not only allowed me to work anywhere, it let me skip the tedious setup, device discovery, and remote debugging required (my code ran on a device other than my dev box). I’d estimate that setting up each debug run would have taken me 5 minutes with a real h/w and it was more like seconds with unit tests. Over the course of three months, this saved hundreds of hours of my time.

When my first release of the TiVo code wrapped up, I was in-between assignments, so I wrote a “season subscription” feature. This feature would periodically scan the device for new shows and download them if they were part of a season that you wanted. By this time, the test team and marketing had taken my TV and all my TiVOs. I ended up spending 3 days writing all the code and only validating with new unit tests. I then just emailed the new installer to the test group and everything worked fine on the hardware.

Catch Issues Before They Get Shipped

This is the one part of unit testing that you will likely hear from a manager armed with a “quality” slide. It is in fact true. Bugs are cheaper to fix earlier, customers are happier, and better yet, bugs or quirks in your API don’t become an implicit part of the API.

Allow Me to Really Understand an API/Project

If you are new to a project and really want to understand how the API works, there’s no better way than writing unit tests. You can really learn the API, the “quirks” of the code, and help the project out at the same time.

Broken Unit Tests

Unit tests are only useful if test failures are taken seriously, and they’re only taken seriously if the build (hopefully continuous) breaks when they fail. Never underestimate the ability of people to forget (intentionally or otherwise) to run unit tests or to ignore failures. The tests I worked on for the Ubuntu Accomplishments Daemon had been broken for months and were so outdated they had to be scrapped completely. The new tests will run during every build which is run at least daily. If the tests fail, the deb is not built.

Ubuntu Accomplishments Daemon Unit Tests

I’ve spend the last week or so writing unit tests for the Ubuntu Accomplishments Daemon. You can see the pile of tests here in the tests.py file. While writing the tests we found about 5 bugs in the accomplishments-daemon, most were small fixes, but most of them also caused the API to abort, which would kill the daemon in a real environment. Making the daemon more resilient will make Ubuntu Accomplishments much friendlier to end-users.

There are still plenty of APIs that are not yet tested, so if you’d like to dive in, please pull the code and look at the to do list at the top of tests.py.

Conclusion

There are entire books written about unit testing, about how to do it, and about why. However, if you’re like me, you’ll need to see the benefits first-hand to really want to write unit-tests. So go read the books, and then give it a try, I promise it will reduce your bug count and improve your speed of execution.

Read more

Last month, an interesting thread emerged on ubuntu-devel. A proposal to change the way we as ubuntu look at testing and quality. In many ways it was more of a codification of ideas and thoughts from the precise cycle than a proposal. 

One of the outcomes of this was a change to how to test isos. Rather than focus on arbitrary moments in time, we've been asked to stick to a two week cadence for testing. What that means is a regular checkup of our images every two weeks. Quite a task, but not impossible! Given the fact the change happened mid-cycle, there has been some confusion over what exactly this means. I decided to put together a post detailing exactly what's on the table for us as a community and more importantly how you can help!


If you have a look at this wiki page, I've listed the images we test and produce for ubuntu. So far during the quantal cycle we have achieved 100% coverage for most of these iso's for our mandatory testcases. That's thanks to the wonderful efforts of folks like you testing isos! But in some cases, like our first non-milestone cadence last week, the coverage was provided by a single person -- meaning we have only 1 confirmation of success or failure. I'd like us as a community to take this to the next level. I'm asking for you to commit to an iso over the course of the cycle. Would you be willing to commit to running through the mandatory testcases every 2 weeks for the iso and making sure it's in good shape? If so signup on that wiki page, underneath the iso in question. Don't be afraid if this sort of testing sounds scary. I and the rest of the community are happy to help you through your first testcase. As part of iso testing, I'm still growing my knowledge of linux and ubuntu and interacting with wonderful and talented people while doing it. This is a natural expansion of the 'adopt an iso campaign' with a new cadence. Powerpc, and mac users, this is your chance to make a difference as your hardware is less common and therefore harder to ensure proper testing for.


During the precise cycle over 100 people submitted a result to the isotracker for an iso they downloaded and tested as part of daily testing. That's excellent work, and I thank all of you very much! Many more of you downloaded and installed iso's throughout the cycle, but perhaps didn't report your work. I would encourage you to get involved and help share your results with others. If we have 100 people signup in support of iso testing, the workload required of each individual will be quite small. Yet the benefits for us as a community will be huge. More hardware and more testing results in more bugs caught sooner. We all want a good upgrade experience in October. This is your chance to be a part of making sure it happens.


As a small addendum, I'd like to point out the results of the work this testing achieves. If you have a look on this page, you'll notice a very long list of bugs; many of which are rated as high or critical in launchpad. All of these bugs were found during iso testing -- a testament to those who have tested before us. We all thank you.

Read more