Canonical Voices

Posts tagged with 'testing'

Matt Fischer

Yesterday I was playing around with determining webcam resolution by reverse engineering how fswebcam worked, using the ioctl VIDIOC_TRY_FMT. I keep working that evening on my code and was thinking of filing a bug against v4l2-ctl or the driver until I made a discovery: having fswebcam take a picture at a specified resolution actually changes what v4l2-ctl returns, here’s an example that’s been truncated to keep it short:

mfisch@caprica:/tmp/luvcview-0.2.6$ v4l2-ctl -V
Width/Height : 1280/720

mfisch@caprica:/tmp/luvcview-0.2.6$ fswebcam -d /dev/video0 -r640x480 /tmp/foo.jpg
Writing JPEG image to '/tmp/foo.jpg'.

mfisch@caprica:/tmp/luvcview-0.2.6$ v4l2-ctl -V
Width/Height : 640/480

Hey look, v4l2-ctl changed it’s answer! So is this a bug? I dug into the v4l2 docs and it turns out that you can make it change it’s answer again by picking a different resolution with fswebcam. This has to do with how fswebcam works.

fswebcam first calls VIDIOC_TRY_FMT and that ioctl returns and sets some parameters on the image. For example, if you request an image that’s 10000×10000 it will return back a success, but set the width and height to something like 1280×720 (depending on your hardware).

Next fswebcam calls VIDIOC_S_FMT. Unlike VIDIOC_TRY_FMT, S_FMT actually sets what the driver’s format is. So, if you take a picture at 640×480, the driver will keep that internal format as default until you call it again with a larger value. This is all described in the v4l2 spec.

So, I was wrong, the driver is fine and is in fact behaving as the spec requires. So given this information, I figured that to find the true max, I could just start with a huge value, like 10000×10000 and see what value TRY_FMT came back with. That’s where I left it last night, until a colleague, Josh Poulson, pointed me to luvcview.

luvcview can list all the supported formats and even the time intervals between frames, it prints a long list, so here’s a truncated version:

Device information:
Device path: /dev/video0
{ pixelformat = 'YUYV', description = 'YUV 4:2:2 (YUYV)' }
{ discrete: width = 640, height = 480 }
Time interval between frame: 1/30, 1/20, 1/15, 1/10, 1/5,
{ discrete: width = 640, height = 400 }
Time interval between frame: 1/30, 1/20, 1/15, 1/10, 1/5,
{ discrete: width = 352, height = 288 }
Time interval between frame: 1/30, 1/20, 1/15, 1/10, 1/5,
....

And that’s what I needed. It turns out that there’s an experimental ioctl called VIDIOC_ENUM_FRAMESIZES that luvcview is using. This information highlights the good and bad parts of the web. It was easy for me to take existing tools and dive into the problem I was trying to solve with virtually no knowledge of v4l2, but I made some bad assumptions. But it’s also thanks to the web, and my partially incorrect blog post that I know so much more now.

Read more
Matt Fischer

After you read this, please read my follow-up post here.

Update the bottom

I spent some time today trying to determine the answer to a seemingly simple question, “What resolution is my web cam?” There are a myriad of tools that display webcam info including v4l-info, v4l2-ctl, xawtv, fswebcam, etc. v4l-info and v4l2-ctl both claimed 1280×720, but I had some issues with the images before, so I wasn’t sure I trusted those values. So I decided to use fswebcam to take some stills. I was able to take 640×480 pictures with fswebcam just fine, so I tried 800×600, fswebcam balked at this value:


mfisch@caprica:~$ fswebcam -d /dev/video0 -r800x600 /tmp/foo.jpg

--- Opening /dev/video0...
Trying source module v4l2...
/dev/video0 opened.
No input was specified, using the first.
Adjusting resolution from 800x600 to 640x480.
--- Capturing frame...
Captured frame in 0.00 seconds.
--- Processing captured image...
Writing JPEG image to '/tmp/foo.jpg'.

Note that fswebcam adjusted the resolution down to 640×480. Maybe that’s my max resolution? That doesn’t match what the other tools report so I tried again with 1280×720:


mfisch@caprica:~$ fswebcam -d /dev/video0 -r1280x720 /tmp/foo.jpg
--- Opening /dev/video0...
Trying source module v4l2...
/dev/video0 opened.
No input was specified, using the first.
--- Capturing frame...
Captured frame in 0.00 seconds.
--- Processing captured image...
Writing JPEG image to '/tmp/foo.jpg'.

Note that it didn’t adjust the resolution this time. Let’s go for the limit, maybe it can do 10000p!


Writing JPEG image to '/tmp/foo.jpg'.
mfisch@caprica:~$ fswebcam -d /dev/video0 -r10000x10000 /tmp/foo.jpg
--- Opening /dev/video0...
Trying source module v4l2...
/dev/video0 opened.
No input was specified, using the first.
Adjusting resolution from 10000x10000 to 1280x720.
--- Capturing frame...
Captured frame in 0.00 seconds.
--- Processing captured image...
Writing JPEG image to '/tmp/foo.jpg'.

Nope, sadly it adjusted down again to 1280×720, so no 10000p for me. I dug into the code inside fswebcam to see how this works and it all boils down to an ioctl() called VIDIOC_TRY_FMT. What this ioctl does essentially is say “hey, can you give me this resolution and this format”? If it returns a non-negative that means “yes, I support that format”, and the driver modifies the fmt structure to report back what resolution it can support, generally (perhaps always?) adjusting the size downwards. You can see this being used in the file src_v4l2.c file in the fswebcam package. So when I requested a 10000×10000 image captured in MJPEG (the default) it downsized my request to the nearest supported size, 1280×720.

Note: VIDIOC_TRY_FMT is explained in detail in this lwn article.

1280×720 matches what v4l2-ctl -V showed me here:

mfisch@caprica:~$ v4l2-ctl -V
Format Video Capture:
Width/Height : 1280/720
Pixel Format : 'MJPG'
Field : None
Bytes per Line: 0
Size Image : 1843789
Colorspace : SRGB

What is more interesting is that I had a co-worker run this same set of tests and his results didn’t match. v4l2-ctl claimed his max was 640×480, but fswebcam’s calls to VIDIOC_TRY_FMT would allow 1280×1024, so his driver was caught in a lie!

My co-worker uses a System76 Lemur Ultra which has a 1.3MP camera. 1280×1024 = 1.3MB, so I think that it is taking pictures that size and not upscaling them (which was my earlier thought).

Given this info, I’m still planning on using v4l2-ctl -V, but I think if you want to be sure you should call VIDIOC_TRY_FMT and using fswebcam is a simple way to do so.

Update:

It seems that sometimes my driver is wrong as well. After taking a bunch of still images tonight during testing, look what mine now reports:

mfisch@caprica:~/qa/checkbox/scripts$ v4l2-ctl -V
Format Video Capture:
Width/Height : 640/480
Pixel Format : 'MJPG'
Field : None
Bytes per Line: 0
Size Image : 614400
Colorspace : SRGB

There is some kernel bug at work here, if I figure out more, I’ll file a bug and post an update here.

Read more
Matt Fischer

Anyone who has kids knows that three-year olds are starting to learn that the world is full of complex rules. Don’t go on red. Look before you cross the street. You can’t just watch TV all day. What every three year old wants to know is “why?”. I answer “why?” to my kid about twenty times a day. I’ve found that from all the teams I’ve worked on, coders are the same. Tell a coder that they need to fill out a new bug report form or write unit tests and they will ask “why”. Now, like a three year old you can threaten a coder (usually with a bad performance review rather than a timeout), you can bribe a coder (maybe a bonus or award instead of candy, or maybe just candy), or you can convince that it’s in their best interest to it. Like three year olds, it’s best to convince coders that doing something is in their best interest rather than threatening or bribing.

When it’s time to discuss unit testing, what most managers do is bring out some slides about quality and fill them with buzzwords about “quality metrics” or even worse, they bring up some industry standard that promises lots of paper work like CMM or ISO9000. I’ve got to be honest, while almost every coder I know cares about the product and the customer experience, most I know could care less about something like CMM or ISO9000.

So what is the right way to make coders care about unit tests? From my personal experience, unit tests give me four key benefits:

  1. Prove that the API is not broken
  2. Increase speed of execution
  3. Catch issues before they get shipped
  4. Allow me to really understand an API/project

Prove That the API is Not Broken

I really like writing unit tests that test an external API. I can use my tests to ensure that the API is staying stable. This is important not just to catch places where someone adds an argument to a function, but when someone makes a change that modifies the behavior of the API. Once an API ships, people start relying on the behavior of it, whether it was specified or not (wrongly or not). For example, some accomplishments-daemon code will throw a KeyError if you pass in an invalid key to a dictionary. So regardless as to whether this was specified in the documentation, someone may have code that relies on that, and so I test it. (I know that relying on implicit, undocumented behavior of an API is bad, but almost everyone does it.) If the code is changed next week to catch the KeyError instead and return None, the unit tests will at least tell us and we can decide whether at the very least the API version needs to roll.

Increase Speed of Execution

This is my favorite reason for writing unit tests. Well written tests let you isolate testing of one function all by itself and make it easier to debug an algorithm without all the setup and teardown. In a previous job I was writing software to get show listings from a TiVo and download shows. I could not make much progress unless I was in my office on the same subnet with my three TiVo devices. Also these devices were sometimes taken by the test group or marketing for various reasons, delaying my work. So what I ended up doing was writing unit tests and making a fake TiVo device, a mock, that behaved exactly like a real device (from an API stand-point). This not only allowed me to work anywhere, it let me skip the tedious setup, device discovery, and remote debugging required (my code ran on a device other than my dev box). I’d estimate that setting up each debug run would have taken me 5 minutes with a real h/w and it was more like seconds with unit tests. Over the course of three months, this saved hundreds of hours of my time.

When my first release of the TiVo code wrapped up, I was in-between assignments, so I wrote a “season subscription” feature. This feature would periodically scan the device for new shows and download them if they were part of a season that you wanted. By this time, the test team and marketing had taken my TV and all my TiVOs. I ended up spending 3 days writing all the code and only validating with new unit tests. I then just emailed the new installer to the test group and everything worked fine on the hardware.

Catch Issues Before They Get Shipped

This is the one part of unit testing that you will likely hear from a manager armed with a “quality” slide. It is in fact true. Bugs are cheaper to fix earlier, customers are happier, and better yet, bugs or quirks in your API don’t become an implicit part of the API.

Allow Me to Really Understand an API/Project

If you are new to a project and really want to understand how the API works, there’s no better way than writing unit tests. You can really learn the API, the “quirks” of the code, and help the project out at the same time.

Broken Unit Tests

Unit tests are only useful if test failures are taken seriously, and they’re only taken seriously if the build (hopefully continuous) breaks when they fail. Never underestimate the ability of people to forget (intentionally or otherwise) to run unit tests or to ignore failures. The tests I worked on for the Ubuntu Accomplishments Daemon had been broken for months and were so outdated they had to be scrapped completely. The new tests will run during every build which is run at least daily. If the tests fail, the deb is not built.

Ubuntu Accomplishments Daemon Unit Tests

I’ve spend the last week or so writing unit tests for the Ubuntu Accomplishments Daemon. You can see the pile of tests here in the tests.py file. While writing the tests we found about 5 bugs in the accomplishments-daemon, most were small fixes, but most of them also caused the API to abort, which would kill the daemon in a real environment. Making the daemon more resilient will make Ubuntu Accomplishments much friendlier to end-users.

There are still plenty of APIs that are not yet tested, so if you’d like to dive in, please pull the code and look at the to do list at the top of tests.py.

Conclusion

There are entire books written about unit testing, about how to do it, and about why. However, if you’re like me, you’ll need to see the benefits first-hand to really want to write unit-tests. So go read the books, and then give it a try, I promise it will reduce your bug count and improve your speed of execution.

Read more