Last week, at UDS, I was asked several questions about how Ubuntu Certification worked. I decided I would start a set of blog posts about several aspects of the certification programme, for people willing to know a bit more about how we test and what.

One of the most confusing bits about Ubuntu Certification are BIOS options. In several ocassions I was asked the questions whether UEFI (with and without Secure Boot) was going to be added as part of Ubuntu Certification or not.

The answer is both yes and no. I will explain it a bit more in detail.

In Ubuntu Certification we don’t test BIOS options, nor we change them. We try to install Ubuntu with what it comes with the system from factory, and it needs to work up to a certification level using those options. That means that if it is using Legacy BIOS, we will be testing Legacy BIOS, if it comes with UEFI without Secure Boot, that is what we will be testing. If it comes with Secure Boot enabled, Ubuntu will need to install out of the box with the keys already in that system.

From 12.10 we have started to parse the BIOS information as part of the testing process:

We will be updating the public certification site to show that information there as well.

Same thing for BIOS options. We test with factory default options, and only those. The goal of doing this is that Ubuntu Certified systems should provide and out-of-the-box Ubuntu experience. In some cases, tweaking BIOS options can improve the situation for non working systems, but that is something we don’t want to acept for Ubuntu Certification.

During the Precise cycle we have been working in building a new UI for System Testing, the desktop application to test your hardware with Ubuntu and report results to Launchpad and Ubuntu Friendly.

If you are running Precise, you can test the new UI searching the Dash for “System Testing”. The new UI is now feature complete and you can start using it to test your hardware and submit your results to Friendly.

Here you can find the differences between the old UI and the new one:

Old UI

New UI

I think it looks much better, doesn’t it? The most important thing is that it builds the grounds to be able to keep making improvements to the UI, which were blocked in many cases by the old one.

Now we need to a lot of testing to make sure it is fully working when 12.04 LTS gets released. So, if you are running Precise, I encourage you to give it a try and report as many bugs as possible.

To file bugs, run the following command in a terminal:

ubuntu-bug checkbox

Last week, on the same day Ubuntu 11.10 got released, we launched the Beta of Ubuntu Friendly website. This is a huge milestone for us, as we will be able to collect a lot of data coming from Ubuntu users in order to fix bugs and tweak the scoring system so it is better once Ubuntu Friendly finishes its beta period (once 12.04 LTS gets released).

In the first few days we got the following numbers:

• More than 350 submissions
• More than 100 submissions with more than 3 stars
• 4 submissions with 5 stars
• 7 models with more than 1 rater

I think this is a great achievement for the first week of a beta release. Let’s hope we get many more results, feedback and bugs as possible, so Ubuntu Friendly rocks when 12.04 LTS is released!

The Ubuntu Hardware Certification team is growing!

I am happy to announce that the hardware certification team is growing and we are now looking for a Certification Engineer to work in our lab in Lexington (MA).

The job is great if you like to develop automated testcases, but you also want to get your hands dirty with hardware configuration, networking, etc. You will be able to help a team that helps Ubuntu to work better in a wide range of hardware, helping developers debugging weird kernel bugs. This is also the team behind Ubuntu Friendly, so you will be able to help this great project as well!

So, if you live around the Boston area and you think your the person for the job, please, apply in the Canonical job site. Looking forward to reading your resumes!

As weeks go by, we are getting closer and closer to Ubuntu 11.10 release and, therefore, to Ubuntu Friendly beta release.

First of all, the website is taking shape nicely and you can already see (and use!) an alpha version of the final site.

As you can see from the screenshot, right now all systems have only 1 star. This is the minimum rate a system can get (we give 1 start because, at least, it was able to install Ubuntu on it and submit results). The reason for all the submissions so far to have only 1 star is that a bug in Checkbox was preventing for one test considered to be “core” to run properly.

Call for Testing

The mentioned bug in Checkbox is fixed in our PPA and it is now ready to be used in Oneiric. If you run Oneiric it will be really helpful if you could install Checkbox from our PPA and run the full Ubuntu Friendly suite (this is the suite that runs by default).

• Add our PPA to your software sources
• Install the latest version of checkbox
• Run “System Testing” on your system and submit to Launchpad

This will help us on two sides:

• First, you will run the latest code in trunk. We are planning to release version 0.12.8 to Oneiric in a week, so this is the last time to fix critical issues. Should you find any issues during your testing, please, file a bug against Checkbox project.
• On the server side of things, having a lot of data will help us testing the website to make sure it works fine when 11.10 gets released. Should you find any issues on the website, please, file a bug against the Ubuntu Friendly project.

Thanks for the help and I hope you are as excited as I am about the project!

Now we have a proper “How-to participate in Ubuntu Friendly” including a screencast on how to submit your system. We are just waiting to have a nice domain and Ubuntu 11.10 released to launch officially Ubuntu Friendly Beta!

Wednesday came and went! I will summarize our progresses as usual.

Ubuntu Friendly website

The website is coming together very nicely. Marc, who is working on the backend, worked closely with Mike, who is working on the frontend and they were able to communicate both, send a submission and show that information on the website. The website frontend still needs some polishing, but the basics are already there and working. One of the goals of the sprint was to have a full working solution, although not feature complete. I think we are going to make it.

Apart from this real progress we also worked a bit on new wireframes. Yesterday we got a comment on a previous post about having a mobile version of the site, as people will use their phones while shopping for a new system, to check how well they are reported to work with Ubuntu. We thought that it was a very good point, so we worked on some wireframes for the mobile version of the site:

Of course, we will focus on the desktop version first. But we will try to develop a mobile version based on these mock ups after that.

Checkbox

On the checkbox side we are almost done for what we expect to land in Oneiric. There is just one thing left: we are about to measure how long it takes to run the full suite for Ubuntu Friendly and we will remove the less useful tests to try to cut the running time to around 15 minutes.

Once we finish with that we will upload the new version to the archive.

Yesterday was a great day in the UF sprint and we got a lot of stuff done. I will summarize the interesting bits.

Ubuntu Friendly Website

As I said on the previous post, we still needed a way to visualize what components are working and which are not (to a certain level). To achieve that we have created a new “details” page, once you click on “more” on the configuration page.

That page will give you two pieces of information. First, it will show the complete list of components that were reported for that configuration:

On that list you will see the core components on the top (the ones we use to distinguish a configuration from another one) and a list of the rest of components found in all the units that were reported.

The second piece of information are the results that were reported per component:

On this view you will see the number of people that reported that a particular component was working and a percentage. We will have some colour code for the percentage, to have a quick visual view of it.

The “Related Bugs” tab is just a way of saying that this view could be extended in the future, but currently this is out of the scope for this release.

Checkbox

On the client side we also made some important progress. We finished mapping tests to components, so we can know which components are failing and which ones are passing.

We just merged our changes into checkbox trunk and we hope to have it uploaded to Oneiric early next week.

This week, myself and other 4 members of the Ubuntu Friendly Squad are working together, face to face, to try to have a working prototype of the Ubuntu Friendly programme as a whole. I will be blogging about it during the week to share the progress with the rest of the team.

Ubuntu Friendly Website

The main focus for Monday was to get an idea of how the Ubuntu Friendly site would look like. We think that the most difficult thing to solve is having right the idea of different configurations. As a lot of people pointed out while we were presenting the programme, a model name can contain many different hardware, and work much better or worse depending on those components.

We want to find a compromise between having a UI that’s easy to understand, and having it as complete as possible. For the Ubuntu homepage this is what we came up with:

Every different configuration will be in a different line, with some details of the components that are part of it. If the user wants to know more details about a particular component, hovering over it will do the trick.

You will be able to filter by release, rating or number of results to get you to the results that you are looking for. Once one of these systems is clicked on, the details page will appear that will look like this:

For that particular configuration we will be showing the full details of the hardware, the number of results and their rating it got in the different releases. Also, a box will show similar systems. For this cycle the similar systems will be other configurations of the same model.

There is still one thing that we need to work on for the details page. We need a way to say if a particular component is working or not. But hey! we still have the rest of the week to figure it out.

Checkbox

Another focus for today was to have a complete whitelist that include all the components that we have agreed as part of the UF scope. We added tests for components that were on the list, and removed unrelevant tests.

We are putting our progress in a public branch at Launchpad (https://code.launchpad.net/~ubuntu-friendly-control/checkbox/uf_sprint/). We are hoping to merge to trunk and to have a new version in Oneiric by Thursday, but in the mean time you can check the new whitelist using directly that branch.

While working on the whitelist we discovered a very nasty bug in Checkbox. If you deselect some tests when starting running checkbox, start testing, then close checkbox, and start it again, the whitelist won’t be used anymore, your previous selection will be used instead. This bug affects the Ubuntu Friendly experience, so we will have it fixed before release.

There seems to be a little bit of confusion about what the Ubuntu Friendly programme is and what it is not. People tend to think that Ubuntu Friendly is a certification programme and, because of this, they obviously don’t understand the idea behind core and extra components.

Let’s try to clarify things a bit with a small FAQ:

Is Ubuntu Friendly a certification programme?

No. Ubuntu Friendly is not a certification programme. UF is a community driven hardware validation programme. Its goal is to have a list of systems that people have tested with a particular release of Ubuntu and an associated rating, based on the results of their testing.

Let’s say is the next generation HardwareSupport wiki pages (https://wiki.ubuntu.com/HardwareSupport/Machines/Laptops), but with a better structure, as the tests will be written upfront, and the process to get a rating for a system will be the same for every system.

What does it mean that a system is Ubuntu Friendly?

Nothing, as there is not such concept. Ubuntu Friendly is just the name of the programme and yes, it was chosen because of the concept of a system being Ubuntu friendly (working fine with Ubuntu), but there won’t be a classification of systems that gained the Ubuntu Friendly status or not.

Instead of that, any system tested with the testing tools that we provide will make it to the Ubuntu Friendly list, with a rating associated with it. The rating will be between 0 and 5, that will give the user of the Ubuntu Friendly list an overview of how well a system works with Ubuntu and how many people have tested it.

Will this substitute the Ubuntu Certification Programme?

No. Ubuntu Friendly is not a certification programme and it won’t substitute the current Ubuntu Certification Programme. The UCP is a commercial certification programme, run by Canonical, and it will be still available for commercial partners. Opposite to the Ubuntu Friendly programme, the UCP is a go/no-go decision. A system is certified with Ubuntu or it is not, there are no ratings. To be certified with Ubuntu the system must pass all the tests specified in the UCP coverage list.

Yesterday we had our first meeting to start nailing down the Ubuntu Friendly programme. It was a great meeting, with lots of participation from both Canonical and non Canonical people.

Meeting logs and summary are available at the wiki.

One of the things that was agreed was the final separation of core and extra components.  The final list looks like (the tick represents a core component):

This list is the final list for Ubuntu 11.10 and was mainly based on a survey we conducted within the Ubuntu Friendly community.

We will improve coverage and might modify core components for future releases based on the feedback we get in the first Ubuntu Friendly release.

This year, the Desktop Summit happens in Berlin and although I am not that much involved with the desktop that I was previously it is a great opportunity as it happens locally

I will be attending Saturday, Sunday and, thanks to my employer, Canonical, Monday as well, using one of the conference day paid leave that we have as part of the employee benefits. I will also be attending the parties, of course!

Really looking forward to this weekend and to seeing again some friends who are coming to the summit as well.

Picture by theodevil

The Free Software community is a great community, but a bit narrow. Most of the people are geeks and there are many more men than women. Disclaimer: I would consider myself a geek. It is true that the older I get the less geek I am, but during my childhood, my early youth and my University years I was a true geek. And that was almost yesterday…

Artists is a very influential collective, they are one of the best PR any product can get. They influence people, the media and even governments. We need writers writing novels using Ubuntu, musicians making music using Ubuntu and video artists editing their films using Ubuntu.

Designers are also influential. Let’s face it, blogs by designers talk about more beautiful stuff that those of hackers. If you read Planet Ubuntu, a lot of the information there comes in the form of code. On a designer blog you’d rarely see the word branch if they are not talking about trees. Why Ruby on Rails got so much attention? Was because it was the best piece of free software ever? Probably not, but it attracted designers. Designers and developers starting working together using the same tools and they spread the word.

One of the examples I always use for the FLOSS community being too geek is PyRoom. PyRoom is a simple but great distractions-free writing application that, obviously, is coded in Python. I love the application and use it almost on a daily basis, but using the name of the programming language as part of your application is a clear indication that the technology you’re using is as important as the features your coding with it.

I am not saying that we need less geeks in our community, I am just saying that if we want to reach the 200M users goal, we need to have a wider community. And that’s why I backed Novacut on their Kickstarter project. Because they are focusing in talking to film makers as part of their design process. Because they have one in the team. Because I think they could help Ubuntu getting a wider and very much needed community. And they can fail, as anyone can fail, but I think it is worth supporting those brave enough to try. I am looking forward to seeing what Novacut brings.

As a side benefit I think that having a less geeky community will bring more women into free software. I’ve always defended that one of the reasons of the low ratio of women to men in FLOSS communities was mainly due to the geekyness of these communities. For social reasons (or a combination of social and biological ones), the proportion between men and women among geeks is far from the fifty percent. I am not trying to explain why: I am not a sociologist, nor a psychologist, but I think most people would agree that that’s the case. Bring more non-geeks to Ubuntu and women will be among them.

Last week, and thanks to the efforts of Brendan (among others), a lot of tests that used to be private were moved to Checkbox, the tool that we are going to use to test Ubuntu for the Ubuntu Friendly project, and are now public.

This means that a lot of new strings are ready to be translated:

We want Ubuntu Friendly to be as open as possible and that involves having the tests translated to as many languages as possible. We are aiming to collect as many results as possible, so allowing people who don’t speak English to test their systems and provide results is a very desirable feature.

If you speak a non-English language (or want tests to be in perfect Queen’s English), come to our translations page and submit your translation suggestions. Thanks!

Picture by madame_furie

One of the decisions that we took when defining the Ubuntu Friendly process was that we would distinguish between a core component and an extra component. A submission received with a failing core component would get a rating of zero (who wants to buy a bike with broken pedals?), and a submission with all its core components working would get a rating of 3 out of 5. The extra 2 points will be gained with working extra components.

But the question is, which components of a system should be considered core components?

This is a tricky question, as it will depend a lot of the uses a person gives to a system. But, in the end, we all have to take decisions, so we can make things happen. Obviously, core components in desktops won’t be the same ones as in laptops/netbooks, as their uses are completely different.

We have also to take into account that a system with all its core components will only get a 3 out of 5 from those core components, so it is totally OK to leave outside of the core components an important one, as it will only mean that if it fails, the system won’t get a 5 out of 5. For example, for a laptop I would consider wireless a core component. I just can’t imagine using a laptop (if it comes with a wireless card) if the wireless network is not working. But I can imagine using  a laptop where the wired ethernet card is not working. I would consider the wired network an extra component for a laptop. Obviously, the perfect laptop will have both working, that’s why we would need both working to be able to reach a rating of 5 out of 5.

So, what are your ideas? What components are vital for your laptop/netbook? What components in your desktop you couldn’t live without? Comments welcome!

Picture by stefanvds

Last week, current members of the Ubuntu Friendly Control gather together and we took the opportunity to discuss some of the Ubuntu Friendly processes. One of the biggest questions that we wanted to answer was what it meant for a system to be Ubuntu Friendly. How and where should we draw the line between an Ubuntu Friendly system and the rest of systems. After a very good discussion and brainstorming we came up with a different solution: we wouldn’t draw that line.

The basic idea from where we will define the way a system is tested and have an Ubuntu Friendly rating is the division between core components and extra components. What is a core component or an extra component we care about will be decided later in the process.

Testing

Ubuntu Friendly tests will be grouped by the component that they test. In order for a submission to be consider valid and accepted in our rating system, all the tests that cover the core components should have been tested (either pass or fail, but tested).

If a submission does not contain results for ALL tests that cover, the system will reject that submission and it won’t count for Ubuntu Friendly status.

Once that the core components have been covered, the submission will be accepted.

Ratings

Rating per submission

The rating for a particular accepted submission (at least all tests for core components) will be determined by the number of components that passed or failed. The rating will be between 0 and 5.

If any of the tests that cover the core components failed, the submission will get a value of 0. Even if any of the extra components worked.

If the all the tests that cover the core components passed, the submission gets a rating of 3.

The two remaining points to get to a rating of 5 are covered by the extra components with a simple rule of three. If a system does not have any extra components, and, again, all tests covering core components passed, the submission will receive a rating of 5.

Examples:

We have a list of 10 core components (UFC) and 7 extra components (UFE) that we care about. Any other component we will list it as additional component (AC).

Global rating per system

The global rating for a particular system will be the average of all the submission for that particular system.

Ubuntu Friendly Website

Each release of Ubuntu will get its own list. By default, the list shown will be for the latest release of Ubuntu.

Systems will be ordered by global rating first, and number of submission after.

Systems with very bad rating will also appear in the list. There will be no boarder line to call a system Friendly or not, it will be just a rating systems

Example:

****************************************************
Ubuntu 11.04 Friendly

Latitude 2120             4.3   (30 submissions)
Vostro                    4.3   (13 submissions)
Acer EeePC                3.2   (40 submissions)
Mac Book Pro              2.9   (3  submissions)
WinPro Laptop             0.3   (50 submissions)
****************************************************

The representation of the ratings will be graphical (stars, bars, etc). Clicking on any of the results will give us the details of each of the submissions.

People will be able to filter by model, type of system, minimum rating, etc.

Giving Feedback on Results

Any user will be able to give feedback and comments for a particular system without needing to run the tests. The feedback will be things like “bluetooth is supposed to be working, but it is not working for me”. These comments will be showed on the details of a particular system, but they won’t affect the actual rating of the system.

Waiting on your comments!

As in previous processes and thoughts about Ubuntu Friendly, we are looking to get as much feedback as possible, and this part of Ubuntu Friendly is no different. Please, join us in the Ubuntu Friendly Squad and have your say!

Click here to view the video on YouTube.

A nice talk about how Google tests. I would love to see something like that in Ubuntu.

Watch this video on YouTube.

As you might already know, we will do some changes to Ubuntu Certification in the Oneiric Release. Victor Palau has written a nice summary of some of those changes:

[...] we are planning to close down the “Ubuntu Ready” programme in time for Oneiric Final Release.

The aim is to simplify the public Canonical endorsed certification programme to only one:“Ubuntu Certified“.

To straighten out any confusion about what our certification offering will be here is a quick fact sheet about certification:

Existing Ubuntu Ready Certificates
We will not be offering new “Ubuntu Ready” services to OEM/ODMs. The existing Ubuntu Ready certificates will be maintained on the public website until the applicable releases reach end of life.

Remote Testing
We will continue to offer testing tools to partners and the community.The objective is for a common test tool for partners and community will be available within the Ubuntu ISO (from Oneiric).

Ubuntu Certified for Clients
Ubuntu Certified will continue to require hardware to be submitted to Canonical for testing. Ubuntu fortnightly Stable Release Updates means that certified systems are required to be tested every 2 weeks to ensure no regressions are introduced.
Remote testing can be used by partners as a way to assess if certification will be successful before engaging in a contract with Canonical.

Ubuntu Certified for Servers
While Certification of single servers follows the same process than client certification, we are concentrating our efforts on Certifying full server lines from OEMs.
In order to achieve this objective, the full server line is analysed by the appropriate TAM, a component matrix is produced and small representative set of servers is provided to Canonical by the OEM to test in-house, while the rest of servers are test remotely.

Ubuntu Certified (Pre-install Only)
A OEM or ODM shipping a pre-install custom ISO with their systems can apply for Ubuntu Certified (Pre-install Only).

Ubuntu Friendly
Ubuntu Friendly in not a Canonical certification programme. Ubuntu Friendly is a Ubuntu community hardware validation programme that recognises the need for community and partners to list various degrees of working Ubuntu hardware publicly. At the same time, we expect this initiative to increase the visibility of which hardware components work with Ubuntu.
Participation in Ubuntu Friendly is free, done in the open and will utilise the remote testing tools provide by Platform Services.

 

One of the things that we will need to discuss and decide to run the Ubuntu Friendly programme is what makes a particular system Ubuntu Friendly. Obviously, one of the ways to do this is to make the Ubuntu Friendly Control review the submissions that are considered complete and correct and determine whether those results are enough to consider the system Ubuntu Friendly.

This, obviously, scale badly if the number of submissions grow to a number that make the revisions of submissions the bottleneck of the programme.

The other option, that I think we should explore is create a way to auto regulate results, partly manually and partly automatically. Think askubuntu.com. On that site, people asking questions can accept or reject answers from people, answers can be voted up or down, etc. The same way, I foresee an Ubuntu Friendly site where people from the Ubuntu Friendly Squad could:

• Reject results that are incomplete or invalid
• Vote up results that are complete
• Ask for more information to the tester, if some discrepancies are found
• etc.

With this information, an algorithm could then decide whether a particular system is Ubuntu Friendly or not. The details of that decision would always be public. For example:

Of course, at any point, members of the Ubuntu Friendly Control could override the algorithm decision, and remove a system from the list of Ubuntu Friendly if necessary.

Some weeks ago we announced the Ubuntu Friendly programme, a new community driven HW validation programme. We had at UDS a couple of sessions about it and we were very happy to see that a lot of people were interested in participating.

I am happy to announce today that we have created the Ubuntu Friendly Squad team to start participating in the programme.  This programme is on its very early stages and almost everything needs to be implemented and organized. People joining the team at this stage are people willing to work in shaping the idea behind the programme, its tools, its governance, etc. If you prefer to wait until the programme is more organized, that’s totally fine: Ubuntu Friendly Squad will always be open to join, we just decided to create the team a bit earlier, as there were many people at UDS willing to participate. The team has a mailing list to start discussing what needs to be done to kick it off, so make sure to subscribe and start the discussions. I have also created a project in Launchpad, to be able to use it to organize our tasks and tools.

One of our first tasks will be discussing how people in the Ubuntu Friendly Squad can be part of Ubuntu Friendly Control, the administrator of the programme, and propose that to the Community Council.