Canonical Voices

Posts tagged with 'research'

Max

Impulse

Bug hunting on iOS 6.3.1

Me and my SO recently got an old iPad2, which we were planning to use for everyday usage around the house. Meaning, having it ready to research a recipe, get information about a topic or researching things to do together.

Unfortunately though, the iOS version that it came preinstalled with was 9.3.5. On this later version of iOS there is quite a problem with the responsiveness though.
Probably due to the big alterations on the design side the iPad feels slow and has noticeable lags when switching between applications, and even when using the web browser. The argument for this was, that it never seemed that slow before, and it should not be the hardware that wore down.
So I decided to try and downgrade the iPad and compare a much older version, namely 6.3.1, to the 9.3.5 that was installed.

After doing so successfully via ITunes and an old [IPSW](https://en.wikipedia.org/wiki/IPSW) file it was clear that the assumption was correct. Being back on the old version improved the performance significantly.
This only goes as far as the web though. When visiting certain websites errors started bringing the user satisfaction back down again.

Here are a a few screenshots of some broken styling:

And some websites were still a pleasure to use, here are the BBC and RA as examples:

Reflection

This made me think to go onto my own blog and start breaking it.
Given all the modern tech that I use for this theme, I would not be surprised to find some thing not working correctly. Mostly due to the CSS Grid specification I am using for the styling.

And indeed it did not take long:
First of all the header is broken, with the navigation links appearing in a horizontal manner, instead of a vertical one.
Additionally after some scrolling the header starts covering most of the content.

Bug hunting on iOS 6.3.1
Broken Header
Bug hunting on iOS 6.3.1
Header covering content

My speculation for the first bug being the CSS Grid, while the latter one could be related to the vh and vw units I am using.

Action

Since I am now in possession of a great testing device I set out to fix this.
Setting up my development environment by changing the the address of my development server to 0.0.0.0:8000, thus exposing it my local network, and accessing it on my iPad via IP.OF.MY.LAPTOP:8000, I have a quick feedback loop to test out my ideas and refactorings.

After looking at the stylings for the header the position: fixed; top: 0; turned out to be the problem.
Since this was a problem that was not solved elegantly initially I decided to throw it out completely and instead reiterate on the website design.

Since design in general is an iterative process I will take this as a major stepping stone in finding the right one for my blog.
For now the user (so you) will have to scroll back up to the top of the page in order to navigate. Given all the advanced technology that is on this site though, and its relative simplicity I will start from scratch for the layouting. This time using old methods and then adding modern techniques on top via cascading rules.

Resume

This rather random chain on events has again taught me, and hopefully some of you that are reading this, a lesson. While I always speak up for accessibility and its importantance in our work, I wanted to write the theme for this blog, using all the latest technologies, making my developer life comfortable and the code elegant.
This hypocrisy was made obvious right now, and I will have to backtrack and rework some parts. It also comes to show that some problems stem from architectural decisions rather than a specific bug. My assumptions about what could be breaking were not too far off, but as it turns out the problem has more depth.
While the CSS is easy to refactor and well decoupled, this could have become a nightmare if the project was bigger, since I have already accumulated tech-debt and would just keep on growing it.
But as always, a mistake made now is a mistake avoided in the future.

Cover photo by Tim Goedhart on Unsplash

Read more
Max

Impulse

Bug hunting on iOS 6.3.1

Me and my SO recently got an old iPad2, which we were planning to use for everyday usage around the house. Meaning, having it ready to research a recipe, get information about a topic or researching things to do together.

Unfortunately though, the iOS version that it came preinstalled with was 9.3.5. On this later version of iOS there is quite a problem with the responsiveness though.
Probably due to the big alterations on the design side the iPad feels slow and has noticeable lags when switching between applications, and even when using the web browser. The argument for this was, that it never seemed that slow before, and it should not be the hardware that wore down.
So I decided to try and downgrade the iPad and compare a much older version, namely 6.3.1, to the 9.3.5 that was installed.

After doing so successfully via ITunes and an old [IPSW](https://en.wikipedia.org/wiki/IPSW) file it was clear that the assumption was correct. Being back on the old version improved the performance significantly.
This only goes as far as the web though. When visiting certain websites errors started bringing the user satisfaction back down again.

Here are a a few screenshots of some broken styling:

And some websites were still a pleasure to use, here are the BBC and RA as examples:

Reflection

This made me think to go onto my own blog and start breaking it.
Given all the modern tech that I use for this theme, I would not be surprised to find some thing not working correctly. Mostly due to the CSS Grid specification I am using for the styling.

And indeed it did not take long:
First of all the header is broken, with the navigation links appearing in a horizontal manner, instead of a vertical one.
Additionally after some scrolling the header starts covering most of the content.

Bug hunting on iOS 6.3.1
Broken Header
Bug hunting on iOS 6.3.1
Header covering content

My speculation for the first bug being the CSS Grid, while the latter one could be related to the vh and vw units I am using.

Action

Since I am now in possession of a great testing device I set out to fix this.
Setting up my development environment by changing the the address of my development server to 0.0.0.0:8000, thus exposing it my local network, and accessing it on my iPad via IP.OF.MY.LAPTOP:8000, I have a quick feedback loop to test out my ideas and refactorings.

After looking at the stylings for the header the position: fixed; top: 0; turned out to be the problem.
Since this was a problem that was not solved elegantly initially I decided to throw it out completely and instead reiterate on the website design.

Since design in general is an iterative process I will take this as a major stepping stone in finding the right one for my blog.
For now the user (so you) will have to scroll back up to the top of the page in order to navigate. Given all the advanced technology that is on this site though, and its relative simplicity I will start from scratch for the layouting. This time using old methods and then adding modern techniques on top via cascading rules.

Resume

This rather random chain on events has again taught me, and hopefully some of you that are reading this, a lesson. While I always speak up for accessibility and its importantance in our work, I wanted to write the theme for this blog, using all the latest technologies, making my developer life comfortable and the code elegant.
This hypocrisy was made obvious right now, and I will have to backtrack and rework some parts. It also comes to show that some problems stem from architectural decisions rather than a specific bug. My assumptions about what could be breaking were not too far off, but as it turns out the problem has more depth.
While the CSS is easy to refactor and well decoupled, this could have become a nightmare if the project was bigger, since I have already accumulated tech-debt and would just keep on growing it.
But as always, a mistake made now is a mistake avoided in the future.

Cover photo by Tim Goedhart on Unsplash

Read more
Robin Winslow

I’ve been thinking about the usability of command-line terminals a lot recently.

Command-line interfaces remain mystifying to many people. Usability hobbyists seem as inclined to ask why the terminal exists, as how to optimise it. I’ve also had it suggested to me that the discipline of User Experience (UX) has little to offer the Command-Line Interface (CLI), because the habits of terminal users are too inherent or instinctive to be defined and optimised by usability experts.

As an experienced terminal user with a keen interest in usability, I disagree that usability has little to offer the CLI experience. I believe that the experience can be improved through the application of usability principles just as much as for more graphical domains.

Steps to learn a new CLI tool

To help demystify the command-line experience, I’m going to lay out some of the patterns of thinking and behaviour that define my use of the CLI.

New CLI tools I’ve learned recently include snap, kubectl and nghttp2, and I’ve also dabbled in writing command-line tools myself.

Below I’ll map out an example of the steps I might go through when discovering a new command-line tool, as a basis for exploring how these tools could be optimised for CLI users.

  1. Install the tool
    • First, I might try apt install {tool} (or brew install {tool} on a mac)
    • If that fails, I’ll probably search the internet for “Install {tool}” and hope to find the official documentation
  2. Check it is installed, and if tab-complete works
    • Type the first few characters of the command name (sna for snap) followed by <tab> <tab>, to see if the command name auto-completes, signifying that the system is aware of its existence
    • Hit space, and then <tab> <tab> again, to see if it shows me a list of available sub-commands, indicating that tab completion is set up correctly for the tool
  3. Try my first command
    • I’m probably following some documentation at this point, which will be telling me the first command to run (e.g. snap install {something}), so I’ll try that out and expect prompt succinct feedback to show me that it’s working
    • For basic tools, this may complete my initial interaction with the tool. For more complex tools like kubectl or git I may continue playing with it
  4. Try to do something more complex
    • Now I’m likely no longer following a tutorial, instead I’m experimenting on my own, trying to discover more about the tool
    • If what I want to do seems complex, I’ll straight away search the internet for how to do it
    • If it seems more simple, I’ll start looking for a list of subcommands to achieve my goal
    • I start with {tool} <tab> <tab> to see if it gives me a list of subcommands, in case it will be obvious what to do next from that list
    • If that fails I’ll try, in order, {tool} <enter>, {tool} -h, {tool} --help, {tool} help or {tool} /?
    • If none of those work then I’ll try man {tool}, looking for a Unix manual entry
    • If that fails then I’ll fall back to searching the internet again

UX recommendations

Considering my own experience of CLI tools, I am reasonably confident the following recommendations make good general practice guidelines:

  • Always implement a --help option on the main command and all subcommands, and if appropriate print out some help when no options are provided ({tool} <enter>)
  • Provide both short- (e.g. -h) and long- (e.g. --help) form options, and make them guessable
  • Carefully consider the naming of all subcommands and options, use familiar words where possible (e.g. help, clean, create)
  • Be consistent with your naming – have a clear philosophy behind your use of subcommands vs options, verbs vs nouns etc.
  • Provide helpful, readable output at all times – especially when there’s an error (npm I’m looking at you)
  • Use long-form options in documentation, to make commands more self-explanatory
  • Make the tool easy to install with common software management systems (snap, apt, Homebrew, or sometimes NPM or pip)
  • Provide tab-completion. If it can’t be installed with the tool, make it easy to install and document how to set it up in your installation guide
  • Command outputs should use the appropriate output streams (STDOUT and STDERR) and should be as user-friendly and succinct as possible, and ideally make use of terminal colours

Some of these recommendations are easier to implement than others. Ideally every command should consider their subcommands and options carefully, and implement --help. But writing auto-complete scripts is a significant undertaking.

Similarly, packaging your tool as a snap is significantly easier than, for example, adding software to the official Ubuntu software sources.

Although I believe all of the above to be good general advice, I would very much welcome research to highlight the relative importance of addressing each concern.

Outstanding questions

There are a number of further questions for which the answers don’t seem obvious to me, but I’d love to somehow find out the answers:

  • Once users have learned the short-form options (e.g. -h) do they ever use the long-form (e.g. --help)?
  • Do users prefer subcommands (mytool create {something}) or options (mytool --create {something})?
  • For multi-level commands, do users prefer {tool} {object} {verb} (e.g. git remote add {remote_name}), or {tool} {verb} {object} (e.g. kubectl get pod {pod_name}), or perhaps {tool} {verb}-{object} (e.g. juju remove-application {app_name})?
  • What patterns exist for formatting command output? What’s the optimal length for users to read, and what types of formatting do users find easiest to understand?

If you know of either authoritative recommendations or existing research on these topics, please let me know in the comments below.

I’ll try to write a more in-depth follow-up to this post when I’ve explored a bit further on some of these topics.

Read more
Inayaili de León Persson

Last month the web team ran its first design sprint as outlined in The Sprint Book, by Google Ventures’ Jake Knapp. Some of us had read the book recently and really wanted to give the method a try, following the book to the letter.

In this post I will outline what we’ve learned from our pilot design sprint, what went well, what could have gone better, and what happened during the five sprint days. I won’t go into too much detail about explaining what each step of the design sprint consists of — for that you have the book. If you don’t have that kind of time, but would still like to know what I’m talking about, here’s an 8-minute video that explains the concept:

 

Before the sprint

One of the first things you need to do when running a design sprint is to agree on a challenge you’d like to tackle. Luckily, we had a big challenge that we wanted to solve: ubuntu.com‘s navigation system.

 

ubuntu.com navigation layers: global nav, main nav, second and third level navubuntu.com’s different levels of navigation

 

Assigning roles

If you’ve decided to run a design sprint, you’ve also probably decided who will be the Facilitator. If you haven’t, you should, as this person will have work to do before the sprint starts. In our case, I was the Facilitator.

My first Facilitator task was to make sure we knew who was going to be the Decider at our sprint.

We also agreed on who was going to participate, and booked one of our meeting rooms for the whole week plus an extra one for testing on Friday.

My suggestion for anyone running a sprint for the first time is to also name an Assistant. There is so much work to do before and during the sprint, that it will make the Facilitator’s life a lot easier. Even though we didn’t officially name anyone, Greg was effectively helping to plan the sprint too.

Evangelising the sprint

In the week that preceded the sprint, I had a few conversations with other team members who told me the sprint sounded really great and they were going to ‘pop in’ whenever they could throughout the week. I had to explain that, sadly, this wasn’t going to be possible.

If you need to do the same, explain why it’s important that the participants commit to the entire week, focusing on the importance of continuity and of accumulated knowledge that the sprint’s team will gather throughout the week. Similarly, be pleasant but firm when participants tell you they will have to ‘pop out’ throughout the week to attend to other matters — only the Decider should be allowed to do this, and even so, there should be a deputy Decider in the room at all times.

Logistics

Before the sprint, you also need to make sure that you have all the supplies you need. I tried as much as possible to follow the suggestions for materials outlined in the book, and I even got a Time Timer. In retrospect, it would have been fine for the Facilitator to just keep time on a phone, or a less expensive gadget if you really want to be strict with the no-phones-in-the-room policy.

Even though the book says you should start recruiting participants for the Friday testing during the sprint, we started a week before that. Greg took over that side of the preparation, sending prompts on social media and mailing lists for people to sign up. When participants didn’t materialise in this manner, Greg sent a call for participants to the mailing list of the office building we work at, which worked wonders for us.

Know your stuff

Assuming you have read the book before your sprint, if it’s your first sprint, I recommend re-reading the chapter for the following day the evening before, and take notes.

I printed out the checklists provided in the book’s website and wrote down my notes for the following day, so everything would be in one place.

 

Facilitator checklist with handwritten notesFacilitator checklists with handwritten notes

 

I also watched the official video for the day (which you can get emailed to you by the Sprint Bot the evening before), and read all the comments in the Q&A discussions linked to from the emails. These questions and comments from other people who have run sprints was incredibly useful throughout the week.

 

Sprint Bot emailSprint Bot email for the first day of the sprint

 

Does this sound like a lot of work? It was. I think if/when we do another sprint the time spent preparing will probably be reduced by at least 50%. The uncertainty of doing something as involved as this for the first time made it more stressful than preparing for a normal workshop, but it’s important to spend the time doing it so that things run smoothly during the sprint week.

Day 1

The morning of the sprint I got in with plenty of time to spare to set up the room for the kick-off at 10am.

I bought lots of healthy snacks (which were promptly frowned on by the team, who were hoping for sweater treats); brought a jug of water and cups, and all the supplies to the room; cleared the whiteboards; and set up the chairs.

What follows are some of the outcomes, questions and other observations from our five days.

Morning

In the morning of day 1 you define a long term goal for your project, list the ways in which the project could fail in question format, and draw a flowchart, or map, of how customers interact with your product.

  • Starting the map was a little bit tricky as it wasn’t clear how the map should look when there are more than one type of customer who might have different outcomes
  • In the book there are no examples with more than one type of customer, which meant we had to read and re-read that part of the book until we decided how to proceed as we have several customer types to cater for
  • Moments like these can take the confidence in the process away from the team, that’s why it’s important for the Facilitator to read everything carefully more than once, and ideally for him or her not to be the only person to do so
  • We did the morning exercises much faster than prescribed, but the same didn’t happen in the afternoon!

 

The team discussing the target for the sprint in front of the journey mapDiscussing the target for the sprint

 

Afternoon

In the afternoon experts from the sprint and guests come into the room and you ask them lots of questions about your product and how things work. Throughout the interviews the team is taking notes in the “How Might We” format (for example, “How might we reduce the amount of copy?”). By the end of the interviews, you group the notes into themes, vote on the ones you find most useful or interesting, move the most voted notes onto their right place within your customer map and pick a target in the map as the focus for the rest of the sprint.

  • If you have time, explain “How Might We” notes work before the lunch break, so you save that time for interviews in the afternoon
  • Each expert interview should last for about 15-30 minutes, which didn’t fee like long enough to get all the valuable knowledge from our experts — we had to interrupt them somewhat abruptly to make sure the interviews didn’t run over. Next time it might be easier to have a list of questions we want to cover before the interviews start
  • Choreographing the expert interviews was a bit tricky as we weren’t sure how long each would take. If possible, tell people you’ll call them a couple of minutes before you need them rather than set a fixed time — we had to send people back a few times because we weren’t yet finished asking all the question to the previous person!
  • It took us a little longer than expected to organise the notes, but in the end, the most voted notes did cluster around the key section of the map, as predicted in the book!

 

How Might We notes on the wallsSome of the How Might We notes on the wall after the expert interviews

 

Other thoughts on day 1

  • Sprint participants might cancel at the last minute. If this happens, ask yourself if they could still appear as experts on Monday afternoon? If not, it’s probably better to write them off the sprint completely
  • There was a lot of checking the book as the day went by, to confirm we were doing the right thing
  • We wondered if this comes up in design sprints frequently: what if the problem you set out to solve pre-sprint doesn’t match the target area of the map at the end of day 1? In our case, we had planned to focus on navigation but the target area was focused on how users learn more about the products/services we offer

A full day of thinking about the problem and mapping it doesn’t come naturally, but it was certainly useful. We conduct frequent user research and usability testing, and are used to watching interviews and analysing findings, nevertheless the expert interviews and listening to different perspectives from within the company was very interesting and gave us a different type of insight that we could build upon during the sprint.

Day 2

By the start of day 2, it felt like we had been in the sprint for a lot longer than just one day — we had accomplished a lot on Monday!

Morning

The morning of day 2 is spent doing “Lightning Demos” after a quick 20-minute research. These can be anything that might be interesting, from competitor products to previous internal attempts at solving the sprint challenge. Before lunch, the team decides who will sketch what in the afternoon: will everyone sketch the same thing or different parts of the map.

  • We thought the “Lightning Demos” was a great way to do demos — it was fast and captured the most important thing quickly
  • Deciding who would sketch what wasn’t as straightforward as we might have thought. We decided that everyone should do a journey through our cloud offerings so we’d get different ideas on Wednesday, knowing there was the risk of not everything being covered in the sketches
  • Before we started sketching, we made a list of sections/pages that should be covered in the storyboards
  • As on day 1, the morning exercises were done faster than prescribed, we were finished by 12:30 with a 30 minute break from 11-11:30

 

Sketches from lightning demosOur sketches from the lightning demos

 

Afternoon

In the afternoon, you take a few minutes to walk around the sprint room and take down notes of anything that might be useful for the sketching. You then sketch, starting with quick ideas and moving onto a more detailed sketch. You don’t look at the final sketches until Wednesday morning.

  • We spent the first few minutes of the afternoon looking at the current list of participants for the Friday testing to decide which products to focus on in our sketches, as our options were many
  • We had a little bit of trouble with the “Crazy 8s” exercise, where you’re supposed to sketch 8 variations of one idea in 8 minutes. It wasn’t clear what we had to do so we re-read that part a few times. This is probably the point of the exercise: to remove you from your comfort zone, make you think of alternative solutions and get your creative muscles warmed up
  • We had to look at the examples of detailed sketches in the book to have a better idea of what was expected from our sketches
  • It took us a while to get started sketching but after a few minutes everyone seemed to be confidently and quietly sketching away
  • With complicated product offerings there’s the instinct to want to have access to devices to check product names, features, etc – I assumed this was not allowed but some people were sneakily checking their laptops!
  • Naming your sketch wasn’t as easy as it sounded
  • Contrary to what we expected, the afternoon sketching exercises took longer than the morning’s, at 5pm some people were still sketching

 

The team sketchingEveryone sketching in silence on Tuesday afternoon

 

Tuesday was lots of fun. Starting the day with the demos, without much discussion on the validity of the ideas, creates a positive mood in the team. Sketching in a very structured manner removes some of the fear of the blank page, as you build up from loose ideas to a very well-defined sketch. The silent sketching was also great as it meant we had some quiet time to pause and think a solution through, giving the people who tend to be more quiet an opportunity to have their ideas heard on par with everyone else.

Day 3

No-one had seen the sketches done on Tuesday, so the build-up to the unveiling on day 3 was more exciting than for the usual design review!

Morning

On the Wednesday morning, you decide which sketch (or sketches) you will prototype. You stick the sketches on the wall and review them in silence, discuss each sketch briefly and each person votes on their favourite. After this, the Decider casts three votes, which can follow or not the votes of the rest of the team. Whatever the Decider votes on will be prototyped. Before lunch, you decide whether you will need to create one or more prototypes, depending on whether the Decider’s (or Deciders’) votes fit together or not.

  • We had 6 sketches to review
  • Although the book wasn’t clear as to when the guest Decider should participate, we invited ours from 10am to 11.30am as it seemed that he should participate in the entire morning review process — this worked out well
  • During the speed critique people started debating the validity or feasibility of solutions, which was expected but meant some work for the Facilitator to steer the conversation back on track
  • The morning exercises put everyone in a positive mood, it was an interesting way to review and select ideas
  • Narrating the sketches was harder than what might seem at first, and narrating your own sketch isn’t much easier either!
  • It was interesting to see that many of the sketches included similar solutions — there were definite patterns that emerged
  • Even though I emphasised that the book recommends more than one prototype, the team wasn’t keen on it and the focus of the pre-lunch discussion was mostly on how to merge all the voted solutions into one prototype
  • As for all other days, and because we decided for an all-in-one prototype, we finished the morning exercises by noon

 

Reviewing the sketches in silenceThe team reviewing the sketches in silence on Wednesday morning

 

Afternoon

In the afternoon of day 3, you sketch a storyboard of the prototype together, starting one or two steps before the customer encounters your prototype. You should move the existing sketches into the frames of the storyboard when possible, and add only enough detail that will make it easy to build the prototype the following day.

  • Using masking tape was easier than drawing lines for the storyboard frames
  • It was too easy to come up with new ideas while we were drawing the storyboard and it was tricky to tell people that we couldn’t change the plan at this point
  • It was hard to decide the level of detail we needed to discuss and add to the storyboard. We finished the first iteration of the storyboard a few minutes before 3pm. Our first instinct was to start making more detailed wireframes with the remaining time, but we decided to take a break for coffee and come back to see where we needed more detail in the storyboard instead
  • It was useful to keep asking the team what else we needed to define as we drew the storyboard before we started building the prototype the following day
  • Because we read out the different roles in preparation for Thursday, we ended up assigning roles straight away

 

Drawing the storyboardDiscussing what to add to our storyboard

 

Other thoughts on day 3

  • One sprint participant couldn’t attend on Tuesday, but was back on Wednesday, which wasn’t ideal but didn’t impact negatively
  • While setting up for the third day, I wasn’t sure if the ideas from the “Lightning Demos” could be erased from the whiteboard, so I took a photo of them and erased it as, even with the luxury of massive whiteboards, we wouldn’t have had space for the storyboard later on!

By the end of Wednesday we were past the halfway mark of the sprint, and the excitement in anticipation for the Friday tests was palpable. We had some time left before the clock hit 5 and wondered if we should start building the prototype straight away, but decided against it — we needed a good night’s sleep to be ready for day 4.

Day 4

Thursday is all about prototyping. You need to choose which tools you will use, prioritising speed over perfection, and you also need to assign different roles for the team so everyone knows what they need to do throughout the day. The interviewer should write the interview script for Friday’s tests.

  • For the prototype building day, we assigned: two writers, one interviewer, one stitcher, two makers and one asset collector
  • We decided to build the pages we needed with HTML and CSS (instead of using a tool like Keynote or InVision) as we could build upon our existing CSS framework
  • Early in the afternoon we were on track, but we were soon delayed by a wifi outage which lasted for almost 1,5 hours
  • It’s important to keep communication flowing throughout the day to make sure all the assets and content that are needed are created or collected in time for the stitcher to start stitching
  • We were finished by 7pm — if you don’t count the wifi outage, we probably would have been finished by 6pm. The extra hour could have been curtailed if there had been just a little bit more detail in the storyboard page wireframes and in the content delivered to the stitcher, and fewer last minute tiny changes, but all-in-all we did pretty well!

 

Maker and asset collector working on the prototypeJoana and Greg working on the prototype

 

Other thoughts on day 4

  • We had our sprint in our office, so it would have been possible for us to ask for help from people outside of the sprint, but we didn’t know whether this was “allowed”
  • We could have assigned more work to the asset collector: the makers and the stitcher were looking for assets themselves as they created the different components and pages rather than delegating the search to the asset collector, which is how we normally work
  • The makers were finished with their tasks more quickly than expected — not having to go through multiple rounds of reviews that sometimes can take weeks makes things much faster!

By the end of Thursday there was no denying we were tired, but happy about what we had accomplished in such a small amount of time: we had a fully working prototype and five participants lined up for Friday testing. We couldn’t wait for the next day!

Day 5

We were all really excited about the Friday testing. We managed to confirm all five participants for the day, and had an excellent interviewer and solid prototype. As the Facilitator, I was also happy to have a day where I didn’t have a lot to do, for a change!

Thoughts and notes on day 5

On Friday, you test your prototype with five users, taking notes throughout. At the end of the day, you identify patterns within the notes and based on these you decide which should be the next steps for your project.

  • We’re lucky to work in a building with lots of companies who employ our target audience, but we wonder how difficult it would have been to find and book the right participants within just 4 days if we needed different types of users or were based somewhere else
  • We filled up an entire whiteboard with notes from the first interview and had to go get extra boards during the break
  • Throughout the day, we removed duplicate notes from the boards to make them easier to scan
  • Some participants don’t talk a lot naturally and need a lot of constant reminding to think out loud
  • We had the benefit of having an excellent researcher in our team who already knows and does everything the book recommends doing. It might have been harder for someone with less research experience to make sure the interviews were unbiased and ran smoothly
  • At the end of the interviews, after listing the patterns we found, we weren’t sure whether we could/should do more thorough analysis of the testing later or if we should chuck the post-it notes in the bin and move on
  • Our end-of-sprint decision was to have a workshop the following week where we’d plan a roadmap based on the findings — could this be considered “cheating” as we’re only delaying making a decision?

 

The team in the observation roomThe team observing the interviews on Friday

 

A wall of interview notesA wall of interview notes

 

The Sprint Book notes that you can have one of two results at the end of your sprint: an efficient failure, or a flawed success. If your prototype doesn’t go down well with the participants, your team has only spent 5 days working on it, rather than weeks or potentially months — you’ve failed efficiently. And if the prototype receives positive feedback from participants, most likely there will still be areas that can be improved and retested — you’ve succeeded imperfectly.

At the end of Friday we all agreed that we our prototype was a flawed success: there were things we tested that we’d had never think to try before and that received great feedback, but some aspects certainly needed a lot more work to get right. An excellent conclusion to 5 intense days of work!

Final words

Despite the hard work involved in planning and getting the logistics right, running the web team’s trial design sprint was fun.

The web team is small and stretched over many websites and products. We really wanted to test this approach so we could propose it to the other teams we work with as an efficient way to collaborate at key points in our release schedules.

We certainly achieved this goal. The people who participated directly in the sprint learned a great deal during the five days. Those in the web team who didn’t participate were impressed with what was achieved in one week and welcoming of the changes it initiated. And the teams we work with seem eager to try the process out in their teams, now that they’ve seen what kind of results can be produced in such a short time.

How about you? Have you run a design sprint? Do you have any advice for us before we do it again? Leave your thoughts in the comments section.

Read more
Rae Shambrook

Recently I have been working on the visual design for RCS (which stands for rich communications service) group chat. While working on the “Group Info” screen, we found ourselves wondering what the best way to display an online/offline status. Some of us thought text would be more explicit but others thought  it adds more noise to the screen. We decided that we needed some real data in order to make the best decision.

Usually our user testing is done by a designated Researcher but usually their plates are full and projects bigger, so I decided to make my first foray into user testing. I got some tips from designers who had more experience with user testing on our cloud team; Maria  Vrachni, Carla Berkers and Luca Paulina.

I then set about finding my user testing group. I chose 5 people to start with because you can uncover up to 80% of usability issues from speaking to 5 people. I tried to recruit a range of people to test with and they were:

  1. Billy: software engineer, very tech savvy and tech enthusiast.
  2. Magda: Our former PM and very familiar with our product and designs.
  3. Stefanie: Our Office Manager who knows our products but not so familiar with our designs.
  4. Rodney: Our IS Associate who is tech savvy but not familiar with our design work
  5. Ben: A copyeditor who has no background in tech or design and a light phone user.

The tool I decided to use was Invision. It has a lot of good features and I already had some experience creating lightweight prototypes with it. I made four minimal prototypes where the group info screen had a mixture of dots vs text to represent online status and variations on placement.  I then put these on my phone so my test subjects could interact with it and feel like they were looking at a full fledged app and have the same expectations.

group_chat_testing

During testing, I made sure not to ask my subjects any leading questions. I only asked them very broad questions like “Do you see everything you expect to on this page?” “Is anything unclear?” etc. When testing, it’s important not to lead the test subjects so they can be as objective as possible. Keeping this in mind, it was interesting to to see what the testers noticed and brought up on their own and what patterns arise.

My findings were as follows:

Online status: Text or Green Dot

Unanimously they all preferred online status to be depicted with colour and 4 out of 5 preferred the green dot rather than text because of its scannability.

Online status placement:

This one was close but having the green dot next to the avatar had the edge, again because of its scannability. One tester preferred the dot next to the arrow and another didn’t have a preference on placement.

Pending status:

What was also interesting is that three out of the four thought “pending” had the wrong placement. They felt it should have the same placement as online and offline status.

Overall, it was very interesting to collect real data to support our work and looking forward to the next time which will hopefully be bigger in scope.

group_chat_final

The finished design

Read more
Carla Berkers

OpenStack is the leading open cloud platform, and Ubuntu is the world’s most popular operating system for OpenStack. Over the past two years we have created a tool that allows users to build an Ubuntu OpenStack cloud on their own hardware in a few simple steps: Autopilot.

This post covers the design process we followed on our journey from alpha to beta to release.

Alpha release: getting the basics right

We started by mapping out a basic Autopilot journey based on stakeholder requirements and designed a first cut of all the necessary steps to build a cloud:

  1. Choose the cloud configuration from a range of OpenStack optionsChoose cloud configuration
  1. Select the hardware the cloud should be built on
    Select the hardware
  1. View deployment status while the cloud is being built
    View deployment status
  1. Monitor the status and usage of the cloud
    Monitor Cloud

After the initial design phase Autopilot was developed and released as an alpha and a beta. This means that for over a year, there was a product to play around with, test and improve before it was made generally available.

Beta release: feedback and improvements

Providing a better overview: increased clarity in the dashboard

Almost immediately after the engineering team started building our new designs, we discovered that we needed to display an additional set of data on the storage graphs. On top of that, some guerilla testing sessions with Canonical engineers brought to light that the CPU and the storage graphs were easily misinterpreted.

dashboard-sketches

After some more competitive research and exploratory sketching, we decided to merge the graphs for each section by putting the utilisation on a vertical axis and the time on the horizontal axis. This seemed to improve the experience for our engineers, but we also wanted to validate with users in usability testing, so we tested the designs with eight participants that were potential Autopilot users. From this testing we learned to include more information on the axes and to include detailed information on hover.

The current graphs are quite an evolution compared to what we started with:
Improved dashboard graphs

Setting users up for success: information and help before the process begins

Before a user gets to the Autopilot wizard, they have to configure their hardware, install an application called MAAS to register machines and install Landscape to get access to Autopilot. A third tool called Juju is installed to help Autopilot behind the scenes.

All these bits of software work together to allow users to build their clouds; however, they are all developed as stand-alone products by different teams. This means that during the initial design phase, it was a challenge to map out the entire journey and get a good idea of how the different components work together.

Only when the Autopilot beta was released, was it finally possible for us to find some hardware and go through the entire journey ourselves, step by step. This really helped us to identify common roadblocks and points in the journey where more documentation or in-app explanation was required.

Increasing transparency of the process: helping users anticipate what they need and when configuration is complete

Following our walk-through, we identified a number of points in the Autopilot journey where contextual help was required. In collaboration with the engineering team we gathered definitions of technical concepts, technical requirement, and system restrictions.

Autopilot walk-through

Based on this info, we made adjustments to the UI. We designed a landing page  with a checklist and introduction copy, and we added headings, help text, and tooltips to the installation and dashboard page. We also included a summary panel on the configuration page, to guide users through the journey and provide instant feedback.

BR_step-by-step

GA release: getting Autopilot ready for the general public

Perhaps the most rewarding type of feedback we gathered from the beta release — our early customers liked Autopilot but wanted more features. From the first designs Autopilot has aimed to help users quickly set up a test cloud. But to use Autopilot to build a production cloud, additional features were required.

Testing without the hardware: try Autopilot on VMware

One of the biggest improvements for GA release was making it easy to try Autopilot, even for people that don’t have enough spare hardware to build a cloud. Our solution: try Autopilot using VMware!

Supporting customisation:  user-defined roles for selected hardware

In the alpha version a user could already select nodes, but in most enterprises users want more flexibility. Often there are different types of hardware for different roles in the cloud, so users don’t always want to automatically distribute all the OpenStack services over all the machines. We designed the ability to choose specific roles like storage or compute for machines, to allow users to make the most of their hardware.

Machine roles

Allowing users more control: a scalable cloud on monitored hardware

The first feature we added was the ability to add hardware to the cloud. This makes it possible to grow a small test cloud into a production sized solution. We also added the ability to integrate the cloud with Nagios, a common monitoring tool. This means if something happens on any of the cloud hardware, users would receive a notification through their existing monitoring system.

BR-Nagios

The benefits of early release

This month we are celebrating another  release of OpenStack Autopilot. In the two years since we started designing Autopilot, we have been able to add many improvements and it has been a great experience for us as designers to contribute to a maturing product.

We will continue to iterate and refine the features that are launched and we’re currently mapping the roadmap for the months ahead. Our goal remains for Autopilot to be a tool for users to maintain and upgrade an enterprise grade cloud that can be at the core of their operations.

 

Read more
Carla Berkers

As the number of Juju users has been rapidly increasing over the past year, so has the number of new solutions in the form of charms and bundles. To help users assess and choose solutions we felt it would be useful to improve the visual presentation of charm and bundle details on manage.jujucharms.com.

While we were in Las Vegas, we took advantage of the opportunity to work with the Juju developers and solutions team to find out how they find and use existing charms and bundles in their own deployments. Together we evaluated the existing browsing experience in the Juju GUI and went through JSON-files line by line to understand what information we hold on charms.

carla.jpg

We used post-its to capture every piece of information that the database holds about a bundle or charm that is submitted to charmworld.

 

We created small screen wireframes first to really focus on the most important content and how it could potentially be displayed in a linear way. After showing the wireframes to a couple more people we used our guidelines to create mobile designs that we can scale out to tablet and desktop.

With the grouped and prioritised information in mind we created the first draft of the wireframes.

 

In order to verify and test our designs, we made them modular. Over time it will be easy to move content around if we want to test if another priority works better for a certain solution. The mobile-first approach is a great tool for making sense of complex information and forced us to prioritise the content around user’s needs.

jaas-store

First version designs.

Read more
Carla Berkers

I’d like to share my experience working on the project that has been my main focus over the past months: the redesign of canonical.com.

Research methods

As I started talking to people in the design department I quickly discovered we have a lot of information about our site visitors. My colleagues helped me access Google Analytics data and findings from previous user testing sessions. There was also a research-based set of personas, that helped me to put together an initial overview of user needs and tasks.

I was curious to try to validate these user needs and test them against Canonical’s current business requirements. In order to find out more about the company goals I prepared a stakeholder interview script and started putting together a list of people to talk to. I initially planned to limit the number of interviewees to about six to eight stakeholders, as too many opinions could potentially slow down the project and complicate the requirements.

Getting to know the company

I started with eight people to talk to, but with each interview I found out about other people that I should add to my list. At the same time, many of the interviewees warned me that every person I would talk to would have different ideas about the site requirements. By the end of the first round of interviews, ending up with too many stakeholders turned out to be the most commonly mentioned risk to endanger the project finishing on time.

I had very diverse conversations about different aspects of the site with a range of people. From strategic insights from our CEO Jane, to brand guidelines and requirements from our Head of Design Marcus and ideas around recruitment from our HR business partner Alice — each conversation brought unique requirements to light.

After I spoke to about fifteen people I summarised the key points from each stakeholder on post it notes and put them all up on a wall in one of the meeting rooms in the office. As I took out the duplicates and restructured the remaining notes, I began to see a familiar pattern.

Conclusions

When I finished grouping the different audiences, I ended up with five groups of users: enterprise customers, (potential) partners, job seekers, media (a varied group that includes press, tech analysts and bloggers), open source advocates and the more generic tech enthusiasts that want to know more about the company backing Ubuntu.

As these groups aligned very well with the persona’s and other pieces of research that I had found, I felt comfortable continuing my process by moving on to the user needs and site goals that will help build a good site structure and generate useful content for each group of users.

I found that talking to many experts from within the company helped me quickly understand the full range of requirements, saving me time rather than making my job more complicated. Furthermore I was happy to get a chance to get to know people from different parts of the company so soon after I got started.

In order to keep the project moving forward, we appointed one key stakeholder to sign off each step of the process, but I’m looking forward to showing the end results to the broader group to see if I managed to meet all their expectations. We will also conduct user-testing to ensure the site answers our core audiences questions and allows them to complete their tasks.

I hope to share more about this project in the months to come.

Read more
Tingting Zhao

In the previous post, we talked about how to design effective user testing tasks to evaluate the usability of an interface. This post continues with this topic by highlighting a number of key strategies that you may need to use when conducting a formative user testing, whose main aim is to identify usability problems and propose design solutions, rather than compare quantitative metrics (summative testing), eg. task completion time and mouse clicks. It is unlikely that the prepared task script could be strictly applied without any changes, since the testing situation tends to be dynamic and often unpredictable. To get useful data, you need to be able to manipulate your task script with flexibilities, while also maintaining consistency.

Changing task orders during testing

Normally, to avoid order effect, the issuing of the tasks should be randomised for each participant. Order effect refers to the effect in which the tasks are presented, which can affect the results, specifically: users may perform better in later tasks as their familiarity with the interface increases; or users may perform worse as they become more fatigued. However, as discussed in the previous post, the tasks are often contextual and dependent on each other, so you need to carefully consider which tasks could be shuffled. For example, it is a good practice to mark on the script to indicate dependent tasks, so that you know these tasks should not be reordered and separated from each other. In other words, the dependent tasks must always be moved together. It is worth noting that the randomisation of task orders may not always be possible for a testing, for example, when the tasks are procedurally related, such as a testing focusing on payment flow.

Sometimes you may need to change the task orders by considering their levels of difficulty. This is useful in the following two scenarios: when you notice a participant appears to be slightly nervous before the testing has started, provide a simple first task to put him/her at ease; or when you observe a participant has failed to solve several tasks in a row, provide one or two easy tasks to reduce the frustration and stress, and boost confidence.

Another type of changing task order is made in response to users’ solicited goals that are associated with a coming task. For example, in one phone testing, after a participant checked the battery level, s/he spontaneously expressed a desire to know if there was a way to switch off some running apps to save battery. In this case, we jumped to the task of closing running apps, rather than waiting until later. This makes the testing feel more natural.

Remove tasks during testing

There are typically two situations that require you to skip tasks:

  • Time restriction

  • Questions being answered with previous tasks

Time restriction: user testing normally has a time limit. Participants are paid for certain lengths of time. Ideally, all the tasks should be carried out by all the participants. However, sometimes they take longer to solve tasks. Or you may discover areas that require more time for investigation. In this case, not all the tasks could be performed by a participant within the given time. Subsequently, you need to be able to quickly decide which tasks should be abandoned for this specific participant. There are two ways to approach this:

  • Omit tasks that are less important: it is always useful to prioritise the tasks in terms of their importance – what are the most important areas that have key questions that need to be answered and require feedback; what could be left for the next testing, if not covered this time?

  • Omit tasks that have already received abundant feedback: skip the tasks from which you have already gathered rich and useful information from other participants.

Questions were answered with previous tasks: Sometimes questions associated with a specific task would be answered while a participant was attempting to solve the previous task – in this case, you could skip this task.

In one of our phone testings, we asked a participant to send a text to a non-contact (a plumber). During the task-solving process, s/he decided to save the number to the contact book first and then send a text. In this case, we skipped the task of ‘saving a number to contact book’.

However sometimes you should not skip a task, even if it might seem repetitive. For example, if you want to test the learnability and memorability of a feature, having the participant perform the same task (with slightly different descriptions) for the second time (after a certain time duration) could afford useful insights.

Add tasks during testing

There are three contexts in which you could consider adding tasks:

  • Where the user formulates new goals

  • Re-examinations

  • Giving the user a second chance

The added task must be relevant to the aim of the testing, and should only be included if the testing time permits.

User formulates new goals: you could add tasks based on user-formulated goals in the task solving process.

For example, in one phone testing, one participant wondered if s/he could customise the tiles on the Windows phone’s home screen. We made this an added task for her/him. Adding tasks based on user articulated new goals follows their thought process and make the testing more natural. It also provides opportunities for us to discover new information.

Re-examinations: sometimes the users may succeed in a task accidently, without knowing how s/he did it. In this case, the same task (with a slightly changed description) could be added to re-assess the usability.

For example, in one phone testing, we had one task: “You want to call you sister Lisa to say thank you for the phone”. One participant experienced great difficulties in performing this task, and only completed it after a long time and by accident. In this case, we added another task to re-evaluate the ease of making a phone call:

“Your call is cut off while you are talking to your sister, so now you want to call her again.”

Similarly in the Gallery app testing, where participants managed to add a picture into a selected album accidently, we asked them to add another picture into a different album.

Re-examination allows us to judge accurately the impact of a problem, as well as to understand the learnability of interface – the extent to which users could detect and learn interaction patterns (even by accident), and apply the rules later.

Giving the user a second chance: in the majority of the user testing, participants used the evaluated interface for the first time. It could be very demanding for them to solve the tasks successfully in their first attempt. However, as the testing progresses, participants might discover more things, such as features and interaction patterns (although possibly by accident). Consequently, their knowledge of the interface may increase. In this case, you could give them another chance to solve the task that they failed earlier in the tests. Again, this helps you to test the learnability of the interface, as well as assess the impact of a problem.

For example, in a tablet testing, one participant could not find the music scope earlier in the testing, but later s/he accidentally discovered the video scope. To test if s/he now understood the concept of dash scopes, we asked the participant to find the music scope again after several other tasks.

Change task descriptions (slightly) during testing

Information gathered from pre-testing brief interview and participants’ testing verbal data could often be used to modify the task description slightly to make the task more realistic to the users. This also gives the user the impression that you are an active listener and interested in their comments, which helps to build a rapport with them. The change should be minor and limited to the details of the scenario (not the aim of the task). It is important that the change does not compromise the consistency with other participants’ task descriptions.

For example, in a tablet testing, where we needed to evaluate the discoverability of the HUD in the context of photo editing, we had this task: “You want to do some advanced editing by adjusting the colour of the picture.” One participant commented that s/he often changed pictures to ‘black and white’ effect. In response to this, we changed the task to “You mentioned that you often change a picture to black and white, and now you want to change this picture to ‘black and white’”. The task change here does not change the aim of the task, nor the requirements for solving the task (in this case, the access to the HUD), but it becomes more relatable to the participant.

Another example is from a phone testing. We changed the task of “you want to go to Twitter” to “you want to go to Facebook” after learning the participant uses Facebook but not Twitter. If we continued to request this participant to find Twitter, it would make the testing become artificial, which would result in invalid data. The aim of the task is to evaluate the ease of navigation in finding an app, therefore changing Twitter to Facebook does not change the nature of the task.

Conclusions

This post outlines a number of main strategies you could use to modify your task script to deal with typical situations that may occur in a formative user testing. To sum up:

Changing tasks orders: randomise tasks for each participant if possible, and move the dependent tasks as a whole; consider the difficulties of the task and issue an easy task to start with if you feel participant is nervous, or provide an easy task if participants failed several tasks in a row. Allow them to perform a later task if they verbalise it as a goal/strategy for solving the current task.

Remove tasks: if time is running out with a particular participant, omit certain tasks. This could be tasks with low priorities; tasks that already have enough feedback from other participants; or tasks the participant has already covered while attempting a previous task.

Add tasks: if time permits, allow users to perform a new task if it is a user initiated goal and is relevant to the testing; repeat a task (with slightly different wording and at an appropriate time) if the user succeeds in a task accidently, or has failed this task earlier, or if the aim is to test the learnability of the system.

Change task description: slightly amend the details of the task scenario (not the aim of the task) based on users’ verbal data to make it more relatable and realistic to the user. This will improve the reliability of the data.

If you have other ways to maneuver the tasks during the testing session, or have situations you are unsure about, feel free to share your experience and thoughts.

Read more
Tingting Zhao

In past years, we have had many Ubuntu users getting involved in helping with our user research. Now we feel it’s time to form a user research network, which we’re calling: UbuntuVoice.

So,  if you want to:

  • be the voice of over 20 million Ubuntu users. You will have the opportunities to take part in a variety of Ubuntu user research with different products, and help shape the Ubuntu experience. You choose the ones that you are interested in.

  • stay up to date with Ubuntu. Get periodic updates (every two months) via email, such as what designers are working on, how feedback is used, and how users behave when interacting with technology.

  • get a little something extra. Some of our research will come with an incentive, or in the form of a ‘Ubuntu goody’ lucky draw, and some research will be voluntary.

…then join us today by clicking here

If you have any questions, please feel free to contact us at: ubuntuvoice@gmail.com

 

Update: Thank you very much for everyone’s support for the UbuntuVoice! We reached our target number of participants in just a day! Since we are a small team, we can’t have more participants at the moment. However, do keep your eyes on the design blog for updates.

Ubuntu user research team

Read more
Tingting Zhao

Previously, Charline Poirier provided an excellent post about how to recruit representative participants for usability testing. To continue the story, we are going to talk about the next stage: developing effective task sets, which is a crucial part of a test protocol.

We conduct usability testing iteratively and throughout the product life cycle. The testing interface could range from being as simple as paper images, to clickable prototypes, to a fully working system.

In order to assess the usability of an interface, we ask users to carry out a number of tasks using the interface. We use tasks that resemble those that users would perform in a real life context, so that the data we collect is accurate. In other words, the user behaviour we observed is representative, and the problems we found are those that users would be likely to encounter.

 

Design testing tasks – ‘a piece of cake’?

 

When I first learnt about usability testing, I thought: ‘It’s simple: you just need to write some tasks and ask people to solve them, and done!’ But after conducting my first ever usability testing, I realised this was not the case.  I had so many questions: I wasn’t sure where to start or what tasks should be used, and there were numerous details that needed to be thought through. You need to carefully craft the tasks.

Now, having conducted hundreds of usability testings, I would like to share my experience with you about how to design effective tasks. There are three main stages involved:

  • Decide on the tasks

  • Formulate the tasks

  • Be tactful in presenting the order of the tasks

 

Stage 1: Decide on the tasks

Before you sit down to compose a set of tasks, you are likely to go through the following stages:

  • Clearly establish the goal of the testing: specifically what are the main features/areas that require feedback. When we conduct testing, we always have a face to face meeting with the design team to understand their focus and needs.

  • ‘Walkthrough’ with the design team: If testing an early prototype that has not been fully implemented, it’s important to go through the prototype with the designers so that you are aware of how it works, what is working and what is broken.

  • Inspection : go through the test interface at least three times. The first time to get an idea of the general flow and interaction of the interface; the second time to ‘put on the user’s hat’, and examine the interface by thinking about what users would do, and pay attention to any possible difficulties they may experience. This is the stage where you could start to write down some of the potential tasks you could use, which cover the features you need to assess, and the predicted problematic areas; and the third time, you should focus on developing tasks when you are going through the interface again. This gives you the opportunity to evaluate the tasks you identified, and add or remove tasks. By the end, you will have a number of potential task banks to work on.

Dumas and Fox (2008, p1131) provide a very good summary of the kind of tasks that are likely to be involved in usability testing. It is in line with those that we used in our testing sessions in most contexts. These include:

  • tasks that are important, such as frequently performed tasks or tasks that relate to important functions;

  • tasks where evaluators predict users will have difficulties;

  • tasks that enable a more thorough examination of the system, such as those that can only be accomplished by navigating to the bottom of the system hierarchy, or tasks that have multi-links or shortcuts;

  • tasks that influence business goals;

  • tasks that examine the re-designed areas;

  • tasks that relate to newly-added features.

For this step, you don’t need to worry about how to phrase the task descriptions, but make sure all areas that you need to investigate are covered by your tasks.

Stage 2: Formulate the tasks

How well the tasks are formulated determines the reliability and the validity of the usability testing and the usefulness of the data. It’s crucial to get this right. You should consider:

  • The formats of tasks to be used
  • The articulation of the tasks

The formats of tasks

The tasks could be categorised into two main formats:

  • Direct tasks or Scenario tasks

  • Open-ended or Closed task

You need to decide what should be used, and when.

Scenario task or Direct task

A scenario task is presented as a mini user story: often it has the character, the context and the necessary details for achieving the goal. For example, to test the browser and bottom menu on the phone:

You are holding a dinner party this Saturday. You want to find a chicken curry recipe from the BBC food site.

A direct task is purely instructional. For instance, to use the above example:

Find a chicken curry recipe from the BBC food site.

Among these two types, we often use the scenario tasks in the testing. This is because it emulates real-world context that participants can easily relate to, and consequently they are more likely to behave in a natural way. This helps to mitigate the artificiality of user testing to a great extent.  The closer they are related to the reality, the more reliable the test results can be (eg. Rubin, 1994; Dumas and Fox, 2008). In addition, some research (eg. Shi, 2010) shows that the scenario tasks work more effectively with Asian participants.

Interesting research: for Indian participants, Apala Lahiri Chavan’s research (Schaffer, 2002) shows that using a ‘Bollywood’ style task would elicit more useful feedback. For example:

Your innocent and young sister is going to get married this Saturday, and you just get a news the prospective groom is already married! So you want to book a flight ticket as soon as possible to find your sister and save her.

The researchers found that Indian participants feel reluctant to make criticisms to the unfamiliar facilitator, but once they phrased the task in a film-like story, the participants became more talkative and open.

Closed task or Open-ended task

 A closed task is specific to what the participants need to do. This type of task has one correct answer, and therefore allows us to measure if participants solved or failed a task. It is the most commonly used format. For example, to test the telephony on the phone:

 You want to text your landlord to say you will give her the rent tomorrow. Her number is: 7921233290.

An open-ended task contains minimum information and less specific direction as to what you want a participant to do. It gives users more freedom to explore the system. This is particularly useful if you want to find out about what areas users would spontaneously interact with, or which ones matter most to them.

For example, in our Ubuntu.com testing, designers wanted to understand what information was important for users to get to know about Ubuntu. In this case, an open-ended task would be appropriate. I used the task:

 You heard your friends mention something called ‘Ubuntu’. You are interested in it and want to find out more about what Ubuntu is and what it can offer you?

There are three main limitations  of using open-ended tasks:

  • Since participants have control over the task, features that require user feedback might be missed; or vise versa, they may spend too much time on something that is not the focus of the testing. The remedy would be to prepare for a number of closed-tasks, so if certain features are not covered by the participants, these could be used.

  • Some participants may experience uncertainty as to where to look and when they have accomplished the task. Others may be more interested in getting the test done, and therefore do not put in as much effort as what they would in reality.

  • You cannot assign task success rates to open-ended tasks, as there is no correct answer, so it is not suitable if a performance comparison is needed.

The articulation of the tasks

  • Avoid task cues that would lead users to the answers. Make sure the tasks do not contain task solving related actions or terms that are used on the system. For example, in the Juju testing we wanted to know if participants understood the ‘browse’ link for browsing all the charms. We asked participants to find out the types of charms that are available instead of saying ‘you want to browse the charms’.

  • Be realistic and avoid ambiguity. The tasks should be those that would be carried out in the real context, and the descriptions should be unambiguous.

  • Ensure an appropriate level of details. It should contain just enough information so that participants understand what they are supposed to do, but not too much that they are restricted from exploring naturally in their own way. The description of context should not be too lengthy, otherwise participants may lose their focus or forget about it. When closed tasks are used, make sure they are specific enough, so it is clear to the participants as to when they would accomplish their goals. For example, compare the description of ‘You want to show your friends a picture’ to ‘You want to show your friends a picture of a cow’ – which one is better? For the former, the goal is more vague and participants would be likely to click on the first image or a random picture, and assume the task is done. As a result, we might miss usability problems. For the latter,  the task communicates the requirements more effectively: it would be accomplished once they found the picture of a cow. Furthermore, it also provides us with more opportunities to assess the navigation and interaction further, as participants need to navigate among the pictures to find the relevant one.

 

Stage 3: Be tactful in presenting the order of the tasks

In general, the tasks are designed to be independent from each other for two reasons: to grant flexibility in terms of changing the orders of the tasks for different participants; and to allow participants to continue to the next task, even if they failed the previous one.

However, in some contexts, we use dependent tasks (proceeding on to one task depends on whether or not participants solved another task successfully ) on purpose, for instance:

  • When there is a logistic flow involved and the stages of procedures must be followed. To use a very simple example, in order to test account ‘log in’ and ‘out’, we need a task for ‘log in’ first, and then a task for ‘log out’.

  • When testing ‘revisiting’/’back’ navigation (eg. if participants could navigate back to a specific location they visited before) and multitasking concepts (eg. if participants know to use the multitasking facility). For example, when testing the tablet, I had the tasks as follows:

You want to write down a shopping list for all the ingredients you need for this recipe using an app

Here, the participants will need to find the note app and enter ingredients.

Then I had several tasks that were not related to the task above, for example:

 You remember that you will have an important meeting with John this coming Thursday at 10:00 in your office. You want to put it on your calendar before you forget.

Then I instructed participants:

You want to continue with your shopping list by adding kitchen roll on it.

 This requests the participants to go back to the note app that they opened earlier, from which we could find out if they knew to use the right edge swipe to get to the running apps – in other words, whether or not they understood the multitasking feature.

Now you will have your first version of tasks, and on completion, you should always try the tasks out by using the interface to check that they all make sense.

 

Summing up

We use tasks to discover the usability and user experience of an interface. The task quality determines how useful and accurate your testing results would be. It requires time to hone your skills in writing tasks.  Let me sum up some of the main points:

  • Define the goal(s) of the testing;

  • Familiarise yourself with the test interface and go through this interface at least 3 times;

  • Use the appropriate task formats and avoid any inclusion of task-solving cues;

  • Ensure the description is realistic, is at the right level of detail, and avoids ambiguity;

  • Consider the ordering of the tasks, and whether or not you need to use dependent tasks;

  • Pilot the task set with yourself.

What happens next, after you have the list of tasks ready for the  the usability testing? It doesn’t end here.

If time allows, we always pilot the tasks with someone to make sure they are understandable, and that the orders of the tasks work. There are always changes you could make to improve the task sets.

In addition, you will realise that once you are in the actual testing,  no matter how perfect the task sets are,  you will need to react instantly and make adjustments in response to the dynamics of the testing environment: we cannot predict what participants will do. It is therefore important to know how to manipulate the task sets in the real testing condition. We will discuss this in the next post.

References

Dumas, J.S. & Loring, B.A. (2008). Moderating Usability Tests: Principles and Practices for Interacting. San Francisco, CA: Morgan Kaufmann.

Rubin, J. (1994). Handbook of Usability Testing: How to Plan, Design and Conduct Effective Tests. New York: John Wiley & Sons.

Schaffer, E. (2002) Bollywood technique, http://www.humanfactors.com/downloads/jun02.asp#bollywood

Shi, Q. (2010). An Empirical Study of Thinking Aloud Usability Testing from a Cultural Perspective. PhD thesis. Denmark: University of Copenhagen.

 

 

Read more
Tingting Zhao

Understanding user behaviour through user research is an integral part of our design process. In the last ubuntu.com website testing, some insights surfaced about user behaviour, which could help to shape a great user experience for our website. We share the three mains ones here. They have been much discussed in the UX, and the findings from the testing reinforced their importance.

Who were the participants?

12 participants took part in this research. They belonged to two different groups:

  • Ubuntu novices: those who have limited computer knowledge and had not heard of or used Ubuntu before. 8 participants were from this group. They were professionally recruited and of mixed genders.
  • Ubuntu users: those who use Ubuntu OS on a daily basis. They were from our Ubuntu users database pool and were recruited via emails.

What were the three main types of user behaviour found?

The Power of Images

“I go straight to the pictures before I go to the words. You look at pictures and they give you a flavour of what it is all about.”(P3)

” I use images to decide on a product. I tend to work very visually. Sometimes it is not easy to understand the jargon, and it is much easier to see what it is like. ” (P6)

“I’m just looking at the picture to see how much learning is required.” (P10)

In the testing process, we observed that participants appeared to rely on images heavily to help them form an opinion about Ubuntu. They used images in multiple ways throughout their interaction process, including:

  • To understand what the interface is about or make sense of an unfamiliar concept/feature
  • To decide whether or not it looks easy to use
  • To compare it with what they are currently using, and to see how much learning it may require

Images are therefore a powerful communication medium for us to build a positive brand with our users.

Take away:

It is important that images are relevant to their context and offer the best presentation of the product. We should use images to reflect the user friendliness and uniqueness of Ubuntu.

The Journey of Persuasion

“When I first came to your site, you need to tell me why I want to use this. This is paramount.” (P2)

“ It (the site) needs to highlight what I don’t know. Why I should use it, and with examples.” (P5)

When participants first landed on the homepage, they expressed their need to be informed about what Ubuntu does, who it is for, and why they should use it. They wanted to be convinced from the very start.

During the exploration process, when they were looking at Ubuntu pages, participants were attentive to the apparent benefits Ubuntu could offer to satisfy their personal needs. They relied on concrete examples and statistical figures to establish and enhance their understanding and trust. They also enjoyed browsing through different quotations from our users.

Take away:
The persuasion process should start from the moment users land on our homepage, until leaving the site. The key proposition messages should be specific, apparent and repeated throughout the user journey.

Make Use of Opportune Moments

“It says free upgrade for life, that’s good. Built in security, that’s good. Thousands of apps, that’s good too. I want to click on these to find out more.” (P3)

Our website has many good design features that grabbed participants’ attention straight away, for instance, the image tiles for ‘Reasons to love Ubuntu’ and the use of bullet points to outline essential information about Ubuntu’s main features. When participants encounter such design features or content that they found interesting, they often wanted to click an icon or topic to explore it further. They were disappointed or even frustrated if these were not clickable.

Take away:
We should make use of these opportune moments to keep users engaged and informed by providing efficient and desirable navigational paths to lead them to more detailed and relevant information.

What’s next ?

The web team has been carrying out changes in response to the user testing results. The aforementioned user behaviour findings will feed into the next web design cycle to help with the design decisions. This will help users to get even more out of their visits to Ubuntu.com.

Read more
Amritpal Singh Bhachu

Back to Lecturing for the day

In my last post, I spoke about my transition from academia to industry. One thing that I felt I would miss were the opportunities to speak to students, and watch their progression throughout the year. So when I was asked to go back to the University to give a talk, I jumped at the chance.

So I prepared what I was going to talk about and set off to the School of Computing at the University of Dundee to meet these talented students. My first job was to help assess their group pressure projects which they had been tasked with the week before. The theme was educational games. Over the next 2 hours, I sat and was amazed by what the groups had produced in such a short period of time.

The Winning Group with their Ubuntu prizes

Several things frustrated me however.

Each group had 3 minutes to present their game and explain what they did. But they all focussed on showing gameplay and illustrated some of the code that they used. A number of groups stood up and highlighted that they felt their game wasn’t very good because they didn’t have strong coders in their team. When I asked them questions about the processes that they had been through before coding, they all showed evidence of brainstorming, wireframing and design. My biggest issue however was that most of the groups started coding before they considered who the user would be, and therefore, they considered a user to meet the code, rather than producing the code for a specific user.

So this lead me to change what I wanted to talk to them about, and I did an interactive session with the 80 odd students to develop a user profile for the remit they had been given. We looked at who the user group was, what were the characteristics of this user, where would they want to play the game, why they would want to play the game and how they would play the game. We brainstormed on a whiteboard and agreed on which attributes to keep, and which to remove. This was all done in half and hour. The students really took on board the importance of considering the user, and how quickly it could be done for the projects that they would be presented with going forward in their education.

It was the most enjoyable lecture that I had ever taken, and I look forward to doing it again soon.

On another note, later that evening I made my triumphant return to the land of stand up comedy. I was invited back to do Bright Club Dundee having performed last year. It was great fun to do, even though I don’t think I’ll be looking at a change in career anytime soon! Below is a photo of the performers….you can quite clearly see the fear in our eyes!

Bright Club Dundee Performers

If you want to see my set (which contains strong language and little humour) then follow this link.

 

Read more
Paul Sladen

Normally the Ubuntu/Canonical Design Team are busy working on our own projects, but it makes a really good change to work on other Free Software design problems for a change. Starting at 22:00 UTC (15:00 PDT) on Monday 7 May 2012, you can join us for the next Ubuntu Design Theatre at the Ubuntu Developer Summit in Oakland, California:

Bring your design issues along and lets see how we can improve them! There should be visual designers, user interface designers, brand designers, … and the many other people who try and work to make users’ lives better with Free Software.

Read more
Paul Sladen

Some of original sketches for Ubuntu Arabic are about to go on display in Berlin! We’ve talked before about the work done by Rayan Abdullah on drawing and designing the original calligraphy behind the Ubuntu Arabic for the Ubuntu Font Family and from tomorrow you will be able to see that work for yourself.

Until 27 May 2012 you can see some of those original sketches and designs featuring in the Typobau exhbition at the Körnerpark Gallery in Neukölln, Berlin,

It includes many of Rayan’s design projects from the last decade, including the Bundesadler (the Federal Eagle of Germany) and his many Arabic graphic design and typography projects including the logos and typefaces for Burberry, McDonalds, Nokia Pure Arabic and the Ubuntu Font Family Arabic script coverage.

For keen visitors, the grand opening is this week, at 19:00 on Friday 20 April 2012. Or for anyone visiting Messe Berlin in May 2012 for Linuxtag 2012 you will still be able to catch the exhibition. Just take the S-Bahn ring anti-clockwise to S-Neukölln and see Ubuntu and Rayan’s exhibition at the same time as Linuxtag!

The “Typobau” exhibition runs between 21 April 2012 and 27 May 2012, 10:00–20:00, Tuesday—Sunday, at Körnerpark Galerie, Schierker Strasse 8, Berlin-Neukölln

Read more
Mika Meskanen

Ubuntu and Canonical had a very strong presence at this year’s Mobile World Congress in Barcelona. The main attraction was our Ubuntu for Android prototype that was published just a week earlier. The beautiful cubic pavilion also housed the Ubuntu TV demo, Ubuntu One, and our established desktop and cloud offerings. The booth attracted a constant flux of curious visitors representing all walks of life: media, industry people, businessmen, technology enthusiasts, students and… competitors.

John Lea, Oren Horev and myself from the Design Team joined Canonical sales, business and technical staff in this bold effort. In addition to running demos and having interesting conversations with the visitors to the booth, we also had the opportunity to have a look at the endless exhibition halls and floors of the conference and do research on what makes the mobile world tick at the moment.

If the MWC 2012 had to be summarised in one tagline, anyone would probably admit, that it was a one massive Androidfest.

Google’s recently upgraded operating system was simply everywhere. Spearheading the Android avalanche were the latest generation supermobiles – every device manufacturer was showing off with their versions of quad-core, high-definition, 4G/LTE smartphones and tablets bumped up to the latest specification.

Bells and whistles ranged from glasses-free 3D displays to Dolby sound to watertight casings – demonstrating that OEM customisations go beyond branding and skinning the interface.

Google themselves hosted an extensive Android area that was more like a theme park than a typical business affair: fans and passers-by were treated to a smoothie bar, a tube slide (presumably an homage to Google offices), grab-a-plush-Android game – and lucky ones could have had their Nexus phones pimped up with Svarovski crystals assembled by an industrial robot.

In stark contrast to Google’s rather playful attitude towards their ecosystem, the manufacturers were more poised for flexing their technological muscle. The impending hockey-stick curve of serious mobile computing power seems to all but prove the concept behind Ubuntu for Android. The phones of the near future are going to effortlessly run desktop and mobile operating systems simultaneously, and those extra cores can do more than just keep your hands warm in your pocket. Similarly, in our hands-on testing, the demoed 4G/LTE connections were lightning fast, signalling that accessing your cloud and thin client applications from a phone running a full productivity desktop can shift the paradigms of your mobile working life.

While this year’s congress was overrun by Android, it will be interesting to see whether this will be repeated next year, when we can assume to see the effects of Google’s Motorola acquisition and the impact of Windows 8. The latter had reached Consumer Preview stage and was presented in a separate session outside the main exhibition.

Most of the manufacturers had an odd Windows Phone in their inventory, but basically its marketing was left to Nokia, who also occupied a substantial exhibition floor not far from us. The newfound underdogs were quite upbeat about their Lumia phones, 41 megapixel cameras and the staff were very approachable in their stripy Marimekko shirts and funny hats.

In one of the quieter affairs, the Nokia Research Centre demoed an indoor positioning system that promises 30 centimere accuracy and presumably lands in a Bluetooth standard in the near future, enabling a range of user experience scenarios for malls, airports and alike. Affordable Asha phones and Nokia Life for emerging markets were featured as well.

Aside from phones, there were a number of smart TV upstarts. We saw a few demos built on old versions of Android, where a phone interface jumped on the screen as soon as the user leaves the home screen. A more captivating demo came from the Korean company Neo Mtel, who showed off a UI with lots of lively widgets and affectionate animations. They also had a tablet-based “second screen” to complement the product vision.

Perhaps a little surprisingly, Opera (of the Opera browser fame) showcased a TV platform based on web technologies.

In Hall 7 we also had the pleasure of having Mozilla as our next door neighbours. They had set up a nice lounge where people could try out the latest Firebox browser for Android phone and tablet. The Boot to Gecko initiative had matured into the Open Web Device together with Telefonica, and resulted in a working demo of a phone OS, based entirely on web technologies with APIs to talk to the handset’s camera, sensors and telephony software, for example. It was also interesting to exchange thoughts on open-source design and development with the fine Mozilla employees.

Meanwhile, there were some interesting evolutions in device form factors to be discovered. Samsung exhibited a 10-inch Galaxy Note tablet with Adobe Photoshop Touch and very precise and responsive drawing stylus. With the exception of tactile feedback the experience is closing in on that of pen and paper – and for many, the benefits of digital malleability can outweigh the constraints of analogue tools.

Notepad-sized phones are parallel to this trend. The Galaxy Note phone got a rival from LG’s 5-inch Optimus Vu. Both devices channel the passport-size Moleskine or Muji notepad and flaunt oversized screens and stylus input. To prove the point, Samsung had dragged a bunch of portrait street artists to capture the likenesses of volunteering visitors on these polished pixelslates.

The requirement of pocketability and one-handed use has caused many (starting with Apple) to overlook this emerging form factor, but not everyone keeps their mobiles in their pockets and many use their phones with two hands anyway. It will be interesting to see how the notepad phones fare in the market and what kind of UI patterns will prevail there.

Last, but not least, the Padphone from ASUS is a very interesting play on device convergence and as such resonates with Ubuntu for Android. The Padphone is a smartphone that docks into a tablet shell and instantly becomes a tablet. The tablet with the phone inside can then be docked into a keyboard, turning the device into a laptop. While some clunkiness with the hardware remains, the user interface seems to transition from phone to tablet seamlessly and in a snap. However, there’s less wow in the tablet-to-laptop transition, where just a mouse pointer is added into the mix. Since Android is designed for touch this is no surprise, but there’s some added value in having a physical keyboard for typing.

Amidst all the sensory overload and throughout the four days of congress, the Ubuntu booth felt like an oasis of good vibes all the time. The interest and support from people we encountered was really encouraging and very heartwarming. Hands-on videos from the booth went viral across the internet. Many said that Ubuntu for Android was the highlight of the Mobile World Congress 2012.

Visit the Ubuntu for Android site for more…

Read more
Charline Poirier

Every three months, I conduct benchmark usability testing.  I’m calling these tests ‘benchmark testing’ because the aim of these sessions is to measure our progress towards achieving a great user experience with Ubuntu.  Last testing took place in October 2011.  I am now preparing for testing 12.04 to take place a couple of weeks from now.

When I publish the results of usability testing, I get many questions about my process.  So I have thought that the best way to explain how I approach usability is to take you along the preparation and execution of my benchmark testing. Over the next month, I will take you, step by step through my process, from recruiting participants, to writing a test protocol to conducting and analysing usability sessions and writing up results.  This will afford you the possibility of ‘accompanying me’, so to speak, and of conducting usability in parallel, if you are so inclined.

For this post, I walk through the first stage of any testing: recruiting participants.

Recruiting

This is a crucial part of any successful and meaningful testing.  Some argue that just anyone you can get hold of will do.  This attitude, in my view, puts the software before the people who will use it, and carries the implicit assumption that software, by its very nature, is usable. But the simple fact, which we actually all realise, is that it isn’t. Take music players, for instance.  The challenge for this type of software is to fit into the lives of people who want to listen to music.  It doesn’t have to work well for those who don’t listen to music but who are, for instance, heavily into photo editing.  In a word, testing your software with your grandmother or your partner might not provide all the feedback you need to create a user-friendly product if they are not engaged in the activities your software is meant to facilitate.

So, the basic idea is:  in preparing the testing, recruit the right people. The type of participants you work with will determine the quality and reliability of the results you get.

There are some basic rules for writing a screener questionnaire.

Rule 1:  Recruit according to your testing goals

Is your goal to test, for instance, adoption: that is, are you going to assess how new users respond to your software the first time they encounter it and how delighted they are by it?  Alternatively, is your goal to test learning: do you want to assess how easily a novice can figure out how to use your software and how they progress over time? Or are you really interested in expert usage:  do you want to assess how performative your software is in a specific context of use involving expert tasks?  There are, of course, other scenarios as well.  The point here is that you need to be clear about your goal before you begin.

With Unity, we have 2 basic goals:  1) adoption:  we want to know how easy to use and attractive Unity is to someone who has not encountered it before; and 2) expert usage:  we want to know how performative Unity is with highly competent users who are fairly familiar with it.

Given these very different goals, I will need to conduct 2 different user testing sessions with different recruiting screeners or questionnaires, and different protocols.

In this blog, I concentrate on my first project, to test for adoption.

Rule 2:  Know your software

You need to review your software carefully:  you need to (1) identify the main purpose of the software and the activities or tasks that it is meant to facilitate; and (2) identify where you think potential usability weaknesses are.

When I prepare a usability test, and before I even think about recruiting participants, I spend a significant amount of time trying out the software, and even more time discussing with the designers and developers their own concerns.  From this evaluation of the usefulness and usability of the software, I’m able to sketch a profile of participants.  Bear in mind that, given my goals as set out above, the participants will need to be able to use the software right away even if they’ve never used Ubuntu, since I am not testing for learning.

Given what Unity aims to allow users to do, we need to confirm (or not) in the testing that Unity users can easily get set up for and can conduct at least the following activities:

  • writing, saving, printing documents
  • finding, opening applications
  • listening to music
  • watching a movie
  • managing and editing photos
  • customising their computer: organising icons and short-cuts and changing setting
  • browsing the internet
  • communicating

Additionally, the OS should make it easy for users to:

  • multi task
  • navigate and use special features like alt-tab
  • be aware of what’s going on with their computer
  • create short-cuts
  • understand icons, notifications and generally the visual language

In this instance, I want as well to test the new features we have designed since 11.10

Given my goals, my recruitment screener should be written in a way that will provide me with participants who engage in these activities on a regular basis.

Rule 3: Make sure you have an appropriate number of participants, with an appropriate range of expertise, with appropriately different experiences

I’ve often heard it said that all you need is a handful of participants – for example, 5 will do.  While this may be true for very specific testing, when your participants come from a homogeneous group (for example, cardiologists, for testing a piece of cardiology software), it is not true generally.  Much more often, software is meant to be used by a variety of people who have differing goals, and differing relevant experience and contexts of use.

You need to take these into account for 2 purposes: 1) to be able to test the usefulness and appropriateness of the software for different users; and 2) to be able to assess the reasons and origins of any usability problem that you find – these can be explained by comparing differences between users. A usability problem will have a different design solution if it is created by a user’s lack of expertise than if it is created by a shortcoming of the software that stumped all user groups.  It will also help rate the severity of the discovered problems.

Some of the factors a competent recruiting will take into account are:

Different levels of expertise: for example, in the case of software for photo-editing, you probably need to assess the ease of use for people who have been editing their photos for more than 5 years, and for those who have been editing for less than 1 year.  Expertise can be reflected in the length of time they have been engaged in the activity and also in the complexity of their activities.  You may want to recruit people who do basic editing, like eliminating red-eye; and then, to compare their use of your software to the use by people who do special effects, montages, presentations and the like.  This way, you get feedback on a wide range of the software’s features and functionalities.

Different kinds of uses:  potential users will have different needs and different potential uses for the software.  For example, if the software is healthcare related, it may well be used by doctors, nurses, radiologists – and sometimes even patients.  It is useful, when considering recruiting, to include participants from these various professions and other walks of life, so that you will be able to determine how well your software serves the range of needs, processes and work conditions represented by the likely (range of) users.

Different operating systems:  you may want to select participants who use, at least, Windows, Mac and Ubuntu. Users who are new to Ubuntu have acquired habits and expectations from using another OS. These habits and expectations become with time equated with ease of use for these users because of their familiarity.  Recruiting participants with different habits and expectations will help you to understand the impact of these expectations as well as receptivity to innovation.

Recruiting your participants with precision will allow you to understand the usability of your software in a complex and holistic way and will dictate more innovative and effective design solutions.

Keep in mind, however, that the more diverse the kinds of persons who you envisage will be primary users for the software are, the larger the number of participants you will need.  You should recruit at the very least 5 similar participants per group – for instance, in the healthcare example, at least 5 doctors, 5 nurses, and 5 patients.

A few more things to consider explicitly putting into your questionnaire/screener, particularly if you are writing it for a recruiting firm:

It is advisable to have a mix of male and female participants;

Participants from different age groups often have different experiences with technologies, and so you should include a good mix of ages;

The perceived level of comfort with a computer can also help the moderator understand the participant’s context of use.  A question about how participants assess themselves as computer users can very often be helpful;

You should always add a general open question to your screener to judge the degree of facility with which the potential participant expresses ideas and points of view.  The moderator is dependent on the participant to express, in a quite short amount of time, the immediate experience of using the software.  Consequently, being able to understand the participant quickly and precisely is vital to obtaining rich and reliable data.  The individual who makes the recruitment needs to be able to evaluate the communication proficiency of the potential participant.

Rule 4: Observe the basics of writing the recruitment screener

The most reliable way to obtain the desired participants is to get them to describe their behaviours rather than relying on their judgment when they respond to the screening questionnaire.  For example, if you want a participant who has a good experience in photography, instead of formulating your questions as:

Question:  Do you have extensive experience in photography?

Choice of answers:

Yes
No

You should formulate your question in a way to make sure the person has some level of familiarity with photography:

Question:  During the last 6 months I have taken:
Choice of answers:
between 20 and 50 photos a month [Recruit]
Less than 20 photos a month [Reject]

By matching potential participants to actual behaviours, you can make a reasonable guess, for example, here, that someone who has been taking 50 photos every months in the last 6 months is indeed competent in photography, whereas when you rely on the person’s own assessment that they have extensive experience, you can’t know for sure that they are using the same criteria as you do to evaluate themselves.

Your screener should be created from a succession of questions representing a reasonable measure of familiarity and competence with the tasks you will test in your software.

That said, your screener should not be too long, as the recruitment agency personnel will probably spend no more than 10 minutes to qualify candidates they are speaking with on the phone.  At the same time though, you need to ensure that you cover questions about all the key tasks that you will ask participants to perform during the test.

Summing up

Let me sum up the basics I’ve just covered by showing you the requirements I have in my screener for testing the ease of use of Unity by the general public user, not necessarily familiar with Ubuntu. They include that:

  1. there should be a mix of males and females;
  2. there should be a variety of ages;
  3. participants should not have participated in more than 5 market research efforts (because people who regularly participate in market research might not be as candid as others would be);
  4. there should be a mix of Windows, Mac and Ubuntu users;
  5. participants should:
    • have broadband at home (being an indicator of interest in and use of computer during personal time);
    • spend 10 hours or more per week on computer for personal reasons (which shows engagement with activities on computer);
    • be comfortable with the computer, or be a techy user;
    • use 2 monitors on a daily basis (I want to test our new multi-monitor design) to carry out a variety of activities online (part of the designs I want to test relate to managing documents, photos, music, and so forth and  I want my participants to be familiar with these activities already);
    • use alt-tab to navigate between applications and documents (another feature I intend to test for usability);
    • have a general interest in technologies (I want to make sure that their attitude towards new technologies is positive, so they are open naturally to our design);
    • express ideas and thoughts clearly.

 

In closing let me add that testing with friends and relatives is very difficult at many levels.  First, you can’t ask all the questions you need to:  there are many ‘common understandings’ that prevent the moderator from asking ‘basic/evident/challenging’ questions that might need to be asked to participants. Second, participants might not be sincere or candid about their experience:  someone who knows you and understands your commitment to the software might not express what they think, and they may not identify problems they are experiencing and thus, they might minimise the impact of a usability issue or even take the blame for it.  Third, of course, they might not fit as precisely as they should the recruitment screener.

Feel free to use this screener to recruit participants if you would like to conduct testing sessions along with the ones I will be doing at Canonical.

In a couple of days, I will write a blog post about writing the protocol for this round of testing  – which is the next step you’ll need to take while you’re waiting for participants to be recruited.

Read more
Paul Sladen

Ispravka ?irili?nog fonta «?» «? ? ? ?» «? ? ? ?»!
????? ?????? ??? «?»:

Amélie Bonet at Dalton Maag has drawn up redesigns for a number of the Cyrillic and Serbian/Balkans characters that weren’t as clear, or ideal as they could have been. If you use these characters, please help give feedback about whether the suggested improvements are sufficient, or whether they could be improved further. For Greek, there is also a proposed fix to monospace Gamma:

Many appreciations to those who reported the original bugs about the Ubuntu Font Family. We have tried to follow up to the original reports at Blog Russia and at Opennet.ru (thank you to ????????? ???????? and also all those on the #ubuntu-rs IRC channel.

Please comment directly on the bug reports. You can use your own language if it is easier (eg. Russian, Serbian, English, Greek…). ??????? ???????!

Read more
Paul Sladen

UbuntuBetaArabicF in print,

A beta of Ubuntu Font Family Arabic, in print as part of the testing and debugging process for the Arabic coverage. The Arabic script support will cover Arabic, Urdu, Pashto, Kashmiri and other written languages using the base Arabic script.

The magazine is an intriguing tri-lingual production published by the Cultural Office of Saudi Arabia in Germany with the layout prepared by Professor Rayan Abdullah’s team at Markenbau. The magazine starts with German and English articles using Latin script at one cover (reading left-to-right) and articles written in Arabic from the other cover (reading right-to-left).

Ubuntu Arabic, now has horizontal, instead of diagonal dots

Following on from the recent posts about adding Kashmiri/Pashto ringed characters and the Arabic update from the start of 2011, the most significant change to highlight is the that the diagonal dots (?i???m / ??????) have been changed to a horizontal layout.

The resulting arrangement is now closer to an equilateral triangle, and the dots closer to a circle.

(Thank you to Abdallah, Björn Ali Göransson, Chamfay, Masoud, Muhammad Negm, Nizarus, Reda Lazr and others who each took the time to comment and give feedback about the earlier diagonal dot angle).

Read more
Charline Poirier

Recently we hired an external consultant to compare the usability of 2 email clients: Thunderbird and Evolution. I have taken some highlights from the report to compose this blog.

Setting of the usability session

The sessions took place in early June at the Canonical Office in London. Thirty participants were recruited. All of them used at least 2 email clients.
Methodology

One email account was set up in preparation for the sessions; all users were asked to use this account’s details to set up the email package. Days prior to the testing, messages were sent to this account and it was also subscribed to a mailing list, in order to ensure a realistic influx of emails to this Inbox.

Half of the participants interacted with Thunderbird and the other half with Evolution; Thunderbird 5.0 build 1 and Evolution 3.1.2 were used on a desktop machine using Ubuntu 11.10.

During each 60 minute session, participants were asked to:

  • set up the relevant email package for an existing account;
  • create a signature;
  • compose an email and change font colour;
  • manage emails in folders;
  • locate a specific email containing an attachment (sent prior to the session);
  • respond to an email containing an attachment and send the latter back to the sender;
  • create a contact list and send a message using contacts.

Highlights of Report

What Participants Liked

Thunderbird

  • Straightforward and familiar set-up
  • One-click Add to Address Book feature in email preview and window
  • Window/message tabbing system
  • Familiar, intuitive language
  • Quick search that meets expectations

Evolution

  • Useful guiding steps in mail configuration assistant
  • Intuitive contextual menu option to Add Contact in email preview and window
  • Menu items easily accessed as alternative to button shortcuts

Both

  • Both were seen as having a familiar layout
  • Both met expectations in terms of generally intuitive access to contextual menus
  • Both provided intuitive access to search facility


Where Participants Struggled

Thunderbird

  • Confusion over search use (severe)

Users were confused by the existence of two search fields, often opting for the All messages search box as they intuitively saw this as highest in the hierarchy. This choice often resulted in disappointment when users did not expect to be taken away from the folder they were searching in; in addition, they found the search results confusing and inefficient, reporting that they expected the item they were searching for to be more easily visible.

Participants were further frustrated by the fact that if they had misspelled an entry or their search returned no results, they would not be aware of this until taken to the search results tab, which they saw as a frustrating waste.

  • Difficulty locating and managing folders (severe)

The majority of participants successfully created folders, either by right-clicking in the folder area or using the New functionality in the top menu. However, most of these users were unable to easily locate the folder created or move once they had located it. This was due to them not realising they were creating subfolders; once subfolders had been created, unless that folder already had folders within it and it was expanded, users did not notice the expand icon next to the folder and bypassed it. Finally, once they found the created folder, users attempted to relocate them to the desired place; majority of users failed in doing this successfully.

  • Difficulty personalising message text (mild)

More than half of users struggled finding the function to change font colour; the majority looked for this in the message text toolbar, bypassing the colour functionality because they expected the icon to look differently. Users eventually found this with the use of tooltips, but not after looking through all toolbar and top menu options first. Participants voiced the issue for this to be the icon being in black and therefore too subtle; they mentioned preferring a more colourful icon or one resembling a palette.

  • Unclear server options (mild)

Participants reported liking the apparent ease of setting up but most were confused by the server options provided in the second, and final, step. About half reported that they would navigate away from the window and research more into their options, with the rest either ignoring this message and go with the IMAP option already selected or choosing the POP option which caused them some issues finding emails later on. The majority of users reported preferring helpful information and guidance on the options provided in the set-up screen, in order to avoid navigating away or uncertainty.

  • Difficulty finding and personalising signature (mild)

The majority of participants were unsure where to find the signature functionality, with the majority expecting it to be either in the main toolbar, message toolbar or Message menu section. Most participants were unable to find this feature on their own without looking up help or reporting that they would ask a friend for help.


Evolution

  • Longwinded, unexpected set-up (severe)

Despite appreciating the guiding steps outlined in the mail configuration assistant, the majority of participants reported feeling that this process was unexpectedly too long and found the options provided very technical and confusing. For the majority of users, this culminated just at the second step, where they thought they were being asked to retrieve backed-up files, rather than being offered option to set up this feature. Some users failed at this point, reporting that they were confused by this and would revert to using the current email set-up they had.

  • Locating account email (severe)

The majority of participants had difficulty initially locating account email due to the email folders displaying an Inbox and an account-specific email section. Most participants did not notice that the account email area was collapsed and were confused about the ‘Inbox’ shown at the top of the folder list not showing any messages. Users attempted to view the account Inbox by selecting Send/Receive and then clicking through all folders available. Eventually users noticed the email account folder with the expand icon next to it and accessed the account folders that way. This experience caused great alarm in these users, particularly as it was at the beginning of interaction with the system; as a result, many reported loss of trust in the package and considering ending its use.

  • Unintuitive message search (severe)

As discussed, search was intuitively used by participants to quickly find a required message in a large Inbox. Many participants failed finding the required search results because they carried out a search unaware that they had selected an irrelevant folder. This resulted in no results being returned and users being confused because they had expected to be able to search all folders.

  • Once email opened, difficulty getting back to Inbox (severe)

Half of participants naturally double-clicked to read email in more detail; however, in Evolution, this resulted in email opening up over the main Inbox window, hiding the email list. Participants were confused by this and struggled to get back to the message list; majority reported looking for a button or link to Inbox and were extremely weary of closing the window down (either via the buttons or menu items) because they were nervous about potentially closing down the entire email application.

  • Inability to personalise email text (severe)

Almost all participants were unable to personalise message text in Evolution; they expected access to font colour to be available along with the other font toolbar options and entirely bypassed the HTML option. One participant selected HTML and still missed the font colour option. Participants were very disappointed by the lack of this feature and looked at all toolbar and top menu options for access to this.

  • Despite long set-up, confusion over lack of password request (mild)

In addition to finding the Evolution email set-up longwinded, participants were confused why this had not asked them for account password details. The majority saw this as a frustrating time waster, particularly as they were asked for this separately, once their email had been set up.

  • Difficulty locating and managing folders (mild)

As with Thunderbird, the majority of participants successfully created folders, but here mainly by using the Folder functionality in the top menu. All participants expected to be able to create a folder by right-clicking in the folder area and only a few right-clicked on a folder to look for this functionality. Despite being able to create the folders using the top menu, users were disappointed with the lack of quicker access to this feature in the folder area (either by right-clicking or with the use of a button) or a button in one of the top toolbars.


Usability Issues Common to Both

  • Difficulty finding and personalising signature (mild)

As with Thunderbird, the majority of participants were unsure where to find the signature functionality, with the majority expecting it to be either in the main toolbar, message toolbar or Message menu section. When users found the Signature option in the message toolbar, they were very frustrated that this did not provide a shortcut to signature creation. Most participants were unable to find this feature on their own without looking up help or reporting that they would ask a friend for help.

When they were taken to the signature feature, users were again frustrated at the fact they could not find a font editing facility, despite the interface looking like it should allow for this.

Conclusions

As discussed, users gave both positive and negative feedback on their interactions with Thunderbird and Evolution, with Thunderbird consistently being perceived by users as easier to use and fit for purpose than Evolution.

Thunderbird was widely liked for the perceived straightforward set-up and facilitated access to contact save, search and open windows features. In addition, users commented on the familiar language used in the application.

However, participants encountered a few severe issues which tarred their image of the system. These consisted of extreme, at time show stopping, difficulty with:

  • successfully understanding and choosing the relevant search field to use;
  • locating and managing the preferred location of folders.

Finally, these users encountered some lack of clarity over server options in set-up and frustration at the inability to easily format email text.

Participants who interacted with Evolution liked the guiding steps in the mail configuration assistance, the intuitive contextual mention options to add contacts and the ability to easily access alternatives to button shortcuts in the menu.

Users reported multiple severe issues around the Evolution set-up, locating account email, message search use, formatting email text and navigating back to the Inbox. All of these issues were so major that users encountering them reported lack of trust in the Evolution package and a reluctance to continue its use.

One major fact to keep in mind is that, especially as the majority of participants were new to Ubuntu, they saw the email application they used as a representative of the operating system. This is particularly pertinent to the email system that is a system default and it should be ensured that, before either one of these products is chosen for this purpose, the severe issues reported here are addressed.

Read more