Canonical Voices

Posts tagged with 'usability testing'

Tingting Zhao

In the previous post, we talked about how to design effective user testing tasks to evaluate the usability of an interface. This post continues with this topic by highlighting a number of key strategies that you may need to use when conducting a formative user testing, whose main aim is to identify usability problems and propose design solutions, rather than compare quantitative metrics (summative testing), eg. task completion time and mouse clicks. It is unlikely that the prepared task script could be strictly applied without any changes, since the testing situation tends to be dynamic and often unpredictable. To get useful data, you need to be able to manipulate your task script with flexibilities, while also maintaining consistency.

Changing task orders during testing

Normally, to avoid order effect, the issuing of the tasks should be randomised for each participant. Order effect refers to the effect in which the tasks are presented, which can affect the results, specifically: users may perform better in later tasks as their familiarity with the interface increases; or users may perform worse as they become more fatigued. However, as discussed in the previous post, the tasks are often contextual and dependent on each other, so you need to carefully consider which tasks could be shuffled. For example, it is a good practice to mark on the script to indicate dependent tasks, so that you know these tasks should not be reordered and separated from each other. In other words, the dependent tasks must always be moved together. It is worth noting that the randomisation of task orders may not always be possible for a testing, for example, when the tasks are procedurally related, such as a testing focusing on payment flow.

Sometimes you may need to change the task orders by considering their levels of difficulty. This is useful in the following two scenarios: when you notice a participant appears to be slightly nervous before the testing has started, provide a simple first task to put him/her at ease; or when you observe a participant has failed to solve several tasks in a row, provide one or two easy tasks to reduce the frustration and stress, and boost confidence.

Another type of changing task order is made in response to users’ solicited goals that are associated with a coming task. For example, in one phone testing, after a participant checked the battery level, s/he spontaneously expressed a desire to know if there was a way to switch off some running apps to save battery. In this case, we jumped to the task of closing running apps, rather than waiting until later. This makes the testing feel more natural.

Remove tasks during testing

There are typically two situations that require you to skip tasks:

  • Time restriction

  • Questions being answered with previous tasks

Time restriction: user testing normally has a time limit. Participants are paid for certain lengths of time. Ideally, all the tasks should be carried out by all the participants. However, sometimes they take longer to solve tasks. Or you may discover areas that require more time for investigation. In this case, not all the tasks could be performed by a participant within the given time. Subsequently, you need to be able to quickly decide which tasks should be abandoned for this specific participant. There are two ways to approach this:

  • Omit tasks that are less important: it is always useful to prioritise the tasks in terms of their importance – what are the most important areas that have key questions that need to be answered and require feedback; what could be left for the next testing, if not covered this time?

  • Omit tasks that have already received abundant feedback: skip the tasks from which you have already gathered rich and useful information from other participants.

Questions were answered with previous tasks: Sometimes questions associated with a specific task would be answered while a participant was attempting to solve the previous task – in this case, you could skip this task.

In one of our phone testings, we asked a participant to send a text to a non-contact (a plumber). During the task-solving process, s/he decided to save the number to the contact book first and then send a text. In this case, we skipped the task of ‘saving a number to contact book’.

However sometimes you should not skip a task, even if it might seem repetitive. For example, if you want to test the learnability and memorability of a feature, having the participant perform the same task (with slightly different descriptions) for the second time (after a certain time duration) could afford useful insights.

Add tasks during testing

There are three contexts in which you could consider adding tasks:

  • Where the user formulates new goals

  • Re-examinations

  • Giving the user a second chance

The added task must be relevant to the aim of the testing, and should only be included if the testing time permits.

User formulates new goals: you could add tasks based on user-formulated goals in the task solving process.

For example, in one phone testing, one participant wondered if s/he could customise the tiles on the Windows phone’s home screen. We made this an added task for her/him. Adding tasks based on user articulated new goals follows their thought process and make the testing more natural. It also provides opportunities for us to discover new information.

Re-examinations: sometimes the users may succeed in a task accidently, without knowing how s/he did it. In this case, the same task (with a slightly changed description) could be added to re-assess the usability.

For example, in one phone testing, we had one task: “You want to call you sister Lisa to say thank you for the phone”. One participant experienced great difficulties in performing this task, and only completed it after a long time and by accident. In this case, we added another task to re-evaluate the ease of making a phone call:

“Your call is cut off while you are talking to your sister, so now you want to call her again.”

Similarly in the Gallery app testing, where participants managed to add a picture into a selected album accidently, we asked them to add another picture into a different album.

Re-examination allows us to judge accurately the impact of a problem, as well as to understand the learnability of interface – the extent to which users could detect and learn interaction patterns (even by accident), and apply the rules later.

Giving the user a second chance: in the majority of the user testing, participants used the evaluated interface for the first time. It could be very demanding for them to solve the tasks successfully in their first attempt. However, as the testing progresses, participants might discover more things, such as features and interaction patterns (although possibly by accident). Consequently, their knowledge of the interface may increase. In this case, you could give them another chance to solve the task that they failed earlier in the tests. Again, this helps you to test the learnability of the interface, as well as assess the impact of a problem.

For example, in a tablet testing, one participant could not find the music scope earlier in the testing, but later s/he accidentally discovered the video scope. To test if s/he now understood the concept of dash scopes, we asked the participant to find the music scope again after several other tasks.

Change task descriptions (slightly) during testing

Information gathered from pre-testing brief interview and participants’ testing verbal data could often be used to modify the task description slightly to make the task more realistic to the users. This also gives the user the impression that you are an active listener and interested in their comments, which helps to build a rapport with them. The change should be minor and limited to the details of the scenario (not the aim of the task). It is important that the change does not compromise the consistency with other participants’ task descriptions.

For example, in a tablet testing, where we needed to evaluate the discoverability of the HUD in the context of photo editing, we had this task: “You want to do some advanced editing by adjusting the colour of the picture.” One participant commented that s/he often changed pictures to ‘black and white’ effect. In response to this, we changed the task to “You mentioned that you often change a picture to black and white, and now you want to change this picture to ‘black and white’”. The task change here does not change the aim of the task, nor the requirements for solving the task (in this case, the access to the HUD), but it becomes more relatable to the participant.

Another example is from a phone testing. We changed the task of “you want to go to Twitter” to “you want to go to Facebook” after learning the participant uses Facebook but not Twitter. If we continued to request this participant to find Twitter, it would make the testing become artificial, which would result in invalid data. The aim of the task is to evaluate the ease of navigation in finding an app, therefore changing Twitter to Facebook does not change the nature of the task.


This post outlines a number of main strategies you could use to modify your task script to deal with typical situations that may occur in a formative user testing. To sum up:

Changing tasks orders: randomise tasks for each participant if possible, and move the dependent tasks as a whole; consider the difficulties of the task and issue an easy task to start with if you feel participant is nervous, or provide an easy task if participants failed several tasks in a row. Allow them to perform a later task if they verbalise it as a goal/strategy for solving the current task.

Remove tasks: if time is running out with a particular participant, omit certain tasks. This could be tasks with low priorities; tasks that already have enough feedback from other participants; or tasks the participant has already covered while attempting a previous task.

Add tasks: if time permits, allow users to perform a new task if it is a user initiated goal and is relevant to the testing; repeat a task (with slightly different wording and at an appropriate time) if the user succeeds in a task accidently, or has failed this task earlier, or if the aim is to test the learnability of the system.

Change task description: slightly amend the details of the task scenario (not the aim of the task) based on users’ verbal data to make it more relatable and realistic to the user. This will improve the reliability of the data.

If you have other ways to maneuver the tasks during the testing session, or have situations you are unsure about, feel free to share your experience and thoughts.

Read more
Charline Poirier

In preparation for UDS, we conducted usability testing of Unity with general public users.  We are now better informed as to where we should expend further efforts to enhance the experience for users working with Unity.

I summarize my findings below.

Participants Selection

We asked an external recruiting firm to find 15 participants who answered to the following criteria:
1.  Mix of males and females: roughly equal
2.  Age:

  • 7 people between the ages of 18 and 30
  • 6 people between the ages of 30 and 50
  • 2 people 50 or over

3. Employment:  employed full-time or to be full-time student
4. Involvement in marketing research: no such involvement in the last 6 months.
5.  Basic technology:  Each participant needed:

  • To have a broadband connection at home
  • To be a daily internet visitor, staying online for at least 2 hours every day for personal reasons

6.  Online activities:  Participants needed to have done all of the following while online in the last 2 months:

  • Look for information
  • Read comments written by others
  • Read blogs written by others

In addition, they needed to have done at least 4 of the following in the last 2 months:

  • Create or review a social networking profile
  • Post comments or reviews
  • Write a blog and conduct some activities on social networking sites
  • Shop
  • Share photos
  • Play games
  • Download music on a music player.

7.  Participants needed to have a strong interest in technology.
8.  Each participant needed to own at least one portable music device, one mobile phone, and one computer/laptop.

Of the 15 participants recruited, 13 were Windows users, 1 was a Mac user, and 1 used both Windows and Mac.  None of the participants was familiar with Ubuntu.


During one-on-one sessions, Unity was presented on a netbook.  The goal of the sessions was to get participants to experience most of Unity’s features and functionalities.  I introduced the session with the following instruction:  “Imagine this is your new computer.  What do you want to do with it?”  I had songs and pictures available for them to import.  During the 60-minute session, I asked them to do some of the activities they normally do, which included importing songs and photos, engaging in social networking, etc..  However, some of the tasks were tailored to fit the specific interests and knowledge of given participants.  For example, if a participant was a professional musician, I proposed to him or her more tasks related to music management; similarly, if a participant was a student, more of the writing and reporting capabilities of Unity were explored.


What participants liked about Unity

The look and feel:  First impressions of Unity were positive and many comments pertained to its elegance.

One participant said:  “Simple and clear tabs on the side.  I don’t like to be crowded with stuff on the screen.  It looks quite approachable.

The core concept of Unity:  Many participants intuited the new concept underlying Unity’s look and feel and direction.

One participant  said:  “I like the application idea and that it is application-oriented and you can go there and have a quite user-friendly interactivity.  I  like the tabs at the top and when I hover it tells me what it is and it is big and clear, easy to see.”

Workspaces:  Many participants were unfamiliar with the concept of workspaces and were intrigued by its overall potential, and its friendliness.
One participant said:  “What you can do is that you can have different applications in different workspaces.  This is a nice feature.”

Dash:  Participants liked the simplicity of the dash and its look and feel.

Software Centre:  Generally, the software centre was seen as impressive, particularly its large number of available free software.

Usability issues

Performance of Unity

Participants were challenged by various functionalities and conceptual design features of Unity. It is noteworthy that one factor stood out as principally responsible for many usability problems: this was Unity’s slowness and suboptimal responsiveness during testing. The level of performance in this regard significantly impaired the flow of use and the user experience. Testing was done on a mid-range netbook that users were likely to own, and consequently the user experience in this regard was similar to what users would have experienced outside of the usability lab.

Multitasking  – Multitasking on Unity is disconnected and difficult at times

Task flow is interrupted: While working on a task, participants wanted to have all the documents and websites they were using easily available to them.  For them, the task was the unit of organisation of all resources and tools.  Thus, while working on a task, participants expected that Unity would provide them with a representation or visibility of what was available to them and how to easily access what they needed at any given point.  Unity does not, however, make evident the resources and tools users have at their disposal — whether it be multiple documents, programmes or websites. In a word, while Unity relies on users to keep track of the resources they are currently using, users are habituated to relying on the software to “keep track for them,” by making the resources highly visible, for example, by means of tabs. With Unity, resources are hidden from view.

Overall, participants found the navigation between documents to be cumbersome. Often, they looked for a way, a back button or breadcrumb, that would navigate them through their open applications and  documents.  They didn’t find either. As a consequence, they ended up going through the files and folders section to access their open documents. Clicking on the OpenOffice icon to access a document already open was not immediately discoverable.

One participant noted: “[navigation] seem[s] awkward.  I’m not getting around as quickly as I should.  The icons are supposed to tell you but I don’t know what they are.  That is a problem.

Another said: “It seems a bit pedantic to have to go through the application menu to navigate.”

Poor visibility of applications and of windows that are opened: Participants experienced difficulties with the following:

Minimising a document: When participants minimised a document, the document seemed to have disappeared when they expected the mininised document to be shown at the bottom of the screen.  In Unity, there were no traces of the document on the screen and participants could not find it again.

“I’m not entirely sure how to get back into my new document [after minimising it].  [It is] not clear that it opened a new page for it.  I don’t know how to get back to it.  I don’t know, it’s weird.  I expect it to be a tab and be able to switch between the two.”

Knowing which documents are currently opened:  Participants made it clear that they not only wanted to know if their documents were opened, but also wanted to know how many documents or windows were open at any given time.  But they did not always see the white triangle that indicates that a programme is running or that documents are opened.  Consequently, they were not aware of what was available to them.  In short, the general indicator was not enough for their needs for awareness.

Being able to distinguish between similar documents:  In the exposé view, documents are very small and, when 2 documents are similar, participants could not tell which was the one they wanted without opening each one individually.

Identifying a document:  When participants were in the process of writing a document, they looked for the name of the document they were working on in the top menu bar, but couldn’t find it.  They were also looking for such information in the exposé view.

Being aware of multiple open documents or windows they needed to consult or modify: Almost all participants wanted to organise their work by opening several windows and physically positioning them in a way to maintain an awareness of all of them.  For example, one participant opened Facebook and a word document.  She wanted to move the word document so she could expose the top part of Facebook.

“Can you move the window down at all?  I like to have a little bit of space for organisation.  I want to be able to move the window down.  It would be nice to [be able to do that], especially if you want to start working, you want to organise your environment.  You want to feel the sense of relief and organisation before you start working and [the sense] that what will come to you will be manageable.”

Immediate access to documents they currently are working on:  Exposé is not readily discoverable.  The majority of participants could not find the exposé feature.  As it is not a feature they are used to having, they need a prompt to help them explore it.

Exposé does not allow users to work on documents side-by-side.  Participants expected to use the exposé feature to be able to cut and paste between documents directly without having to bring one document up in full screen and then having to navigate to the next document by means of the workspaces icon.  As it is, exposé is useful for viewing only.  Participants would have liked to have had the option to directly navigate and conduct operations (e.g., cut and paste) between documents.

Documents cannot be resized and placed side-by-side:  No participant could find a way to resize his/her Open-Office documents in such a way that they could be placed side-by-side while working on both at once.

Overview of what’s going on: Many participants wished they could have a view of what resides in various parts of their computer, as is facilitated by Windows’s “my computer”.

Back button:  Many participants needed to “go back” to a document or an application that was currently open.  But the back button doesn’t work coherently and seemed to bring participants to arbitrary pages.

“The thing here, backward and forward buttons don’t work.  Back button doesn’t get me back to where I was.  I don’t know where it is taking                 me, I’m frustrated.  The back button is taking me somewhere [else].”

“If you can’t navigate you feel like a fool and you feel ashamed even if you’re alone in your room.”

“I’m struggling to find short-cuts and a back button.  Right now, to me, it feels very clunky to go from one place to another.”

Document management

Open documents: As already indicated briefly, participants didn’t know how many documents they had opened.  The white triangle indicator, meaning that the programme is working or that at least one document is currently open, is too general for users’ needs.

Delete documents:  Participants could not delete existing documents from their files and folders.  They first tried to drag and drop documents in the waste basket; then they right-clicked to see if they had such an option.

Copy and paste:  Copy and paste from one document to another didn’t always work for participants.
“Copy and paste is not going to work.  I’m not impressed, it must work.  I’m one for copying and pasting.”

Lack of feedback and guidance

Unity is often slow, and as a result participants tended to be confused about what was going on. They didn’t know:
1) if the system was slow and would eventually produce the desired effect, or
2) if the system had not registered the participant’s command and as a consequence, they should repeat their action, or even
3) if the system had crashed.

Unity needs to clearly show if it is engaged in a process or not.

“Has it crashed?”

“Let’s make sure I clicked it.  [She opens a document.  She's getting confused navigating between documents and she closes the document without saving it.]  When I close, it should ask me to save it.  This is not good [that it doesn't].  You rely on the computer to save the document or not.”


When entering a search term in the search bar, there is no feedback to show that Unity is indeed engaged in a process.  Many participants were confused by this.

“How do you know this is searching?  Is there any sort of symbol to tell you if it is doing something?  I would like to see if there is a symbol that says the computer is working so I am not clicking around.”

Other issues:  Interactions with the launcher, finding the dash, lack of reference to understanding Ubuntu features

Interactions with the launcher

Adding an icon to the launcher:  Many participants were not able to add a short-cut of an application to the launcher.  They used various strategies but failed:

  1. They tried to drag and drop icons directly from the applications section to the launcher.
  2. They right-clicked, expecting an option to copy.
  3. A few participants noticed that when an application is open, it appears in the launcher, and when they close it, it disappears.  They were confused by this feature, but at the same time perceived it as a clue.  So they tried to drag and drop the icon from the applications section onto the similar icon visible in the launcher.

Changing the order of icons in the launcher:  Most participants failed to reorganise the order of icons in the launcher.  They selected an icon and tried to drag it upwards within the launcher; that is, they did not realise that they needed to remove the icon from the deck first before they could relocate it.

Deleting an icon from the launcher:  Participants did not succeed in deleting an icon from the launcher.  Most selected the icon and then dragged it into the waste basket at the bottom of the launcher.  They did not see that they could remove an icon by dragging it horizontally away from the launcher.

Finding the dash:  The majority of participants who found the dash found it by accident. They were not sure what it was, and didn’t know how they had gotten there if they accidentally had.  As a consequence, they were not able to find it again. The problem for participants here was that the logo is not recognisable to them.  Of course, they were not familiar with Ubuntu; but, equally important, the logo is less visible in size and colour than the icons contained in the launcher.  It took participants a while to see it at all.

“The logo should stand out more.  It would be nice if the icon itself [was] coming at you, more of a three-dimensional thing.

Lack of reference to understanding Ubuntu unique features

Software centre: Almost all participants were surprised when I suggested that they install a game that was not currently installed on their computer.  All of them immediately went to the internet.  When I told them that such games could be found on Ubuntu, they were confused and many didn’t know where to find them.  They went to the applications section, where there remembered a section called “installed” software.  They expected either a section below this one with a list of uninstalled software or (at the very least) a link to the software centre.

Moreover, most participants expected to be able to click on an icon in the dash, for example ‘music,’ and directly access their albums and songs.  They were a bit surprised to be taken to music applications instead, especially in light of the interactions of the other icons in the launcher, which  are quite different.

“I expect to see music files, names of songs or albums, anything you’ve downloaded.  Titles.”

Familiar short-cuts: Participants who normally used short-cuts to navigate didn’t find the familiar short-cuts and were frustrated.

“There are things that are nice to keep from one OS to another, like right click and short-cuts.  It’s nice to have two possible solutions.”

Changing the wallpaper

Many participants did not succeed in changing their wallpaper because the default screen of appearance was open in full screen by default.  Because of this, the action of immediate wallpaper change, as users select a wallpaper, was not visible to them. After participants had selected a wallpaper, they expected a step to confirm that they wanted their wallpaper to be changed.  Instead they were given  other options that were not relevant to what they were trying to do, and at which they had already (but unbeknownst to them) succeeded.


The search functionalities felt limited to participants. When searching, they didn’t know what the field and scope were that were covered by the search engine they were using.

Grey icons in launcher

Participants thought that the grey icons in the launcher were inactive, especially when they clicked on them and the system was very slow in responding

Read more