| Introduction to Usability Testing |
| Home | Usability | Usability testing | When to test | Methods | Resources |
Usability testing methods
Jakob Nielsen (1993) and Reeves & Hedberg (2001) discuss several methods for collecting usability data. A combination of methods is often useful for improved usability testing. Usability testing methods include:
Observation
In a usability observation a representative user is asked to do authentic tasks
using the prototype or instructional resource. A video camera captures what
is happening on the users screen, their interaction and navigation of the resource,
and the user's "think aloud" comments as they interact with the product.
Additional video cameras can be used to record the users facial expressions, body language, keyboard and mouse, or any other important part of the user environment. Fixed usability labs often have two rooms separated by a one way mirror and multiple video cameras so evaluators and designers can observe the user interacting with the product. A portable usability lab can be used to record what happens on the users screen and the users face and comments. One benefit of a portable usability lab is that it can be transported to the user's actual learning environment such as a classroom or office for probably a more authentic evaluation.
A sample size of five user observations can be very productive. As Nielsen points out "the best results come from testing no more than five users and running as many small tests as you can afford. As you add more and more users, you learn less and less because you will keep seeing the same things again and again. After the fifth user, you are wasting your time by observing the same findings repeatedly but not learning much new" (Nielsen, 2000b).
It is important to emphasize to the user that the goal is to test the site, not the participant. If they have problems doing some of the authentic tasks it is the product's fault and not their fault. The purpose of the usability test is to discover problem areas and potential solutions so the product can be improved. Explain to the user that the test is used to evaluate the product's efficiency, design, layout, organization, navigation, etc. to try to improve it so that it more effectively provides what people need.
To truly test the usability of the product, the evaluator should not explain to the user how the product or prototype works or provide directions or help in doing the tasks. The test user should not have any more information about the product than real users will. It is important to observe how the user really interacts with the product--where the user goes and what they do and think about to complete the tasks--to learn how the product or prototype can best meet the users needs. Some of the most valuable information in a usability observation is determining where the user has problems and what they do to overcome the problem. Encourage the user to ask questions as this gives the evaluator insight into what they are thinking about, but explain from the beginning of the test than in order to have an authentic test the evaluator can not provide help using the product or answer questions about the product until after all the tasks are completed.
Prevent user frustration and help measure efficiency by timing each task completion
and limiting the amount of time the user spends on each task to a set time such
as three minutes. Let the user know that if they are not able to do the task
within the set time then there is a problem with the product and it is not their
fault.
Emphasize that the user needs to be totally honest about the site and that the evaluators will not be offended. Tell the user that the whole process is confidential, give them the approximate amount of time needed for the test (try to keep the entire process to about one hour), and let them know that if at any time they become uncomfortable with the test or find it objectionable they are free to quit.
Provide the test participant with written instructions for each task and clarify the task as needed without giving instructions on how to use the product. Record on video and take notes on the users interactions and thoughts spoken aloud as they use the product to complete the tasks. At the end of the test answer any remaining questions the user may have about how the product functions and discuss any interesting behaviors or spoken thoughts that need clarification for better understanding of the users experience.
Review all the data collected in the observations and look for areas where
users had problems. Look also for patterns in user behavior that may indicate
what parts of the product interface were misunderstood. Then try to determine
ways to change the product to prevent the problems and improve the usability.
The comments and suggestions made by the users during the observation often
provide very good ideas on ways to improve the product.
For more information see the protocol for an usability observation of Research Central or Reeves' & Hedberg's sample user observation protocol (2001, p.148-49).
Think aloud
The think aloud method is often used in conjunction with the usability observation.
The user is instructed to speak aloud their thoughts, decisions they make, any
confusion, their reactions to what happens, etc. while interacting with the
product. Encourage the user to tell the evaluator what they are really thinking
as they use the product. As thinking aloud may not come naturally to many users
the user may need to be prompted "and what are you thinking about now?"
or may need to hear an example of thinking aloud by the evaluator. An interesting
variation reported by Reeves & Hedberg (2001) is having two users use the
prototype together. Since two people are using one system, they have to discuss
their interpretations of the interface and negotiate why they think a certain
path or actions should be taken.
Questionnaires, interviews, focus
groups, and user feedback
Often immediately after a usability testing observation the user will be interviewed,
given a questionnaire, and/or asked for feedback about their experience with
the resource. This is often a good time to clarify the user's thought processes,
discuss where they had difficulty interacting with the instructional resource,
and obtain suggestions for ways to increase the effectiveness and efficiency
of the product.
Questionnaires, interviews, focus groups, and user feedback about the usability of a prototype or product can also be obtained from real users that do not necessarily take part in a usability observation. Groups of authentic users can be gathered to provide feedback on usability or the feedback can be gathered electrically via a web based survey, e-mail, or through a paper based questionnaire.
Logging actual use
Another way to learn how users interact with and navigate
a product is by logging actual use of the product utilizing software that records
user's navigation, keystrokes, time spent on each page, etc. as they interact
with the product. Of course there are ethical issues involved such as privacy
and anonymity.
Since it is easy to collect an overwhelming amount of data with this method it may be wise to focus only on collecting actual use data that can be directly analyzed to evaluate usability. The large amount of data gathered must be organized to be of use.
Reeves & Hedberg (2001) point out additional methods of usability testing:
Heuristic evaluation
In heuristic evaluation human factors evaluators and/or
representative users independently examine a product interface to determine
how well it adheres to a list of heuristics (usability principals). The evaluator
goes through the entire product once to get the overview and a feel for the
interaction then a second time to focus on specific parts of the interface.
Problems are identified and solutions brainstormed then the problems are put
on a priority list with time and cost estimates to correct each problem. Priority
is determined by the frequency and impact of the problem. Heuristic evaluation
is relatively inexpensive and results are quickly available.
Pluralistic walkthrough
The goal of pluralistic walkthrough is to systematically
examine the usability of a product interface from a task-based user-centered
point of view within design limitations. End users, product developers, and
human factors experts independently write down each step they would take to
complete a task using the product then the group discusses their findings with
the user presenting findings first. This method works well with paper prototypes
as well as fully functioning products. Being task based it will often identify
more specific problems than general problems.
Formal usability inspection
In formal usability inspection usability is reviewed by using a task performance
model and heuristics in a context of specific user profiles and goal based scenarios.
This evaluates cognitive processing and behavioral tasks from the evaluators
perspective without end users and often outside of the normal task context.
Empirical methods
When testing for usability with empirical methods a hypothesis is made and objective
measures are determined. Users are tested using the product under controlled
conditions to prove or disprove the hypothesis. This method can establish cause
and effect for a specific problem but can be very time consuming and requires
an evaluator skilled in empirical methods, and a working prototype.
Cognitive walkthroughs
Cognitive walkthroughs evaluate the ease of learning to use a product
through exploration. Using a detailed procedure to simulate the users problem
solving process at each step, evaluators try to imagine users thoughts and actions
as they use the product the first time without any training. The evaluators
must agree on input conditions such as type of user, tasks, and steps to complete
each task. The cognitive walkthrough can be done as a group or individually,
then the collected data is analyzed.
This method can predict problems and cognitive processes, but evaluators must consider the user's background knowledge, goals, and the cognitive complexity of the product. The method can interfere with the interaction and can be difficult for those without training in cognitive psychology. It also tends to focus on just one aspect of usability, ease of learning.
Formal design analysis
Formal design analysis is based on the idea that understanding the requirements
of a task gives insight to understanding behavior. The "Goals, Operators,
Methods, and Selection Rules" (GOMS) model created by Card (Eberts, 1994)
is an example that Reeves & Hedberg (2001, p. 158) describe. Tasks to be
done by an expert user are broken down into:
Algorithms are applied to each design to calculate a number that can be compared across variable designs. Formal design analysis can be done by a single person without end users to identify problems early in the design process. But this method misses important behavior components such as task learning, error behavior, and transfer of learning.