The median number of users tested was The mean number tested was 27 with a standard deviation of 46 min. There are many variables that affect the number of users people test such as the product type, budgets and industry. To attempt to control somewhat for this variability I compared the sample sizes for the 33 respondents who reported conducting both Formative and Summative tests.
So the graphs above mask this interesting relationship. In generalizing results from a sample to the larger population representativeness is more important than sample size. Subscribers to my email newsletter are perhaps more quantitatively focused and that might bias their sample sizes upward.
I looked at two other data sources to get an idea of how representative this data was. In Jim Lewis and I reviewed 97 [pdf] Summative datasets. We found the median number of users per test was 10, ranging from 4 to These numbers are virtually identical to the Summative survey sample presented here.
Sadly, few projects collect such user testing metrics because doing so is expensive: it requires 4 times as many users as simple user testing. Nielsen is of course referring to the difficulty and cost associated with quantitative testing versus qualitative testing. However, technology in the year of his study is not comparable to the technology of With contemporary usability testing platforms such as TryMyUI, the cost-value ratio has never been better for the customer.
When considering how many testers any given usability test should have, we typically recommend 10 for a good idea of where your UX stands, and 20 for a near-certainty, echoing the referenced studies. We have deigned our platform and plans around those numbers, and consistently see these baselines offering customers the insightful data they need. At the end of the day, user experience is too valuable to be overlooked, and even a test with 2 users will give anyone the immediate feedback they need to validate further tests and research.
However, two key flaws characterize much of this research, as follows:. The result of these flaws is that many usability professionals accept this research as a guide, without understanding the associated rationale, context, reasoning, and risks.
When quoting statistics, we must be careful to consider the margin of error for any findings. Therefore, we cannot assume that five participants are sufficient for every usability-testing situation. If there are just five participants, the results may not tell the whole story. We must also question the validity of the statistical methods that produced these numbers. In reality, statistical methods are not free of either opinion or bias because they rely on assumptions of some type.
This means we could draw different conclusions from the same research data if we used different statistical methods. Two important issues for problem-discovery studies are that it can be hard both to define a problem and rank the importance of discovered problems.
According to D. Caulton, problems are often a factor of the interaction between a user and the product, not necessarily a static feature of a user interface.
Not only might a specific problem exist for just some participants, a specific problem might exist for just a single participant on one day, but not on another day. Therefore, it can be very difficult to agree on what actually constitutes a problem. Plus, the ranking of problems is highly subjective. When setting up a usability study, you need to consider the probable mean percentage and what minimal level of problem discovery is necessary—in other words, the average percentage of problems you hope to find, as well as the minimum percentage.
As Table 1 shows, going from 5 to 10 participants greatly increases the expected level of problem discovery, but going from 15 to 20 participants has far less impact. These numbers are very important to understand. We can have fewer participants in a study if undiscovered problems would have a small impact on users, but we must have more participants if the stakes are high—for example, in life- or enterprise-critical situations.
We can also have fewer participants if there will be opportunities to find important problems during a later round of testing. Another important consideration is the complexity of the study itself. Nielsen is often criticized for citing research from simple studies with well-defined tasks.
Clearly, the more complicated the tasks, the more complex the study should be, and the more participants may be necessary. In such a case, it is also necessary to consider training issues.
If the entire target user population will receive exactly the same training, this effectively reduces the study complexity, so you can use fewer participants. Studies to evaluate a prototype of a novel user-interface design often concern the discovery of severe show-stopper problems.
Testing usually reveals such severe errors relatively quickly, so these tests often require fewer participants. So, contrary to popular thinking, there is no one-size-fits-all solution for problem-discovery studies. Context and complexity have a big impact on the appropriate number of participants necessary for such a study. This reality is probably a factor in the diversity of widely cited advice.
For example, both Virzi and Perfetti and Landesman found that the appropriate number of participants for many studies ranges between three and twenty. Turner believes seven participants may be ideal, even for complex studies. At the same time, getting the most out of your 5 test users in a research session requires a specific skillset, so your demeanor and positioning in relation to the test user is key.
No test user wants someone hovering over them , asking leading questions, and appearing hurt when criticism is voiced. Conduct yourself accordingly. No test user will respond candidly in an uncomfortable setting. Put test users at ease by opening with an informal chat, start the meeting by clearly explaining your goals, and continue to speak conversationally with curiosity as your main motivation.
How did you feel about X? Why did Y make you feel Z? More streamlined and polished products may, in fact, have a much lower problem frequency. According to binomial probability, this new frequency would change the number of users necessary to find the problem. As a baseline, 5 is still the golden rule in UX usability testing.
0コメント