“You Only Need to Test with 5 Users” – the statement which so many people swear by! Why and how did 5 become the magic number for usability testing? The Common Industry Format (CIF, part of the U.S. Department of Commerce) for Usability Test Reports advises a minimum of eight participants! And, there are a few more widely accepted variations.
It is true that it takes 5 or 8 users (provided they are from your target audience) to test your application in order to identify a good majority of usability problems. However with the more commonly (mis)quoted 5-user rule, there are some caveats and it’s important for people who research, to clearly understand those.
Let’s revert back to the caveats in a bit. The question that the above discussion begs clarity on is “How many people to test with?”.
Short answer: It depends.
“Depends”? … on what?
That is one of the struggles UX Researchers have because there is no fixed and correct answer.
A number of factors come into play when deciding sample size. These could be one or more of the following questions that may help to decide the sample size for usability testing:
A factor that has been out of consideration in the above discussion is about how the usability study is being conducted. A selection between remote or in-person moderated usability testing may not affect the sample size significantly.
However, selecting an unmoderated usability study calls for a slightly larger sample size. To understand this better, the unmoderated remote usability studies are performed without the supervision of a moderator. These studies make use of a usability testing software or user research platform on which the item to be evaluated is uploaded, users taking these studies at their convenient time. While the usability feedback gathered from this method has less bias due to a natural context of use, a few factors to be taken into account. Researchers may end up rejecting a small percentage of responses due to technological limitations, distracted participants or hard-to-comprehend voice feedback. Based on our experience with thousands of usability tests conducted on the UXArmy platform, we propose that the sample size for unmoderated remote usability testing must be 5 – 8% larger than the same study when conducted under supervision.
Back to demystifying the 5 user folklore, Jakob Nielsen is often quoted with his advice that “Five participants will discover over 85% of the problems”. Taking one step at a time, the NN group website clarifies that the 5-users statement was applicable to a specific context.
The Nielsen Norman Group website recommends applying statistical significance to calculate sample sizes while doing quantitative usability studies. For qualitative usability studies, there are a number of assumptions and a recommendation for doing iterative usability testing with 5 users. It is assumed that the nature of the usability study is qualitative and a number of follow-up usability testing iterations shall be made to thrash out the remaining usability problems in a Design.
The bottom line is, it takes more than 5 users in case your product usability is already good. It’s true that a larger sample size would be needed to find those hard-to-discover usability problems. So if usability testing is a practice at an organization and has been happening for the interface/product in question throughout the development, a larger sample size would be needed towards launch to ensure the remaining usability problems are found and if needed, fixed before the product release.
More participants don’t always mean that the research will be of a higher quality. Recruiting the correct participants in your usability study can meaningfully optimize your sample size without affecting the results. That is why if you’re finding it difficult to recruit high-quality participants, we recommend tapping the help of research recruitment experts. By using a UX Research platform with a dedicated user panel, you can remove the hassle of finding the right participants by yourself. UXArmy has a large pool of high-quality participants you can use for your usability testing needs. Visit uxarmy.com to learn more about UXArmy usability testing platform and our user panel known as the UserAdvocate community.