Although usability testing has existed for a couple of decades, many product development teams still do not apply it in practice. Considering usability testing as an add-on to product development, some executives we meet are in favour of outsourcing usability testing or skipping it altogether. UX training, conferences and workshops are increasingly popular however, many assumptions around usability testing are prevalent in the form of myths and misconceptions. These myths extend beyond those in senior management to designers, project managers and other stakeholders. In this article, we debunk fourteen such myths to clarify that usability testing has huge added value and it saves costs in product development.
Myth 1: Usability Testing is super expensive
Usability testing turns out to be expensive in a few cases: You need to set up a usability lab, when you completely outsource usability testing to an external agency, or when you need to work with a large sample size and combine complex research methods. Conducting one large usability test that includes many different segments of target audience and requires collecting both qualitative and quantitative data would require a higher number of resources and therefore naturally be more expensive.
On the contrary, if you do several small usability tests to surface top usability problems, the costs will be lower. Further reduction in costs and increase in speed of usability tests execution can be achieved by using a remote user testing tool (like UXArmy’s usability testing). On UXArmy, sample size of let’s 10 of your testers costs you less than USD 100. That is less than one tenth of what the latest iPhone costs!
Myth 2: Usability Testing Is Artificial
Usability testing is generally conducted in a controlled environment. In case of in-depth usability evaluation, there is someone watching. Since that environment may have some biases to responses, usability testing appears artificial.
In reality, to some extent usability testing can be artificial. However, with well thought out task scenarios based on users’ real usage of yours or similar product, usability testing can be very effective to get natural feedback. As an effective alternative, researchers and non-researchers use unmoderated usability testing. An unmoderated testing enables users to perform the Tasks in their natural environment, thereby significantly reducing many biases.
Myth 3: User will tell us what the product should do
Usability testing has the opportunity to get the product up close to the users and therefore some members on the project team may believe that issues pointed out by the users must be immediately addressed.
While each issue found during the usability testing must be given it’s due consideration, the project team must decide whether the issue needs to be addressed and sent into solutioning. User preferences are not solutions, so the final call on the implementation must be taken by taking into account the other considerations like requirement evaluation, project planning, etc.
Myth 4: Usability testing is only done in the lab
We have often heard people telling us they are unable to do usability testing because their organization doesn’t have a usability lab.
Actually, usability testing is best done in the actual user environment. Due to COVD-19 pandemic going out and observing users using their product in their natural environment may not be an option. Remote usability testing is a great way of reaching out to the right participants in a safe manner. You need to use a reliable and efficient research recruitment service to reach out to the right participants.
Myth 5: Usability Testing is different than UX testing
Usability testing only uncovers usability problems and metrics like time to task completion, success/ failure, etc. User Experience of my product will need to be tested separately.
That’s not true! Usability testing lets you find feedback on first impressions, emotions and feelings that a product interaction invokes in people. Essentially, it’s the user experience which is being evaluated with specialised focus on effectiveness, efficiency. User satisfaction is a part of usability testing.
Myth 6: Minimum N participants are a must
You may have often heard statements like – test usability with only 5 participants, at least 30 participants are needed for a usability test to be considered valid, calculate sample size based on size of your target audience, etc.
There is no golden formula to calculate a right number of participants for usability testing. Finding as many usability problems, as early as possible, is the most important goal. As far as the number of respondents are concerned, “it depends”. It depends upon how significant the result from usability testing is going to be on your product. During product development, multiple iterations with 3 to 5 users give a lot of value whereas right after the product launch, a higher sample size can cover different types of context of product usage and provide you usability metrics with proper statistical significance. The RITE method suggests conducting multiple iterations of usability testing with 1 or 3 participants.
Myth 7: UX testing doesn’t bring much value
If we are talking in terms of data this is not true as through usability testing we get both qualitative and quantitative data. Usability Metrics helps us in identifying the problem areas and Features like screen recording help us in understanding that cause. For example by analyzing task completion time and navigation paths we can see where the user is getting stuck and then we can analyze the screen recording and voice feedback.
Therefore we can say that usability testing gives us qualitative as well as quantitative data to help us in understanding our customer’s needs and problems. We can find a feasible solution to solve these problems and create a better experience for the customers.
Myth 8: Repeat the same participants who took the earlier User Research
‘If we get the same participants as the last research we can find the improvements’ – that statement may sound very logical.
The truth! Participants are people like yourself. There is a lot happening in their lives. They may have been exposed to different experiences and they may not even remember what they saw the previous time they tried using your product. It is recommended to test with different participants each time because that’d more likely expose your product to people from a diverse set of behaviours and preferences.
Myth 9: Include as much as possible to make most of the opportunity
It’s normal to see a project team trying to stuff as many scenarios for users to test within a single test.
Getting a user to provide feedback is a valuable opportunity. However, users are people like you and me and the quality of feedback is affected by several factors like time of the day, energy levels and even mood of the user. Too many scenarios in a single sitting might exhaust the user and feedback from a fatigued mind is not only biased, it’s dangerous to product success. We at UXArmy do not recommend a usability testing session to exceed 60 minutes for an in-depth interview and 25 minutes in an unmoderated setting. If you have too many questions to ask, prioritise and ask the most important ones.
Myth 10: We pay incentives so they are here
We are paying them so they must follow our time of availability – this way of thinking is common as usability testing becomes common in the internet-tech heavy economy.
It is actually wrong to think that way about your users! You must consider it a favour that a person is willing to give priceless feedback in making your product better. Participation is usability testing is not a Service. While there are people who want to take up every user testing opportunity out there, these individuals are usually a very small segment of the population. Most people are leading their busy lives and it’s advisable to adjust to their availability as much as you can. Participants with a relaxed mind would be able to respond with more thoughtful feedback.
Myth 11: Always make Design changes during testing
Oh user found a usability problem? Let’s change the design before the next user takes the usability test.
Actually making changes on-the-go could be a premature decision. One user is just one user. For the next user, the same usability problem may not be a usability problem at all. Unless you are in very early phases of product design and using a method like RITE, it is not recommended to change the design after every one or two users. You may introduce new versions of the design if the time permits.
Myth 12: Usability Testing is time consuming
While developing a new product, it’s hard to fit the need for user testing into a tight project schedule.
Studies have shown that usability testing ends up saving 50% of time and effort spent in the rework of a product. It’s better to do some user testing than none at all. One of the biggest time consuming factors in usability testing is the logistics for example, finding and recruiting suitable testers. Setting up means to find testers, then targeting the group needed and finalizing everything to conduct the test can take weeks, especially when you have to schedule for a suitable time slot and set everything up (from logistical to technical). With building the initial test and documenting all the user test findings, are these spent weeks even worth it?
Fortunately, remote user testing has become more popular lately, as there is no need to find participants for studies conducted physically (especially in our current state). Research Recruitment has been made more accessible as well. At UXArmy, we already have a growing pool of thousands of people who are willing to take part in usability testing. The large tester pool consists of testers from all around the world and all have different backgrounds. You are also welcome to join as a Usability tester.
Myth 13: You must be a professional to conduct a user test
It’s easy for companies to put off user testing if they feel that only a specialist can conduct UX testing. When you think like this, it can be challenging to prioritise user testing at the early stages of your business, especially if you feel you lack the budget for it.
This is another example of a user testing myth as a business feels that they can only conduct user testing after they have on-boarded a UX professional. ​​Remote Usability testing tools can be used by anyone who knows the user goals which a product aims to fulfil. These platforms allow you to set up multiple tasks / scenarios and integrate it with survey questions. As an outcome of these tests you get usability metrics, screen recording and verbal feedback of people who test your product.
Myth 14: Usability Testing Is same as UAT
Testing with users, YES! That is what we do on the UAT (User acceptance Test). Does it meet their needs? Do they like it?
Usability testing is not UAT. Even though both testing involves the users, in UAT the test is against functionality to ensure they work according to their requirements. UAT is key in discovering bugs that constrain the functions. UAT can also uncover performance issues.
Usability testing is to check if the product meets user expectations. Usability testing may classify a task scenario as pass even if the functionality is not as per product specifications.
Conclusion
Usability testing is a powerful method to use when developing new products. The availability of remote usability testing tools make the process easier to execute from a budget as well as time perspective. Some organizations are on their journey to incorporate usability testing in their product development process. It is normal for the project team to have misconceptions in this transitional state. UX is subjective and lacks standards, leaving room for debates as to how much an organization must adopt or when to conduct usability testing. With digital transformation, business executives and product managers are more likely to understand that product market fit can not be achieved if usability evaluations are done after completion of product development.
