10 mistakes to avoid in usability testing

Better usability testing for websites and apps: the top 10 mistakes to avoid

Done well, usability testing enables organisations to validate ideas and designs for websites and applications with real-world users, helping increase engagement, satisfaction and conversions. But executing a successful user testing strategy brings with it numerous challenges and potential hazards. Think about your redesign and conversion optimisation projects. Are you at risk of committing any common testing mistakes?

Introduction to the value of usability testing

Research from the IEEE has found that fixing problems with software once development’s already begun is twice as expensive as resolving them before development commences.

Conducting lab-based usability testing that provides valuable insight into how users perceive and interact with your site or application before your project goes live can, therefore, deliver significant returns; in fact, it’s been found that for every dollar a company invests to increase usability it receives $10-$100 in benefits, and wins customer satisfaction and loyalty.

Ten most common mistakes in usability testing

However, successfully crafting, conducting and analysing these tests requires careful planning, and there are a number of challenges that you may stumble across. Here are some of the most common mistakes people fall victim to, along with tips to ensure you get the most out of any testing activity you undertake.

• 1. Recruiting unsuitable participants

Even if your testing set-up is perfect, if you’re not testing with the right people your results won’t be as useful as they might be. It’s therefore important to research participants in advance, and build a test base that reflects your audience (for example, if visitors to your website are largely female, splitting test subjects 50/50 according to gender may not be representative).

• 2. Not testing early and often during the project lifecycle

The overheads associated with a lab setup mean that often people wait until the very end of a project to conduct testing sessions, which can lead to costly issues being uncovered when they could have been much more easily solved at an early stage.

Testing throughout the project lifecycle is therefore recommended and there are, in fact, a wide variety of methods available, meaning that whatever your requirements there’s an option to suit you. Here’s a brief comparison of some of the most popular approaches:

A combination of qualitative and quantitative testing throughout the project lifecycle has been found to produce the best results, so think about how you will build this into your own project. For example, a recent project I was involved with tested:

• 3. Following too rigid a test plan
  

Preparing a test plan is vital to keep track of sessions and, perhaps more importantly, guarantee that all important areas have been covered during the testing.

However, you shouldn’t be afraid to go off-course when necessary, as you may find that the answers you get when users are acting freely provide interesting insights you hadn’t considered previously.

Ranking your questions in order of importance can help you manage this unplanned activity while ensuring that critical topics are addressed.

• 4. Not rehearsing your setup

It may sounds obvious but it’s crucial to check your equipment well in advance of your sessions to confirm that everything works as it should, from the devices participants will be using to the recording software you have in place, and even basics such as power points and internet connections.

• 5. Using a one-way mirror

Many usability labs feature one-way mirrors that enable observers to monitor sessions without being seen by participants. However if the user knows they are being watched this can lead to unnatural behaviours and, subsequently, have a negative effect on your results.

For this reason, transmitting audio and visual feeds to a different room is often a better option, and one that also caters to remote teams anywhere in the world.

• 6. Not meeting participants in reception

It’s important to remember that participants may be nervous about the session ahead, particularly if they’ve never been involved in laboratory testing before and are unsure about what’s required. Greeting them as soon as they arrive and making them feel at home helps create a relaxed and comfortable environment to support an open and insightful session.

• 7. Asking leading questions

When creating your test plan, make sure you review all questions for any possible bias, as otherwise you may find that you’re actually leading the user to certain answers or actions based on what you expect (or want) to happen. Also be mindful of body language, as this can provide you with an even greater understanding of user responses.

• 8. Interrupting the participant

While test facilitators should support the participant and, as covered above, explore potentially valuable new areas of enquiry, the main purpose of usability testing is to gather feedback from representative end-users. As such, make sure you give them enough time to express their thoughts, and avoid interrupting until you’re sure they are finished.

• 9. Undertaking two roles in a testing session

Lab-based usability testing sessions involve both a facilitator who guides the user through the tasks, and an observer responsible for making notes and conducting the initial analysis.

If a single consultant is performing both these roles then critical responses could be overlooked, so be sure to allocate appropriate time and budget to testing.

• 10. Not considering external influences

Ideally laboratory usability testing should be performed in facilities purpose-built for the task, as this provides a controlled environment that’s tailored for best results. No matter where your sessions are conducted, though, external factors such as noise, interruptions and building work may impact your activity, but preparing well in advance can help you manage these issues.

Find out more
So, that’s an introduction to some of the issues you should aim to avoid when embarking on a usability testing project, in order to unlock the full value from every one of your sessions and so improve the quality of your finished products and services. If you want find out more about any of the pitfalls discussed in this post, download our white paper 'Ten Common Usability Testing Mistakes', and if you think I’ve missed anything important please leave your comments below.

Thanks to Gavin Harris for sharing their advice and opinions in this post. Gavin Harris is a User Experience Consultant at Box UK You can follow him on Twitter or connect on LinkedIn.