The Importance of User Testing
Apple and Microsoft haven’t always hit it out of the park with their product releases.
Microsoft’s Windows 8 was a user experience nightmare (and financial disaster), while Apple’s Newton certainly wasn’t the iPhone we received several years later.
So what went wrong?
A common oversight. In the rush of excitement to release new product features or the next big thing, designers, product managers, and upper management have a tendency to believe that the “build it and they will come” fallacy doesn’t apply to them. And in believing this, a critical opportunity is missed: product validation through user testing.
Facts trump opinions. User testing is a set of methodologies, practices, and tools that help UX designers discover problems early on in the product development lifecycle. Problem discovery can help avoid financial missteps and lead to improved product rollout.
User testing is performed at all stages of the design process, and the most impactful results are achieved using prototypes due to the immediate value given back to the design team. Much of that value is delivered via qualitative and quantitative feedback allowing designers to obtain a more balanced picture of what works and doesn’t work for users.
User Testing vs. Usability Testing
User testing is sometimes referred to as usability testing, but there is a distinction. User testing is an overarching term that includes usability testing. It asks questions such as, “Does the product fill a need?” or, “Does it solve a problem?”
Usability testing is distinct from user testing in a few ways. The goal of usability testing is to find out if a product performs to user expectations. Usability testing asks, “Can users use the product?” This is associated with heuristic analysis. Heuristic evaluations seek to identify common usability problems of user interfaces.
In both cases, prototypes and mockups can be used for testing; there are no rules that dictate when to use one or the other. However, with user testing in particular, prototypes are the preferred testing method.
The Difference Between User Testing and Usability Testing
|User Testing||Usability Testing|
|Do users need my app?||Can users use my app?|
There are two types of user tests that designers and user researchers can use:
- Comparative – Comparing two versions of a product or website against each other. Also used to compare different features of a product or site.
- Explorative – This method is used to test a range of different services where users are given realistic scenarios to complete. This helps highlight any gaps in the market that can be used to advantage and illustrates where to focus design efforts.
The Benefits of User Testing Prototypes
Performing user tests with prototypes provides valuable feedback that designers can use early in the design process to avoid costly mistakes. A major benefit to prototypes is the ability to determine what users consider to be the product’s purpose.
For example, using a five-second test, users are shown a page or product feature for five seconds and then asked to describe it. This is a particularly relevant test given people’s time constraints and the need to understand in a short period of time what a product can or cannot do for them.
Here are a few more potential benefits prototypes offer when used for user testing:
- Fail fast methodology – It’s better to fail fast and iterate through ideas than it is to waste resources, time, and finances.
- Cost reductions – It saves a lot of money up front by finding out if the product features are a good fit for a particular user segment.
- Unbiased perspective – Users providing feedback both qualitatively and quantitatively gives an objective view.
- Uncovering hidden opportunities – When users give feedback, they also share a lot of information which can be turned into opportunities for new features down the road.
User Testing Tools: The Rise and Fall of Focus Groups
Things were very different before the internet gave us the ability to perform user tests from anywhere, at any time. Generally attributed to sociologist Robert Merton in 1946, the genesis of user testing can be traced back to focus groups.
Merton was a pioneer in the area of group studies. He conducted noteworthy research on the effects of radio broadcasts, as well as army training and morale-boosting films. A few years later, several successful marketing campaigns from Chrysler, Ivory Soap, Barbie, and Betty Crocker were due in part to the golden age of the focus group.
Back then, focus groups and individual user testing had to be conducted in large rooms with expensive video and sound equipment, two-way mirrors, and specialized rooms called “usability labs.” Apple had one, so did IBM as well as many others.
By the end of the 1990s, focus groups had fallen out of favor. Broadband internet spread like wildfire, and researchers started to rely on video-based interactions for user testing. The rest, as they say, is history.
In 2019, designers have a more robust toolset for user testing prototypes. Here are a few popular tools used at different stages of the design process:
- UserTesting – A tool that allows UX designers to gather insights by drawing upon a pool of testers who have registered and are willing to give feedback. The site claims to deliver insights within hours of setting up a test. This allows designers to gather objective feedback without having to solicit users on their own.
- Bugsee – Mainly used for apps, Bugsee allows users to report bugs/crashes and provides a video of the app in question. This is best used for ongoing user testing.
- Appsee – Provides user session recordings while users test out mobile apps and websites. It allows UX designers to watch every user action and understand exactly how they use an app.
- Loop11 – Integrates with many of the best prototyping tools on the market, and they’re constantly adding more integrations as they become available. Loop11 works with Axure, JustInMind, and InVision. UX designers and product teams can see clickstreams, videos, heatmaps, and more.
- Lookback.io – A powerful qualitative interviewing tool. It allows designers to interview users in realtime and take notes as they are completing tasks. Additionally, it offers a feature that allows users to complete tasks in their own time. Lookback.io is a modern approach to a focus group.
- Hotjar – Provides quantitative user testing. The tool allows you to view a heatmap of user activities on your website. It reports on where they go, how often, and obtains insights into areas that don’t work.
- Usabilla – Allows designers to see real-time feedback on a website or app with live user feedback. With the use of a feedback button that is integrated right where they need it to be, users can give feedback anywhere at any time.
- Userlytics – Allows designers to create online and offline user tests for websites, apps, and prototypes. They gather both qualitative and quantitative data and have a global participant panel for designers who want to define targeted personas.
- Maze – A user testing platform specific to prototypes. It gathers actionable insights from users and works with InVision, Marvel, and Sketch. Though it will cost extra, Maze can provide testers from a pool of over 50,000 users.
User Testing Tips
User testing, unlike usability testing, typically seeks to validate the need for a product, and in doing so, forges a different path. We aren’t asking, “Can users use this product?” as with a test such as a heuristic evaluation. Designers and user researchers use methods such as online user testing and focus groups to find out if the product solves an actual problem.
Here are a few best practices and tips for performing user tests:
- Use five participants. It is recommended that studies use five participants for qualitative testing. Why five? According to experts, this is the number needed to detect approximately 85% of problems.
- Write great questions. This is as much an art as it is a science. Avoid leading questions and use “why” and “how” questions as much as possible.
- Perform dry run studies. A great idea is to run one session first, evaluate, and then scale to a larger group of participants. This allows for adjustments to the testing duration, questions, and/or the way in which the test is conducted.
Is it possible to avoid product disasters like Microsoft’s Windows 8 or Apple’s failed Newton? Yes—by ensuring that user testing is a part of design early in the process. To many, user testing and usability testing share no clear distinction, but it is important to understand their difference.
User testing validates the need for a product, while usability testing validates whether the product is usable. There are several useful tools that can be utilized to perform user testing, but the one thing in common is the use of prototypes as a mechanism for gathering feedback. User testing prototypes save time, money, and resources and will ultimately result in better, more successful products.
• • •
Further reading on the Toptal Design Blog:
Understanding the basics
User testing is performed either in person or online using one of several user testing tools. The designer sets up a series of questions, tasks, or activities and users perform them while being observed.
User testing in UX is finding out if a product fills a need. This can be accomplished using online focus groups, user testing tools and apps, and direct observation.
User testing is important so that the right products reach the right people at the right time and place. It is also important in order to avoid product failure.
A user interview is best conducted either in person or via video. It’s important to read nonverbal language and record observations objectively.
We need usability testing to tell us whether a user can actually use a product or app. Usability testing can identify costly problems early on in the product life cycle.