The 5 Most Common User Research Mistakes

At Crux Collaborative, user research is the foundation of our design work and approach. We designed and built a user research lab into our offices because it is as integral to our work as having desks or conference rooms.

Because we specialize in complex and transactional interfaces for regulated industries, we cannot rely on our own opinions or point of view to know we’ve arrived and the right solution. It’s key that we work with clients who understand that the best results come about as a result of collaboration- not just between the client team and our team, but also with end-users.

In the last decade, we logged over 2000 hours of interview time, talked to hundreds of individuals in person, on the phone, and online. We’ve incorporated user research in most of our projects (and felt the pain when we didn’t) and we’ve learned a ton about what works well and what causes user research to fail or not deliver the type of results it could.

Here are the 5 most common mistakes clients and project teams make when it comes to user research:

1. Cutting the user research to save time and money

It’s not uncommon for a client who is looking to shorten the timeline and reduce the budget to ask: Can we cut the user research?

It seems like a simple and logical thing to do to eliminate a few weeks and a few thousand dollars. Unfortunately, in almost every single instance, it yields the opposite result.

We’ve observed over and over again how projects have suffered by moving into the implementation phase with confusing, complex, unnecessary functionality that fails when released, results in customer calls and complaints, or simply goes untouched and unused.

In the end, knowing that a feature or content type is deemed irrelevant or unnecessary by end-users saves organizations thousands of dollars in development costs and results in simpler, easier to use products.

Organizations are often reticent to conduct user research because they don’t have enough time and budget – not recognizing that a research study often results in saving a project both of these things.

The amount of cost and effort to fix and re-release a product or key functionality is significantly higher than the time and money it takes to make sure that the functionality is valid, needed, and usable in the first place.

2. Conducting research for the wrong reason(s)

We’ve seen it over and over again; research conducted to win a political battle or “prove a point” ultimately serves no one – not the organization, and certainly not the end-user.

If a department or group has made some decisions with the website that you don’t care for, commissioning a user research study to show the company how wrong they are is probably not the right approach.

One of the things we often tell clients is that user research is best used to improve products and services, not to prove or disprove whether a group has succeeded or failed.

User research provides maximum value when it is being used to optimize an experience, validate whether an approach to a problem makes sense to end-users, or determine whether a key piece of functionality can be easily found and accessed.

Ultimately, it comes down to the fact that research is an information-gathering tool and should not be used as a blunt instrument to win political battles.

3. Having broad or unclear research objectives

If you’ve ever talked to someone who says something like “We did a usability study but we really didn’t learn anything” it is because their project had poorly defined research objectives.

The point of research is to learn something. If you can’t identify what it is you want to learn at the outset, it’s likely the research will be ineffective.

It’s like the difference between going to the refrigerator to grab a specific item or opening the fridge and staring into it, hoping something jumps out at you. The likelihood for a successful outcome when you don’t know what you’re looking for is low.

When conducting research, always start with a few well-defined questions that should be answered by the end of the study. Research objectives should be specific.

For usability studies, we often aim to define research objectives that can be answered with a yes/no. For example: Can users locate the registration functionality using their mobile device?

The research can only provide clear results if the objectives are specific. Saying you want to learn whether end-users enjoy a shopping process is too broad and is actually comprised of many smaller objectives: Can they browse or search for the product they seek? Can they compare products? Can they access the product details they need? Etc. etc.

Any one of a number of smaller objectives can impact user perception of an entire process or even of an entire company or brand. Therefore, it’s important to have granular research objectives that will help you clearly identify (and resolve) key pain points.

4. Using the wrong research method

The method of user research that most people are familiar with is the focus group. The method of user research that most user experience professionals love-to-hate is… you guessed it, the focus group.

Focus groups have often been used when a different research method would have better served the needs of the project. Using the wrong tool to accomplish a task can be a frustrating endeavor, like trying to use a butter knife to thinly slice a ripe tomato.

Focus groups are great when you’re trying to gather a lot of ideas from a group of people who are familiar with the topic or product they are discussing. They are terrible for validating whether a design approach makes sense.

Similarly, if you’re trying to understand how to improve a complex application that is used for a specific task, a phone interview where you ask a participant to talk to you about their experience and frustrations won’t yield the meaningful information that a contextual interview will.

However, if you’re trying to understand whether a new brand resonates with the intended audience, an 8 person qualitative study is not going to result in enough data to confidently make an assessment.

Unfortunately, using the wrong research tool for the project can do lasting damage to people’s idea of user research as a whole. It’s not uncommon for us to hear clients say, “We did focus groups and didn’t get any information about the reasons that the call center got slammed with after release.” That’s because they used the wrong research approach.

5. Not having the project and client teams observe the research and participate in the analysis of findings

When it comes to user research for complex, interactive systems asking a single individual to analyze and provide recommendations is a recipe for failure.

An online interface for a complex system is the result of hundreds of decisions made by teams of people. Not having the individuals and disciplines who made the decisions leading to the version being tested observe the research hampers the collective understanding of the results and the ability to affect meaningful change. Not having the product owners or leadership observe end-users and watch with their own eyes as person after person either delights in, or struggles with, a given feature does the project a great disservice.

A finding that may appear insignificant to someone unfamiliar with the business could be missed. Similarly, an individual unfamiliar with the technology or data constraints is unlikely to suggest a meaningful solution to problem that is encountered.

The worst type of research reports are the ones that identify a laundry list of issues without providing any information about what specifically could be done to improve or fix the issue. Even worse, reports that suggest solutions that are either needlessly difficult or impossible due to technical, legal, or regulatory constraints.

Having full representation from the project team during the research allows everyone to discuss the issues that are observed, discuss how they can be addressed given the project constraints, and come to a shared consensus around solutions and next steps that teams can actually implement.

Making sure to include end-users in the design process is a key component of any user-centered design.

If you’re wondering about what type of user research would be the best fit for your next project, we’d love to help.