Why Focus Group Style Usability Sessions Fail

When we conduct a task-based usability study, we are occasionally asked if more than one person can participate in the session in order to “maximize” the information and feedback we can get out of a one-hour session. While this may seem like an effective approach, we find that it negatively impacts our findings and drastically reduces the quality and quantity of feedback we get.

In this article, we’re going to explore why having more than one participant in a task-based user research session is ultimately more trouble than it is worth and produces inconsistent results.

One vs. Many

There can be some confusion about the differences between a usability study and a focus group.  Both types of research involve recruiting participants and asking them questions. Each method has its strengths and weaknesses but are not interchangeable.  

We find that focus groups are great for:

• Gathering a broad range of opinions about and overall direction to go 
• Learning about the features and functionality that is most important to a key audience group
• Asking questions about the value of a process
• Learning more about the unique attributes of an audience
• Identifying opportunities for a competitive advantage

One-on-One user research sessions are effective for:

• Uncovering the reasons why there is low engagement with a particular piece of content or functionality
• Assessing the value of a process and where improvements can be made in order to make it better
• Observing the behavior patterns of specific user types and getting answers to questions in order to help build a persona or user profile
• Evaluating body language and facial expressions in order to better understand how an experience causes someone to feel

Focus groups tend to be about a group discussion and they can provide tremendous value when trying to assess whether an idea or concept provides value. However, in our experience, having more than one participant in the same session during task-based user research (usability testing) dilutes the process and creates an environment of competition, derailment, and distraction. 

We recommend conducting a one-on-one user research study to help our clients uncover barriers and to observe and document end-user behavior. During a one-on-one session, the participant is given plenty of time to complete the task naturally without feeling rushed by others. They can formulate their own opinions and recommendations without outside influence.  

Multi-user sessions are also an unrealistic representation about how we experience an application in the real world. Typically, a computer or mobile device has one input device (mouse, keyboard etc.) and one user. You rarely, if ever, use an online application as a group using the same computer or mobile device. The goal of conducting effective one-on-one user research is to most accurately simulate the process someone would use in real life. 

An example of a failed one-on-one research session

Recently, we conducted an online session where the respondent insisted on having their entire team participate. We gave it a shot and here is what happened.

They delayed the start of the session while their team gathered
Since their decision maker was running late, it delayed the start of our session by 10 minutes while we waited for that person to show up. This caused us to rush the test plan and skip certain tasks and key questions.

Two of the 5-6 people on the call were the most opinionated
This meant that the others in the group said very little and just jumped on the bandwagon. They participated by providing redundant feedback that only reiterated what the most vocal and opinionated people said.

They verbally attacked the facilitator 
Since there were so many people in the room they did not listen to the reasons why the research was being conducted or what the goals were. They ignored the facilitator when they were informed of the fact that the version of the application was a prototype with no database behind it and continually complained about it not having accurate data. They completely misunderstood the point of the research and would not be redirected to the tasks the facilitator prompted them to do.

There was a lot of whispering and side conversations 
This created distractions and derailment from the test plan and the facilitator did not know who to address with their questions. This took the focus away from the research objectives. It also diluted the responses we got because they all then needed to weigh-in and make sure the 2 that had the strongest opinions agreed with their position. It was impossible to discern between the individuals and we never really got to understand how the experience caused each of them to feel.  

We had to abandon the test plan out of the gate
They decided right away that they were going to give us their opinions regardless of the questions or parameters we established in order to maintain consistency in the research and feedback. They jumped around, talked over the facilitator, and asked questions that were irrelevant to the objectives of the research.
 

A one-on-one session creates an environment that allows a participant the ability to focus on the tasks they’ve been asked to complete with minimal distractions. It gives us a realistic interpretation of their own actions, candid answers to the questions we ask, and the ability to dig into and uncover the reasons why a process or piece of functionality is confusing or successful.

Contact us to learn more about how User Research can help improve your site or application by talking directly to your target audience.