Episode 008: Avoiding User Research Fails

Hosted by:
With guest: Larisa Brandt
June 2, 2017

Mahtab   This is episode eight of the Crux of It. The second part of my conversation with Larisa about when user research goes badly. I'm Mahtab Rezai. I'm the principal and CEO of Crux Collaborative. We are a user experience consulting firm specializing in regulated industries. And today, I am joined by my friend, my former colleague and current client, Larisa- who is the manager of web user experience at Portico Benefits Services. Thank you for joining me. Hello.
Larisa   Hello, Mahtab.
Mahtab   Let's talk about avoiding failures. We talked about all the different things that can go wrong, the things that lead into failure with design step research. We talked about the importance of recognizing that you want some stuff to go wrong- that's why you're doing it- you're doing it to learn. But let's talk about some of the things we've observed over the years that help make the best use of everyone's time, of the end user's time, of the team's time, of the budget, of all of it. What are some of those things?
Larisa   I think there's an opportunity at an early point to vet concepts and ideas in different ways versus sitting in the lab in usability and the structured plan environment. It kind of goes back to that, it's a really expensive way to vet an idea if you do it too early.
Mahtab   Yes, absolutely. I think another thing is listening to the pros if you have them. And you and I again, have had this experience- I think both together -and then separately we've had the experience where as the UX folks on the project, we are telling someone: “This is not a good a idea. This isn't going to work, it's going to be confusing.” And they are absolutely insistent on their idea and on their approach and they gotta do it. We make a prototype and we get it to the ulab and very painfully watch it suffer and die a slow and painful death.
Larisa   And occasionally you will come up with some valuable insights from a user to maybe solve that problem differently, but more often than not, I have seen it be the moment where someone goes, "Oh, nevermind. Just get rid of it". Which is kind of where you want it to be going in. It's a political vetting process than an actual: "This is paralyzing". Make it a different thing. Move it out of this concept. Maybe you even just need to spend some time educating your audience about this thing before you begin to try and introduce it to them. So, next year.
Mahtab   I also think sometimes you have to let some people witness the pain and suffering. They're not gonna believe you but they are going to believe user after user struggling.
Larisa   I have certainly met my share of people who have to see it with their own eyes. And that's not wrong necessarily. That's a way to do it. I think when you've been doing it for a long time, it becomes a little exhausting.
Mahtab   Yes. I agree. It just becomes like: “Fine. Why am I even gonna share my opinion in the first place?” The other thing I think that's so important is having the right group of people in the observation room. Both like what we just referenced- the right person. So, if there's a person who's insistent, having them be there to see it fail is key. But also the people who can fix it -because it's so hard after the fact if they haven't seen why something was a failure, or how users struggle with something, to go back to a writer or a development team and try to just say: “It didn't work!” as opposed to having them be there in real time and see the why and the how and the actual experience of it not working.
Larisa   Yep, I think there's no better way to humanize something than to see the humans try to use it.
Mahtab   The other thing I think too, is planning to make the changes. I wanna share the experience: last year, you and I were working on a tool together that was super complex and there was a lot of things going on - a lot of things that were new, a lot of things that were complex, a lot of things we suspected weren't going to work - and there was like a single day of research in which we watched several aspects of this one experience just fail and blow up.
Larisa   Again and again.
Mahtab   Again and again. And it quickly became clear that no one at your organization was going to be comfortable with the idea of just “taking a stab at fixing it” and releasing it because of how profoundly it had failed in the first day of research.

So there was this whole, almost like “emergency” second research effort that needed to happen once the candidate's solution had been developed to solve for the problems we observed in usability- because the team was just not comfortable launching without a second round of research -but it was second unplanned full round of research. And that is not only expensive in terms of money, it's expensive in terms of resources of the team of pushing out other deadlines, working the long hours to sort of do that. So, I think one thing that was really smart that your team did with the launch this year is to plan to have some research, time to make revisions and come up with a proposed solution, and then a second round of research. I also think for design step usability where it's an unknown or it's super complex, that's a really great way to help avoid failures, and to help the team really mitigate the realities- which is we're gonna find stuff that's wrong.
Larisa   Yeah, the other thing that that process has done I think, it has led people into the first rounds of usability with a better open mind like: “Obviously this is iterative and we're going to make changes!” So we aren't having that very red light green light- it all has to be right or we're gonna be paralyzed. It's okay, we can take a breath now. We saw a bunch of really good information, we're gonna turn that around. So yeah, we learned from our mistakes.
Mahtab   Yes, alright and then the other thing I think is for whatever reason, some people are really attached to having a super high number of people.
Larisa   Yeah.
Mahtab   And that actually is so tedious.
Larisa   It is super tedious, I agree.
Mahtab   Speaking as a facilitator, it's hella tedious. And then I'm sure as an observer, watching the same task fail over and over again it's like: “Okay we've got it, thank you.”
Larisa   Right. In all my years doing this, it's still the old school rule: seven people is enough. I have always seen enough of what we're gonna see in seven people. Fourteen people, twenty-one people, thirty people- those first seven people proved out 90 percent of the problems that we were gonna have. That's why having to do two rounds of something also I think becomes tedious for everyone. If you split it up a little, it's more iterative unless all at once.
Mahtab   The other thing I think that's important is to have an out.
Larisa   Which you do really well. I've seen you do it more than once.
Mahtab   Yes. I have had to bail myself out of a terrible research situation on many an occasion.
Larisa   That's why you recruit for more than a seven you really need to see.
Mahtab   Yes, exactly. And since we've done this so many times over the years, we thought a fun thing to do would we talked about our most hilarious user research oopses. Bloopers.
Larisa   My very favorite blooper is I was sitting behind a glass and Mahtab was facilitating and it was set up like a conference room, so there was a polycom phone- and the phone rang and it wouldn't stop ringing, so she had to answer it- and she won a cruise!
Mahtab   Yes. It was so loud too.
Larisa   In the middle of a ulab.
Mahtab   “YOU HAVE WON A CRUISE!”. Yeah, that was one time I heard everyone in the backroom laughing through the glass. That was pretty good.

Another thing I think that we have seen is that we both, I remember one study that we were working on together where the prototype was failing profoundly but it was a really odd user interface behavior that the audience found really novel, and so the task was they had to book a flight, but the way that they were doing it made it feel like they were playing some sort of video game. And they could not book this flight to save their lives, so they were completely failing at the task, but boy were they complimenting the interface. They kept being, "This is great! this is so fun!". And you and I were dying because we thought if you were actually trying to book a flight, you would be so angry right now, but here you are in this research lab getting a hundred dollars to spend 45 minutes chasing your mouse around trying to catch this and it's not working. I think that was one.
Larisa   My great fear was that the client would see that as success and they would actually go forward with this craziness.
Mahtab   There was that. It is always just pretty comical when it happens which is the poor recruiting. We've had some real doozeys.
Larisa   I think sometimes the person who shows up isn't really the person represented on the phone when the recruiter talked to them and you get this person who just is absolutely not the audience for the thing you're testing -and sometimes those people can prove to be okay because they're just really sort of things I think of basic usability like: do you understand buttons and links and things like that. But sometimes they're just conceptually so inappropriate that I've been heard to exclaim: “We need a handbrake for the ulab!”
Mahtab   Yes. And I love that phrase of yours because it's so true. We need a handbrake. We just need an emergency exit that can launch me out of there when someone who is supposed to be there to talk about a financial planning tool looks me right in the face and says that they don't know anything about the money, their husband handles all of that. And I'm like, "Oh!”
Larisa   This is going to be a really challenging conversation.
Mahtab   How did you get here on your time machine?
Larisa   We need to build into our prototypes and a crash and burn button to make it look like the computer froze and…
Mahtab   Get me outta there.
Larisa   Sorry the IT guys at lunch, we're just gonna end this session.
Mahtab   Thanks! Yes indeed. And speaking of getting us out of here, I know that you have an appointment, so I'm so appreciative that you took the time, so thank you for taking the time to chat with me.
Larisa   You're so welcome. Thank you.
Mahtab   And thank you for listening, You can find us on SoundCloud, iTunes, Google Play, Stitcher and on Bye.


Hosted by: Mahtab Rezai
Principal & CEO

Mahtab has spent nearly two decades as a user experience designer, researcher, strategist, leader, and mentor. She has designed user experiences for companies ranging from startups to the Fortune 50.


With guest: Larisa Brandt
Manager Website User Experience at Portico Benefit Services

Larisa brings decades of experience to helping companies large an small solve their user experience challenges. At Portico Benefit Services, she leads a team in creating functional web sites and applications by balancing client, user, and system requirements, using user research, information architecture,  and her stellar interaction & visual design skills.
View Larisa’s LinkedIn profile


Let us know your name and email address and we’ll send you our newsletter.

First Name

Last Name

Email Address



Thank you!