Episode 007: When the Prototype is a Fail

Hosted by:
With guest: Larisa Brandt
May 19, 2017

Mahtab   Okay, welcome to episode seven of The Crux of It. Episode seven actually ended up becoming two episodes so, here's part one. I'm Mahtab Rezai, I'm the Principal & CEO of Crux Collaborative. We are a user experience consulting firm specializing in regulated industries. Today I am joined by my friend, my former colleague, and current client, Larisa- who is the Manager of Web User Experience at Portico Benefit Services. Thank you for joining me. Hello.
Larisa   Hello, Mahtab.
Mahtab   So, we're gonna be talking about something that is not often talked about, and something that is a big fear of client and project teams alike, which is what happens when you create a prototype of a concept, you get into a ulab to test it, and the test goes terribly. So, sounds like fun, doesn't it?
Larisa   It does.
Mahtab   It's great, yeah.
Larisa   It's what we live for.
Mahtab   Yeah, so let's get into it. So, first of all: First, and foremost, I want to say that fear that it might go poorly is not a good reason to not do user research.
Larisa   Absolutely, that is why we do it. I think there's folks out there that sort of have this utopian idea that you're going to the lab to prove everything is perfect, and, if that were the case, we wouldn't do it.
Mahtab   Yeah why would you do that? Maybe if you needed endless amounts of validation.
Larisa   Right.
Mahtab   But the whole point of user research is to find the things that aren't working.
Larisa   Exactly, and sometimes they're little things.
Mahtab   Yeah.
Larisa   And sometimes they're huge things.
Mahtab   Yes.
Larisa   And they are almost always some things that you didn't see coming.
Mahtab   Yes, and I think people, before they get into an actual research study, think I'm being a little hyperbolic, or that I'm exaggerating when I say "every single time I learn something". But honestly, every single time I learn something.
Larisa   Absolutely. Again, it's why we do it.
Mahtab   Yes, yes, good point, it's why we do it. So, when we think about, particularly about design step user research, a design step study where you have an idea, you create a prototype of it and you test it. Those are not meant to be a moment to prove or disprove. They are absolutely meant to improve. So, if you're trying to use it for a political reason, if you're trying to use it to win a turf war with somebody else, or to say "This idea was stupid", or see it works- a design step study is not the place to do that.
Larisa   No.
Mahtab   Especially in regulated industries, a lot of times it's not like, I think sometimes in other industries or other types of user experiences, there's sort of like this endless supply of candidate users, right? If you're just talking about moms of kids under five who might shop at Target, that's a very different audience than: retirees of this one union who have this one pension, who need to be able to understand how to get their distributions. Or, this type of surgeon implanting this very specific type of medical device or, a patient who has a specific condition who needs that. So I think in regulated industries and in the type of work we do, you don't have an endless supply of audience people to pull from, it's a limited set. So you also have to be really thoughtful, about when and how you're asking them for their opinion, and their time. And, the teams are even more limited.
Larisa   Right.
Mahtab   And the team's time.
Larisa   Right, and as a team member, you can only see the same thing so many times before you sort of become immune to it, and I think that sort of fatigue can exhaust your attention, maybe?
Mahtab   Absolutely.
Larisa   And your ability to solve new problems in creative or collaborative ways because you've just kind of ... It's like having you're nose pushed up against the glass too far and you can't see what's going on anymore.
Mahtab   Yeah, and I think it can result in people just throwing their hands up and being like "This is just broken".
Larisa   Yep. We're just gonna have to hope some people manage that part for us.
Mahtab   Yeah. How well does that hope go?
Larisa   That hope usually results in, in our case, a more expensive recovery of some sort or, a phone call or, you know, it's much less self service then ...
Mahtab   Yeah, than you'd want it to be, yeah.
Larisa   You pay for that somewhere.
Mahtab   That's what I was just gonna say, you don't get away from paying for it.
Larisa   Exactly.
Mahtab   You absolutely do pay for it, it's just depending on whether you're paying for it during the design process-
Larisa   Right.
Mahtab   Or whether the call center pays for it.
Larisa   Exactly, you're passing the buck somewhere in the system.
Mahtab   Absolutely.
Larisa   By not doing the right stuff up front.
Mahtab   That's such a key point.
Larisa   It's funny cause the right stuff up front is actually really easy to do.
Mahtab   And much cheaper in the long run.
Larisa   Much cheaper in the long run.
Mahtab   I don't know why, why? I think we've been doing this for over 20 years, and it's still a conversation we're having.
Larisa   But yet it seems like the most obvious thing in the world, to me.
Mahtab   Yes, me too, to me too. All right so, another thing, because I think you have a really interesting perspective, we both do, in that we've both had prototypes that have come out of testing with really minimal changes, and we have had prototypes that… yeah Larisa just made an explosion motion with her hand. Yeah, I was gonna say "crash and burn", just FAIL significantly, publicly, profoundly, throughout the course of the study.

So, what are some themes, or things that we're thinking about when, let's talk about like when it just all seems to be going super well. What about that?
Larisa   I think it's just the skeptic in me that, when it comes out of the lab going super well, I'm waiting for the other shoe to drop. Like, there must be something, there must of ... We didn't get all the right users, or we missed a scenario or, there's gotta be some environmental thing that we aren't looking for ...
Mahtab   Sure.
Larisa   Because I'm always worried that there's something else out there that we didn't see, but I think it's just because I'm constantly looking for the improvement, and I don't think that I think you’re ever not improving something.
Mahtab   Sure, sure, and have you had a scenario where it tested really well and then the other shoe dropped, or is that just more like a fear that doesn't pan out?
Larisa   I think you're probably right, it's probably a fear that doesn't pan out. If it's happening I'm not aware of it, so it's not a huge problem.
Mahtab   No that's been my experience as well, every now and then it's all magic and it works and there's just this like "really?"
Larisa   Too good to be true.
Mahtab   Yeah, but then it turns out, yeah-
Larisa   We actually do know what we're doing and-
Mahtab   On occasion, on occasion.
Larisa   Things improve through the process if you let the process take it's motion.
Mahtab   And so what are the things that have contributed to those ones where everything pans out?
Larisa   Early and often collaboration, whether it's with your team or your users. I think that there are a lot of ways to get concepts and things in front of your users, even just in parts, early on. Some people would call them more guerrilla tactics but, I've never seen them backfire on me.
Mahtab   Yes.
Larisa   And I'm sure that happens but, even if you just take a part of a concept and get it out there and talk with some people about it or show them or, whatever the process is, that pans out to your benefit later.
Mahtab   Absolutely, absolutely. Showing something in concept I think is always valuable.

Another thing I think that really helps is helping the team have the right context for what happens during research, because if you have a team that's not familiar with design step research and they think that they're just gonna go in and come out of the day patting themselves on the back- then they have a miserable day and they are in the back room trying to defend their decisions and saying things like: "Oh we didn't know this!" or "This isn't the case!", and all sorts of excuses and reasons and, I think, somebody just shared this quote with me recently and they said: "You can't defend your ideas and learn at the same time. You're either in defense mode or you're in learning mode". And I think you want the team to be in learning mode. So the degree to which you can say to people: "We're gonna go in, we're gonna find things that aren't working, and we're all gonna work together to figure out how to fix them".

When you let them know that's what's going to happen instead of something that might happen if things go poorly no, that's exactly what should happen, as you've said earlier on "that's why we do it".
Larisa   Exactly, it's not personal.
Mahtab   Yes.
Larisa   It's never personal.
Mahtab   Yes.
Larisa   And you have to detach yourself from that in order to improve.
Mahtab   Yeah, a lot of times for more traditional research there's this idea- and it makes sense, when you think of how research originated, that it was someone validating something. But we're not doing FDA drug trials here, we are literally trying to figure out if a particular aspect of an interface is working. So this idea that was more prevalent in traditional research where you had one person design it, and another person to test it, to validate it- that doesn't necessarily hold true when it comes to that -and in fact I think it's such an integral part of the process to have the person, or people, who are the design team as part of the research team as well.
Larisa   Totally agree. Having been on both sides of the glass, I can't imagine not being on both sides of the glass. I sort of actually have a hard time understanding - I know there are organizations who have research groups and you kind of do your work in your design and you throw that over to the research group and then you just take the report they give you and make those changes. I can't imagine not being a part of the process. It's enlightening, and it inspires my creativity.
Mahtab   Yeah, yeah it's such a key part. I also think it keeps you engaged particularly as a member of an in-house team. As a consultant I, from project to project to project they're completely different, areas, topics, industries, vertical - I mean they're just all over the place. But when you work for a single organization and you have for a number of years, so, the inspiration and the variety I think comes from the different parts of the process.
Larisa   Yeah.
Mahtab   If you just had your portion where it comes here and you put this one bolt on, and then it moves down the assembly line.
Larisa   Yeah.
Mahtab   That's not a way to stay engaged.
Larisa   No.
Mahtab   Particularly as a user experience professional.
Larisa   And it gives you the ability to constantly improve and evolve. To constantly hone your ability to have empathy for the people that you're building for, because you just are constantly building this idea of who they are and what they need and how they behave and ...
Mahtab   Yes.
Larisa   At the end of the day by the time you're done getting something up to a point where it's testable, you're not objective anymore and now you need to see somebody else do it but -
Mahtab   Absolutely.
Larisa   You're sort of also building that trove of insights.
Mahtab   Yes.

Okay that was the end of part one of my conversation with Larisa. Thank you for listening, you can find us on SoundCloud, iTunes, Google Play, Stitcher and on Bye.


Hosted by: Mahtab Rezai
Principal & CEO

Mahtab has spent nearly two decades as a user experience designer, researcher, strategist, leader, and mentor. She has designed user experiences for companies ranging from startups to the Fortune 50.


With guest: Larisa Brandt
Manager Website User Experience at Portico Benefit Services

Larisa brings decades of experience to helping companies large an small solve their user experience challenges. At Portico Benefit Services, she leads a team in creating functional web sites and applications by balancing client, user, and system requirements, using user research, information architecture,  and her stellar interaction & visual design skills.
View Larisa’s LinkedIn profile


Let us know your name and email address and we’ll send you our newsletter.

First Name

Last Name

Email Address



Thank you!