Episode 018: If you’re having participant problems, I feel bad for you son

Hosted by:
With guest: Rebecca Grazzini
April 24, 2018

Mahtab   Hello and welcome to episode 18 of The Crux of It. I'm Mahtab Rezai. I'm the principal and CEO of Crux Collaborative. We are a User Experience Consulting firm specializing in regulated industries. I'm joined today by my colleague, Rebecca Grazzini. Hi Rebecca.
Rebecca   Hi Mahtab.
Mahtab   All right, so today we are going to talk about what happens when the research participant is not participating properly and has gone rogue. So …something that you and I have both experienced on more than one occasion.
Rebecca   Yes.
Mahtab   What does this look like when the participant is not cooperating? How does that show up? What are some ways that shows up?
Rebecca   One of the key ways is someone will come in, and they have a story to tell. It is somewhat tangentially related to what you're doing, but they're going to talk about their relationship to the thing you're studying. For example, if it's a pension, they're going to talk about their retirement, how they got laid off, and how this thing happened, and how this plant closed, et cetera, et cetera.
Mahtab   Yes. Some of the reasons why that happens is because they've had a trauma, so like a layoff, or they had a bad experience. You're trying to talk to them about how they would find a piece of information using a website or how they would maybe transfer some money from one account to the other, but they see a logo of a company. They once had a bad experience with that company, and now they just completely get distracted and want to tell you all about the time that they got screwed over by said company.
Rebecca   Right. Right.
Mahtab   Or they're maybe confused.
Rebecca   Yes. Yes. Everyone comes into these sessions. They are coming here to earn money. They want to do a good job, and they just don't want to look dumb.
Mahtab   Yes, absolutely. That's one thing, so storytelling is one way that shows up. What's another way it shows up?
Rebecca   Trying to give you a lecture and educate you about something. You, for example, maybe ask people to come in to talk about their retirement. They're going to tell you about retirement savings that has nothing to do with the interface that you're using.
Mahtab   Yes, and to be frank, most of the time when this happens, it's with white men who would like to lecture us, and it's with people who maybe have historically been used to being in charge and having people listen to them. That's not necessarily the case for them any longer, maybe if they have already retired or if they're in a different phase of their career.
Rebecca   Yup. Yeah.
Mahtab   This could also happen a lot with ... in healthcare, like if we're dealing with surgeons or any role.
Rebecca   Anyone who's been an expert in a thing or feels that they have some expertise, whether they did that professionally or not.
Mahtab   Yeah, and they feel again, because they're here, they're here with us for a certain set of time and they're being compensated, I think that's their way of attempting to provide value. The way that they're defining value is by sharing their expertise, rather than by following what we're asking them to do with the interface. The lecture is a second way that that shows up. What's another way that it shows up?
Rebecca   If they really can't get their head around the scenario or if they take issue with the scenario for some particular reason.
Mahtab   Mm-hmm (affirmative). An example for that might be that you are telling them to imagine that they are going to take their pension as an annuity for a set amount of time, and they just don't believe in doing that personally. You're trying to validate whether they're able to make a selection in a piece of software, whether they can follow a process through to completion to validate something has been designed correctly, and then they don't like the task that you've asked them to do, so they keep trying to do something else, or just refusing to do what you asked them to do.
Rebecca   Exactly.
Mahtab   That's an example of it, or it just goes against their character. For example, if it's someone who has a really strong relationship with a particular specialist that's treating them for a chronic condition, and the site that we are validating or the piece of software that we're validating is about providing people with alternate options for managing their healthcare, or different ways that are less invasive for treating conditions, and they respond -not as a participant who's going to be exploring this- but rather, as though we are asking them to stop doing what they are doing in their actual life and then will just flat-out refuse to keep going with us and imagining the scenario.
Rebecca   Exactly.
Mahtab   Then finally, there's another thing which is that they just simply don't get it. They don't understand what you're saying.
Rebecca   Yeah. Yeah, and they can't interact with the content in a meaningful way. The data that you have there, the scenario you've presented to them, is just so confusing to them. They are incapable of following the instructions that you're asking them to follow.
Mahtab   Sometimes that's because there's been an issue with recruiting. Sometimes they have claimed to be the decision-maker, for example, for their household finances. Then they get in, and we're asking them to look at some financial information. Then they'll say things like, "Oh, my spouse does all that. I don't even know what that is." or sometimes it's that they have claimed to have more familiarity with a topic than they do. I think another way that that shows up is that they are unable to relate to the scenario. This is more rare, but it has happened. When this happens, how do we deal with it?
Rebecca   Well, I think one of the most important things is to remember that the test plan is not a contract. It's not set in stone, and it needs to be flexible. This is the case with people who are participating and cooperating, and people who aren't. You do your best to get something out of the test plan, but move around. Let people sort of lead you to something that they're going to do that's going to give you valuable feedback.
Mahtab   That's trickier and frankly harder to do when you're earlier in your career, because earlier in your career, really having a solid test plan and being able to follow it is what keeps you grounded in the research and getting the right data. Then if suddenly the instrument you're using to navigate this hour with this individual becomes a useless instrument, it can be really hard to know what to do.
Rebecca   Exactly. You have to discern, when is the data that you're getting valuable, and when is it not?
Mahtab   Yes. I think that this is one where we say that, if you've recruited properly, some data is better than no data. If this is an actual individual who meets the set of requirements to be this audience, getting something from them is better than getting nothing from them. However, if the problem is in recruitment and this is someone who would never do what you're asking them to do or is not qualified to look at it, then some data is actually worse than no data, because it's irrelevant. That is sort of like, when it's a recruiting issue, I think you need to know: stop. When it's not a recruiting issue and it's an issue of them being able to put themselves in that scenario or understand it, then I think that becomes a moment where you go, "Well, what can I get?"
Rebecca   Right. I think that's where you really do your best to listen to the ... If they're telling a story, if they're teaching you about a thing, whatever it is that they're doing that isn't in line with the test plan, what can you listen for that can hook you to a moment where you can say, "And how would you do that using this interface?" to redirect them back and try to get some meaningful interaction.
Mahtab   Yeah, to find that connection between what they want to talk about and what you want to talk about, and to sort of thread that loop to come back to it. Another thing you can do is recognize that sometimes you just got to let it go. There's no more to do there, and it's okay. Even if you're 20 minutes into a 45-minute session, it is okay to cut bait.
Rebecca   Exactly. Regardless, however you do it, try to be as kind to the participant as possible.
Mahtab   Yes, kindness always.
Rebecca   We always want them to leave the lab feeling good about the time they spent with us.
Mahtab   Yeah, and just recognizing it's going to happen.
Rebecca   Yes.
Mahtab   Yeah.
Rebecca   You're never going to recruit a full suite of people who are just going to give you 45 minutes of meaningful feedback, minute for minute.
Mahtab   Yeah, absolutely. Just recognizing that's part of it, it's okay, and you'll get what you need in the next one or the one after that.
Rebecca   Yeah, and it's not about you.
Mahtab   Yeah, absolutely. All right. I think that's what we wanted to say about that.
Rebecca   That is.
Mahtab   Let us know if you have any feedback or questions about what we shared. Our contact information is on our website. Thank you for listening, you can find us on SoundCloud, iTunes, Google Play, Stitcher and on Bye.


Hosted by: Mahtab Rezai
Principal & CEO

Mahtab has spent nearly two decades as a user experience designer, researcher, strategist, leader, and mentor. She has designed user experiences for companies ranging from startups to the Fortune 50.


With guest: Rebecca Grazzini
Senior User Experience Specialist

Rebecca has worked on user experiences in a variety of industries including online education, health care, financial services, tourism, energy, and agriculture. She has spent much of her career developing complex transactional experiences under strict regulatory constraints.
View Rebecca’s Profile


Let us know your name and email address and we’ll send you our newsletter.

First Name

Last Name

Email Address



Thank you!