Mahtab: Hello and welcome to episode 18 of The Crux of It. I'm Mahtab Rezai. I'm the principal and CEO of Crux Collaborative. We are a User Experience Consulting firm specializing in regulated industries. I'm joined today by my colleague, Rebecca Grazzini. Hi Rebecca.
Rebecca: Hi Mahtab.
Mahtab: All right, so today we are going to talk about what happens when the research participant is not participating properly and has gone rogue. So …something that you and I have both experienced on more than one occasion.
Mahtab: What does this look like when the participant is not cooperating? How does that show up? What are some ways that shows up?
Rebecca: One of the key ways is someone will come in, and they have a story to tell. It is somewhat tangentially related to what you're doing, but they're going to talk about their relationship to the thing you're studying. For example, if it's a pension, they're going to talk about their retirement, how they got laid off, and how this thing happened, and how this plant closed, et cetera, et cetera.
Mahtab: Yes. Some of the reasons why that happens is because they've had a trauma, so like a layoff, or they had a bad experience. You're trying to talk to them about how they would find a piece of information using a website or how they would maybe transfer some money from one account to the other, but they see a logo of a company. They once had a bad experience with that company, and now they just completely get distracted and want to tell you all about the time that they got screwed over by said company.
Rebecca: Right. Right.
Mahtab: Or they're maybe confused.
Rebecca: Yes. Yes. Everyone comes into these sessions. They are coming here to earn money. They want to do a good job, and they just don't want to look dumb.
Mahtab: Yes, absolutely. That's one thing, so storytelling is one way that shows up. What's another way it shows up?
Rebecca: Trying to give you a lecture and educate you about something. You, for example, maybe ask people to come in to talk about their retirement. They're going to tell you about retirement savings that has nothing to do with the interface that you're using.
Mahtab: Yes, and to be frank, most of the time when this happens, it's with white men who would like to lecture us, and it's with people who maybe have historically been used to being in charge and having people listen to them. That's not necessarily the case for them any longer, maybe if they have already retired or if they're in a different phase of their career.
Rebecca: Yup. Yeah.
Mahtab: This could also happen a lot with ... in healthcare, like if we're dealing with surgeons or any role.
Rebecca: Anyone who's been an expert in a thing or feels that they have some expertise, whether they did that professionally or not.
Mahtab: Yeah, and they feel again, because they're here, they're here with us for a certain set of time and they're being compensated, I think that's their way of attempting to provide value. The way that they're defining value is by sharing their expertise, rather than by following what we're asking them to do with the interface. The lecture is a second way that that shows up. What's another way that it shows up?
Rebecca: If they really can't get their head around the scenario or if they take issue with the scenario for some particular reason.
Mahtab: Mm-hmm (affirmative). An example for that might be that you are telling them to imagine that they are going to take their pension as an annuity for a set amount of time, and they just don't believe in doing that personally. You're trying to validate whether they're able to make a selection in a piece of software, whether they can follow a process through to completion to validate something has been designed correctly, and then they don't like the task that you've asked them to do, so they keep trying to do something else, or just refusing to do what you asked them to do.
Mahtab: That's an example of it, or it just goes against their character. For example, if it's someone who has a really strong relationship with a particular specialist that's treating them for a chronic condition, and the site that we are validating or the piece of software that we're validating is about providing people with alternate options for managing their healthcare, or different ways that are less invasive for treating conditions, and they respond -not as a participant who's going to be exploring this- but rather, as though we are asking them to stop doing what they are doing in their actual life and then will just flat-out refuse to keep going with us and imagining the scenario.
Mahtab: Then finally, there's another thing which is that they just simply don't get it. They don't understand what you're saying.
Rebecca: Yeah. Yeah, and they can't interact with the content in a meaningful way. The data that you have there, the scenario you've presented to them, is just so confusing to them. They are incapable of following the instructions that you're asking them to follow.
Mahtab: Sometimes that's because there's been an issue with recruiting. Sometimes they have claimed to be the decision-maker, for example, for their household finances. Then they get in, and we're asking them to look at some financial information. Then they'll say things like, "Oh, my spouse does all that. I don't even know what that is." or sometimes it's that they have claimed to have more familiarity with a topic than they do. I think another way that that shows up is that they are unable to relate to the scenario. This is more rare, but it has happened. When this happens, how do we deal with it?
Rebecca: Well, I think one of the most important things is to remember that the test plan is not a contract. It's not set in stone, and it needs to be flexible. This is the case with people who are participating and cooperating, and people who aren't. You do your best to get something out of the test plan, but move around. Let people sort of lead you to something that they're going to do that's going to give you valuable feedback.
Mahtab: That's trickier and frankly harder to do when you're earlier in your career, because earlier in your career, really having a solid test plan and being able to follow it is what keeps you grounded in the research and getting the right data. Then if suddenly the instrument you're using to navigate this hour with this individual becomes a useless instrument, it can be really hard to know what to do.
Rebecca: Exactly. You have to discern, when is the data that you're getting valuable, and when is it not?
Mahtab: Yes. I think that this is one where we say that, if you've recruited properly, some data is better than no data. If this is an actual individual who meets the set of requirements to be this audience, getting something from them is better than getting nothing from them. However, if the problem is in recruitment and this is someone who would never do what you're asking them to do or is not qualified to look at it, then some data is actually worse than no data, because it's irrelevant. That is sort of like, when it's a recruiting issue, I think you need to know: stop. When it's not a recruiting issue and it's an issue of them being able to put themselves in that scenario or understand it, then I think that becomes a moment where you go, "Well, what can I get?"
Rebecca: Right. I think that's where you really do your best to listen to the ... If they're telling a story, if they're teaching you about a thing, whatever it is that they're doing that isn't in line with the test plan, what can you listen for that can hook you to a moment where you can say, "And how would you do that using this interface?" to redirect them back and try to get some meaningful interaction.
Mahtab: Yeah, to find that connection between what they want to talk about and what you want to talk about, and to sort of thread that loop to come back to it. Another thing you can do is recognize that sometimes you just got to let it go. There's no more to do there, and it's okay. Even if you're 20 minutes into a 45-minute session, it is okay to cut bait.
Rebecca: Exactly. Regardless, however you do it, try to be as kind to the participant as possible.
Mahtab: Yes, kindness always.
Rebecca: We always want them to leave the lab feeling good about the time they spent with us.
Mahtab: Yeah, and just recognizing it's going to happen.
Rebecca: You're never going to recruit a full suite of people who are just going to give you 45 minutes of meaningful feedback, minute for minute.
Mahtab: Yeah, absolutely. Just recognizing that's part of it, it's okay, and you'll get what you need in the next one or the one after that.
Rebecca: Yeah, and it's not about you.
Mahtab: Yeah, absolutely. All right. I think that's what we wanted to say about that.
Rebecca: That is.
Mahtab: Let us know if you have any feedback or questions about what we shared. Our contact information is on our website. Thank you for listening, you can find us on SoundCloud, iTunes, Google Play, Stitcher and on cruxcollaborative.com/thecruxofit. Bye.