Episode 014: Conducting Research Without a Test Plan

Hosted by:
With guest: John Golden
December 15, 2017

Mahtab   Welcome to episode 14 of The Crux of It. I am Mahtab Rezai, I am the Principal and CEO of Crux Collaborative. We are a user experience consulting firm specializing in regulated industries and for this episode I'm joined by my business partner, John Golden. Hi John.
John   Hello.
Mahtab   Alright, today we are going to be talking about when and why it makes sense to go into user research without a test plan, or …whether… not necessarily without a test plan, or maybe just letting the research participant drive the test plan. So, since not everybody might be familiar with what a test plan is, let's talk about what it is and what is its purpose.
John   Excellent. Yeah, at a very high level a test plan is really to establish the research objectives for the project and usually when we do that we establish them in the form of questions we want answered as a result of the research.

And then, based on those questions we develop a series of tasks and questions to walk the respondent through the site or application that we're looking at. We want to be very careful that we develop those questions in a way where we're not leading the respondent through the test plan, but helping them understand some context as they go so they can give us their honest feedback as they explore the site or application.
Mahtab   Yes.
John   But one thing I would just want to add is that one of the things that we run into a lot is that our clients sometimes interpret a test plan as a script. You know I always say that it's more of a discussion guide, we have a lot of questions and things that we follow up, and as we'll talk about in a few minutes, the idea of being able to almost either freelance, or let the user guide some of those conversations – sometimes that's when you get the best feedback and the most value because you're really hearing it from their point of view without being guided at all.
Mahtab   Yes, and I think that's a really important distinction to make because in certain types of research, for example, if you're doing market research or surveys it's very important to stay on script and to ask everyone the exact same question in the exact same way, particularly for quantitative research where you're getting statistically significant numbers of people. So that's a really great point that for clients who might not be as familiar with qualitative research, they might think this means that these are the exact words we're going to say, no more, no less. It's really more a framework for what we're going to discuss.
John   Right, right.
Mahtab   Yes. Alright so if it's so important to have a test plan and the reason for getting to the information we want, why would we consider going in without it or with one that we’re letting a participant drive?
John   Well, I would say one of the most common things is somebody brings up something that we haven't considered in the test plan but is a valuable thing to explore in terms of our client's business objectives. So, a good example of that is that they start to talk about something that a competitor is doing that may be more effective or that they perceive to be better or provide more value. So that's something that we would want to explore.
Mahtab   Yeah, and then I think another thing is sometimes organizations have an idea of what they think users are coming to the site to do. If we just take that entirely at face value and go through it, we miss key insights because sometimes when we start by saying: “What do you typically come to the site to do?” and then follow the lead of the participant, new things emerge or different things emerge where it turns out that the key things people come to do are different than what the business knew or understood, or are two things combined always together in a way that they didn't know or understand. It's really helpful to not just tell them and walk them through a series of tasks, but also to ask them and let them guide.
John   Right. Another thing to consider too is when somebody has an emotional reaction to something. One of the greatest things about having people here in person is that we can really pick up on those emotional reactions and even if they don't say something, if their facial expression changes, their body language changes or they pull back into their chair and you can ask them a little bit of questions, you know, "I'm noticing you're a little apprehensive about that," or you know, "You made a kind of a funny face when we started to talk about that, can you tell me a little bit more about what's going through your head."

Lots of times that's when we can dig into some things, and then once we start to establish some of those patterns with participants at the beginning of the sessions, then we can start to actually formulate more questions, relevant questions, to ask respondents later on as we're testing. So we may bring up things that aren't in the test plan that surfaced in earlier sessions.
Mahtab   Yes. Yeah, and build from one session to the next.
John   Exactly.
Mahtab   And it continues to evolve. Another thing I think that we've observed is sometimes an organization has designed an experience to follow their business processes and the internal way they do things, and while end users might actually do those things they do them in a completely different order than the business process. And so one of the things by asking participants to lead and just take us through doing something, is we learn how real users think of tasks and sometimes the order in which they do things or what information they ask for, look for first, second and third is completely different than what a business thinks. Because the business might set up their data in a different way, and so that's also really interesting to observe.

So if you want to try this, what are some approaches to take when you're thinking about being more freeform with a test plan or letting users guide the test plan?
John   Well, for me, one of my favorite things to do, especially at the beginning of any session is a free exploration task. So what we do is let the user come into the site naturally or the application naturally and say, "Okay so here's some context about what this is. Now why don't you show me where you would go first and what you would do first." Lots of times we can start to see trends in how people are accessing things.
Mahtab   Yes. Another thing I like to do is to have a pre-questionnaire that asks them: “What are the top three reasons you go to a site like this?” or “When you go to your bank, what are you doing?” or “When you visit your health insurance site…” whatever it is. And then say, you mentioned that the main reason you do this is to do this task. Go ahead and show me how you would do that on this site. It's very much responding to them and they're comfortable with it, because I think one of the things we're always working to do is to figure out how to get end users and research participants comfortable enough so that they don't think they're trying to do what we want them to do, but that they get into a mode of just walking us through what they would naturally do.
John   Yeah, another thing that we see a lot of times is somebody will say, you know they're going through the test plan and all of a sudden they'll say to us, "You know I would never do this, I would never do this on my own."
Mahtab   Yes. Yeah especially when it's like a, "I would never do this," and you find out they already have another means for getting that information or a preferred method that's not going through the site.
John   Yeah and I think we've seen a lot of that lately especially with looking at mobile view versus full desktop view, especially with a complicated application and what tasks and things people want to prioritize, or what they don't trust to do on their phone or mobile device.
Mahtab   Yes. Alright so does all this mean you should never have a formal test plan?
John   Well, I would say writing a formal test plan, if anything, gives you the ability to understand what questions to ask and where you can freelance and what are the objectives that our clients are trying to go for. We have a better understanding about how to be smart about the questions we ask and when to go off the test plan.
Mahtab   Yeah so short answer, no. You still need to write a test plan but I also think another key thing to highlight is that doing this is sort of letting the user lead or being able to freelance off of it takes practice. So recognizing that if you have someone who is just getting into user research, or someone doesn't have a lot of experience, that they might need a formal test plan. Or they might need a test plan that's structured in a modular way so that they can jump between things. They might not have the comfort level to just follow a user's lead down a path.
John   Right.
Mahtab   Wonderful, alright I think that is pretty much what we wanted to say on that topic. Let us know if you have any feedback about what we shared or if there's any other questions or topics you'd like us to discuss. Our contact information is on our website. Thanks for listening. You can find us on SoundCloud, on iTunes, Google Play, Stitcher as well as on our website


Hosted by: Mahtab Rezai
Principal & CEO

Mahtab has spent nearly two decades as a user experience designer, researcher, strategist, leader, and mentor. She has designed user experiences for companies ranging from startups to the Fortune 50.


With guest: John Golden

John believes there is a simple and streamlined approach for every complex problem. He approaches each project with the user’s goals in mind first in order to create a solution that is effective and meets the needs of his client’s business.
View John’s Bio


Let us know your name and email address and we’ll send you our newsletter.

First Name

Last Name

Email Address



Thank you!