.png)
Research is oftentimes considered as an entirely academic discipline. Something that happens in labs, by people in lab coats, examining behaviour through a microscope and then writing a paper that will only be read by 4 of their peers, and even then, only so those peers can comment "sample size inadequate to support theory, please consider revising conclusion" on the paper, preserving the purity of the discipline for another age.
It's certainly not something that the hard-nosed Glengarry Glen Ross "Coffee is for closers" businesspeople of the world should concern themselves with. Why that's all gumption and discipline and sales, sales, sales baby!
Now of course, that's exaggerated for comedic effect but you've worked places and you've seen it. How often are decisions being made on weak evidence and an imagined picture of a customer or user rather than actual observed evidence? I'd be willing to bet that, for all the times we use language like "moving fast" and "placing bets" we're actually often just operating from a position of low knowledge. Now that's actually probably fine if we have really accurate measurement and our decisions will change and revert and move, but most of these businesses also have roadmaps with clear and defined "milestones" to be followed. Changing direction is bad when you've strictly budgeted for these releases. If you get halfway through the year only to find out that the place you're headed at the end of the year doesn't actually sell more things, or sell things to more people. Well that's a rough place to find yourself.
There's a level of strategy required to avoid this for sure, but also, the researcher class is what helps us build the map in the first place. You know how it goes when we don't do this part, or when we do this part badly. We get 'research theatre' where the researcher is asked to simply rubber stamp the opinion held by the decision maker at the start. Often that's framed as "We need to find evidence that shows that...".
I actually had that exact ask in my career. Back when I was working for a very large company. I was a junior CRO guy, very early on in my career. The business had launched a new product area, and management had decided that rather than add this area to the existing site navigation, that we needed a new nav design to drive the sales, retention and customer service goals they had. There was hesitancy in the team around the new nav, as it made decisions around what to show and where that seemed counterintuitive to our research. The good news was, we didn't need to gamble! We can test these things in a really safe and controlled way. I, the plucky young CRO guy said "Hey, we should test this! We would then get data on how the nav performs and use that to make decisions. We'll actually end up with a much better product"
Reader, the response I got was that we would not be testing this nav.
The nav goes live, numbers drop. I then get told, as a CRO guy (who knows the data - because it's part of his job), that we need to find data that can go in a slide deck that shows the new nav is performing well. I look for the data and no such data exists. The numbers are either inconclusive or outright poor. I go back to the senior person requesting the data and let them know the truth. No such data exists to show the nav is performing well.
As you can imagine, all hell breaks loose. Some political infighting happens. The data request goes to about 3 different people, each comes back with either a conclusion that is tenuous (eg. this very specific segment of users on mobile on this channel shows some growth) or they conclude the same as me. The data does not tell the story you want it to, no matter how much you torture it.
I never saw the final slide deck, but the story I heard was that the senior person just made something up. In many ways I think the genesis of Another Web is Possible may have been planted there (shit metaphor, you can't plant a genesis, you'd just get soil in the cartridge slot). I didn't know then that this would span into a +15 year career but I did know that there was an opportunity to do it right, and to do it well, and in that moment power had chosen not to do it right or well and then paid the price for it later. The model I had, that these businesses were all really well-run ships full of brilliant people at the top of their game was cracked slightly. These are human institutions, and they do not run as well-oiled predictable machines. The same flaws and phenomena that happen in our social and domestic politics appear in business, and so if we want to be better, we must choose to be better every time the choice is available.
.png)
So what?
The conceit of this post was to talk about the researcher, plus have a little fun with the Final Fantasy job system framing. Also something of a foil to let me talk about those skills specifically and share some personal stories so I'm going to do that now.
Research in CRO and digital experiences has an almost single function and that is to understand your customer. Even the stuff that isn't about that is about that. For example, researching the market. That's about understanding the environment your customer is in. Imagine them in the marketplace, and you are one stall of many. From that understanding of their environment, you may be able to infer something about their mindset and goals.
You might ask a user to self-report. You might observe. You might capture the data on a small scale (eg, you will watch or interview 10 people) or you might do it on a mass scale by looking at analytics. There's deep skills in there that are useful, much around identifying your own biases and eliminating them from your practice, and around asking the correct questions.
Start with a plan. What do you want to know about your customer? How do you get that in a way that eliminates as much bias from the process as possible? Make sure we don't make any logical leaps or fall into any semantic traps along the way (eg. when somebody self-reports something, you are measuring their sense of self not specifically an objective reality). Do the best research you can with the time and resources you have available and then do something with it.
Like what? I need ideas, Jon! Ok. I hear you. Do you like lists? Here's a list for inspiration:
Behavioural (what people do)
- Web analytics: what pages or screens do people view? What do they click? What do they buy? which segments, which devices, which journeys
- Form analytics: Subset of web analytics. This can hyperfixate on fields in a form which are often the end point of something on a website (eg. payment forms, lead gen forms)
- Session recordings: I'm looking over your shoulder whilst you use the website and seeing if I can identify any patterns from your behaviour
- Heat/scroll/click maps: I think these tend to get overstated because they are highly visual, but maybe useful for attention, ignored content.
- Site search logs: what do people look for and how do they phrase it?
- First-click testing: where people click first for a given task. I find this is mostly good for identifying if you've made something obvious or not.
- Eye tracking: Really limited application this one, especially given how invasive the methodology is.
Attitudinal (what people say)
- User interviews: Typically very low sample sizes and so largely a supportive methodology to infer motivations and context
- Jobs-to-be-done interviews: A subset of the user interview and we sometimes just capture this by survey. What caused a purchase decision, what else was considered etc
- On-site polls and surveys: quantifying attitudes and intent with a hope to do so at scale. Surveying itself is bias heavy though. You capture a specific audience who would fill out a survey, plus it's self-report and a poor substitute for observed behaviour
- Exit or abandonment surveys: Sometimes useful if something is broken but generally weak as an evidence source
- Post-purchase surveys: These are pretty much everywhere now plus a lot are generic data collection that goes unused so your audience will probably be pretty fatigued by it.
- Diary studies: Limited application for web or app design, but widely used in medical trials and such because they help control for sentiment and mood at specific capture points
- Focus groups: Similar to the above, but relatively expensive to run plus, by design, the people in the group are highly engaged and may not represent the core audience
- NPS, CSAT, CES: Standardised ways of capturing satisfaction/chance to recommend etc.
Evaluative (testing design / structure)
- Moderated usability testing: Watching somebody do something and talk out loud. This can be REALLY useful but I will write up a critique of this methodology as it's been industrialised and usefulness has dropped.
- Unmoderated usability testing: As above but even worse. Almost useless without strict controls.
- Five-second testing: This can be good if done in a very specific way, but easy to get this wrong and produce junk data at scale
- Preference testing: Good for capturing preference, but don't fall into the trap of inferring that preference is better performance*
- Prototype testing: A really controlled way of showing off a new thing without releasing to a scaled audience. This can help you get some early bugs out and make decisions for a launch.
- Card sorting: Figures out how users group and think about content
- Tree testing: Whether people can find things in a structure. I remember doing this with white goods once and some audiences struggled to locate washing machines by room. Strong opinions across geographical lines on if they should be in the kitchen or the bathroom.
- Heuristic evaluation or expert review: Some knobhead (me) giving their opinion on your website. Weak evidence unless there's a really clear picture of your customer and what they want. Cheap and fast though.
- Method shopping/Cognitive walkthrough: A heavily recorded and documented task as a user. This can be powerful if analytics guides you too because you can take the most common path.
- Accessibility audit: Has a very clear framework for acceptance. Helps users with and without specific needs. Probably also good for SEO and AI because a lot of it is about readability and structure.
Desk (evidence already laying around)
- Support tickets, chat logs, call transcripts: Good for investigating what people are already getting in touch about. Unlikely to be positive and might not be website specific, but useful regardless.
- Review 'mining' yours and competitors: What are people talking about? Are you explicit about these things on your website or app?
- Frontline staff interviews: Anecdotal but potentially useful, especially if no transcripts exist
- Competitor analysis: What are your competitors doing? Not really evidence that you should do it too, but good for understanding at least.
- CRM and customer data analysis: This can be useful for spotting recurring customers and their behaviour.
- Social listening: How are people talking about you? This can often be beautiful because you get to see a very clear picture of what the business is to people when all marketing/brand stuff is stripped away. eg. "They're not the cheapest, but they are well built and you'll waste much less time in the long run"
* I remember seeing this with a paint retailer and everyone LOVED the vibrant colours, but one look at their sales data would tell you that what people actually bought was cream, beige and grey paints.
.png)
