Against “shoot first, ask questions later” — a measured response to “User research is overrated”
My 2 cents about the discussion emerging in the research and design community today over User Research is Overrated
“Anyone hostile to research is probably benefiting from ignorance.” — Erika Hall
It seems obvious to state that those working in the realm of ‘user experience’ must put the needs of the user first and foremost. Where do you work? Start-up? Agency? Enterprise? Government? The novelty, scale and complexity of our users’ needs may be as varied as they are numerous. But the common thread that unifies our work is understanding real human problems with a view to solving them. And the sole route to understanding those problems is to investigate them as completely as we can, before, during, even after a project. More on that later.
“The systematic investigation into and study of materials and sources in order to establish facts and reach new conclusions” — Oxford English Dictionary on Research
The riskier the problem, the more effort we may need to develop that understanding. There are no one-size-fits-all solutions. No magic bullets. Some problems are so huge, so complicated, they may be unknowable, but that’s not justification to shoot first, ask questions later. That’s the realm of cowboys and flyboys! How might one even begin to know if the problem is big or small?
JC: “Design Sprints have revealed an expensive waste of time in the modern design process… Up-front User Research is a form of Product Procrastination. It’s busy-work, it’s a way to avoid making hard decisions. It delays the need to make something tangible…”
So. Many. Questions.
By what measure was this research rated expensive and wasteful? What was the expected return on investment? Were the right people, with the right experience, expertise and resources deployed? What were we trying to learn? What blocked us from learning what we needed?
How might a reasonable person make those “hard decisions” easier? With better information.
Where does this impulsive “need” to act originate? Pressures from commercial or engineering perhaps? From a shared but unquestioned assumption that to simply act is better that to think? From colleagues itching to get to work — those same people who may later be disappointed by not having a well-realised product, that satisfies identified user needs, to sell to an identified and understood market?
How might we know who that market is and how it thinks and behaves? With better information!
How might we be best informed to know if we’re building the right things for them? With better information!
“One accurate measurement is worth a thousand expert opinions”— Admiral Grace Murray Hopper
And what exactly is the modern design process? As though there’s only one. Again, I’ll talk more on that later.
JC: “I know I shouldn’t be saying this because it’s kind of sacrilege in the design community. I also shouldn’t be saying it because I know a lot of companies still make a lot of money selling a user research phase to companies who don’t know better. I know this because we used to do it at AJ&Smart. It was part of our UX/Product Design package: 2–4 weeks of up-front user research…”
The revelation that so many agencies may be exploiting clients, selling packages that merely tick the boxes but don’t provide value is truly shocking to me. The definition of snake oil commerce.
But more so, a one-size-fits-all approach is far from how experienced designers -paid experts- should serve their clients. The strategy and tactics must fit the problem: This is designing 101.
JC: “We interviewed people, we created personas, we made empathy maps… It’s not that we were trying to milk our clients of as much money as possible, we just didn’t know better, this was the design process! I mean look at every design process diagram ever…”
Here I’ll just fall back on these words by Carissa Carter, Director of Teaching and Learning at the Stanford d.school.
“When you first learn to cook… you might follow a recipe. You are told what ingredients to use in what quantities and instructed on how to combine them… When you’re really good, you invent recipes based on what you have on hand…the needs of those you’ll be sharing the meal with, the vegetables that are in season… The order and process of a recipe helps new cooks get started, but it’s only with practice, inventiveness, experimentation, and constraints that you might begin to call yourself a chef…
Our pedagogy has evolved from the days of five hexagons.
The problem with the hexagons is that they’ve created THE design process, and that sounds grand and all encompassing, but in reality they are just a first recipe, a suggestion for how to get started… while it is sometimes useful to give students a recipe experience with their first encounter with design, I see our most exceptional instructors creating the tools and experience arcs they need specific to the projects and learning goals of the moment…” — “Let’s stop talking about THE design process” by Carissa Carter
Pity the team who take another other team’s design process, or copy some ‘design process’ diagram, copy the inputs and outputs, and expect success, but without first questioning the how and the why. Pity the clients, who put their trust and success in the hands of a team who promised more than they admittedly knew how to deliver; who paid real money for value they were unlikely to gain. But mostly, pity the end user who likely ended up with products that perhaps only partially serves their need.
In the South Park episode “Gnomes”, a global underground network of Gnomes steal underpants from sleeping children for profit. They know they’ve been told to collect underpants for profit. The punchline being nobody actually understands why collecting underpants creates profit, but they persist because that’s all they know.
If we don’t understand the how and the why, if we don’t learn strategically, we are unlikely to achieve success. We don’t have to do this. But our competitors certainly will.
What we do as designers, as problem troubleshooters, is bloody difficult! It takes forever to master. And more often than not, we will get it wrong many times before we get it right. In this age of complexity we are compelled to mitigate that risk in a fast and economical way. This does not mean rushed and cheap. This means smart and pragmatic!
“In the end, UX research is a business tool to mitigate risk”— Kelly Goto (in conversation with Patrick Neeman)
My job as a designer is not to make deliverables for my team and stakeholders per se. That’s a common output, but I exist to uncover and to share, to build common understanding. It’s about the outcomes! If I never made a single deliverable, but still aided my team to design with intention, to make good pragmatic decisions based on insights, then I have done my job well. Hell, if we can all learn it as a team, all the better! That’s great for common understanding, budget and deadlines, and ultimately, a quality end product.
“People are generally better persuaded by the reasons which they have themselves discovered than by those which have come into the mind of others”— Pensées by Blaise Pascal
JC: “It took a while, but eventually we started to notice a worrying pattern: We would do the pre-research for a specific product or service, do the interviews, create the personas, create the documentation then as soon as we got down designing and testing the actual product, we figured out that even though it was nice to “have the user in mind” when designing, the useful data came from the first user tests, not the research. In fact, more often than not, the personas and other documentation just started gathering dust while the rest of the process continued…”
“We collected all these underpants, and we still didn’t make a profit!”
I think at last we’re getting to the root of the problem. Let us avoid the abject horror of reading a UX professional describing user need as a “nice-to-have” and get straight to the insight: It’s not that research is a “expensive waste of time, but rather the superfluous deliverables.
Commit to the bin those useless (yet oh-so-billable) deliverables that don’t create value for our clients and their customers; that do nothing to de-risk a project. It’s admirable and correct, in my humble opinion, to ‘show them the thing’ as quickly as possible. But just make sure to have a good idea who the thing is for. Otherwise, who do we have in mind? Ourselves, or some ambiguous, unvalidated tropes perhaps?
“Our early design decisions are like bets whose outcome we will have to live with iteration after iteration. Since that’s the case, there is a strong incentive to be sure about our early bets. In other words, we want to reduce uncertainty on the first iterations.” — Getting Back Into The (Right) Deliverables Business by Gavin Eliot
The Design Sprint process exposed that our own design process was mostly filler!
Design sprints are a form of research action, not an alternative to other research! (In fact, everything we do to add to our knowledge is research). They’re fast, intensive and economical, and they undeniably empower learning. But don’t throw the baby out with the bath water. Don’t ditch proven methods for the quick and easy. What do they say about fast, cheap and good?
“It turns out that being able to put something tangible in the hands of your potential (or assumed) customer gives you infinitely more valuable data than just researching and documenting, then trying to build assumptions from that. Let me be clear: in both cases we are just making assumptions.”
We all start with assumptions — but speed reduces the time to think, to consider the horizons, the peripheries. It can reintroduce risk . Where you can do something about it, mitigate the risk that less-informed assumptions will bring. That doesn’t have to take 2–4 weeks, but if it needs to, it should, and maybe more.
“Don’t fail fast — Fail cheap, and learn from it.” — In conversation with Fabrice Retkowsky
JC: “We do not really know what our potential users will really respond to, what they will understand or what they’ll hate until we really see them using it. So whether we spend the first 4 weeks trying to learn what they like, how they behave and whether they’re a “mobile native” or not, OR just make assumptions then test them 4 days later, we’re still only seeing the really valuable, usable data once they use the product (or the very realistic prototype of it)…”
This is a common misunderstanding. Research is a broad church, so let’s get to practicalities. Over the lifecycle of a project of any size, I’m trying to validate a few key concerns: Will they use it? Can they use it? Are they using it?
And user testing with your prototype will only answer one of those questions: “Yes, they can use it”. But that’s only one lens. That gives us very little info about why we’re building it, and for whom. How can we be sure the people who need our product, and the people who can simply use it are the same people? If only there was a way we could reasonably and economically know those needs before wasting a lot of time and money building something? 🤔
JC: “And yes, there are cases where up-front research makes a lot of sense… for the most part, though, I recommend ditching the time consuming, wasteful up-front research in favour of tangible results…”
Doing just enough research, just in time is essential in every project of any measure. And if we don’t believe that, we do not understand the problem before us.
The implication that this needs to take up x amount of time or money is pure fallacy. If we only need a single day, then we only need a single day. If we assume we have nothing to learn, we become complacent; We rubber stamp; We tick boxes; We sleep at the wheel; We act as we have always acted…
We do not improve.
If we undertake research without a clear hypothesis or desirable end state in mind, without thinking about the why, then get junk out the backend, is that really a failure of ‘research’? No, it’s a failure of rationale; a failure to learn strategically — A failure to listen to the users.
“If all you ever do is crappy research, you’re gonna think research adds no value. Maybe you should clean up your act before trashing it?” — Jared Spool
