To user-test, or not to user-test? That is the question.
Recently, I started a project with a UX agency. My idea: work on a high-fidelity prototype and test with users asap. The agency was against user-testing. Their argument was that as the UX experts, their knowledge, experience and the persona research conducted up front provided enough information to build a ʻgoodʼ experience. User-testing would not be a good use of time and budget.
My response at the time, ʻHmmmm,ʼ followed by ʻreally? OKʼ. The agency had (combined) more experience than I did, were successful in their field and certainly knew their stuff. I was happy to follow their lead, but still, the flat out ʻNo!ʼ to running a user test didn’t quite sit right.
From my own experience and certainly everything I’ve studied, read, watched, been told, points towards the positive benefits of running a user test. The answer to ʻshould one run a user test?ʼ has always been a big ‘Yes!’ in my book. The results from seeing a user either struggle to understand and use your product or to simply validate your ideas have always proved highly beneficial. However, I also knew this wasn’t an unusual stance and there are many who don’t do any type of pre-testing before the launch.
I needed a second opinion and wrote to a friend, Rob Hamblen, product designer at AJ&Smart (again who was more experienced than me when it came to running user tests). My intention was never to turn this into a ʻwho’s right and who’s wrongʼ thing between myself and the agency but try to find the middle of the road. It seemed, however, I wasn’t alone in running into this predicament. As Rob explained.
“Yeah, we hear this a lot. Also, Iʼve experienced it first hand at IBM . . . why would you not want to test your designs with the target audience who will ultimately validate your design? Are they afraid it will be an outright failure? This can only help the product get to market much more quickly. The other thing is that a lot of good design agencies or UX pros do think they own the rights to what the customer wants! In my experience, these are typically bad designers or designers with little experience! Just because they have the insights and empathy, they think making those decisions makes them feel like they are playing god.
Ooooh, good topic!!”
It was Robʼs last comment “Ooooh, good topic!!” that made me think I’m onto something here. I wanted to investigate further. The plan, post the question ‘User-testing or no user-testing?’ on LinkedIn, contact a few experts, collect the feedback and hopefully come up with a definitive answer. So here goes.
A yes for the test
Out of all the people I asked there was overwhelming support for conducting a user test. Robert Stulle (MD at Edenspiekermann, an experienced voice in the industry) also backed up Rob’s argument.
“My opinion is easy: I am always in favour of a user test. No number of years in the industry and experience as a UX person can outweigh the insight from watching someone use something for the first time. . .”
Leander Huth (Project Manager at cleverclip who produce videos for brands) also replied.
“We always do user-testing with our videos . . . It really helps to find the right wording and also to realise asap if you lose your audience at an early stage.”
Like others who wrote back, Frances Brown (Chief Executive Officer at Fowlam Ltd) made the point that any user test allows the tester to get an insight into how the whole experience works, or doesn’t work in some cases.
“. . .individual elements may work well in isolation, every new combination and interaction of elements creates a new overall experience for the user and you canʼt really ever know how well that experience is working without testing it.”
However, like anything, the answer is never just black and white. Although in general, everyone agreed that a user test should be run, there were however circumstances that pointed toward not conducting a test. There are also other questions and steps that needed to be asked and taken before pressing on to a user test.
Conduct upfront ethnographic research
Michal Mazur (User Experience Designer at Kurt Geiger) also gave a big resounding yes to running user tests. However, Michal did take the question further, or back to the beginning, by introducing the idea of conducting ethnographic research before any design work is even started.
“I think both sides of research are essential to building a good product. If a UX designer claims to be an expert, I laugh, because itʼs impossible to be an expert on every aspect of human behaviour, especially on the agency side. I think there should always be ethnographic research conducted prior to any strategy or design work. It ensures that your general direction is what fits into the userʼs world. During design iterations, testing is necessary to validate the user journeys as well as detailed design decisions. In my opinion, if you miss out on either of those research types, your product is not user-centred, but business-centred/user-informed”.
I asked Michal if he could elaborate on ethnographic research.
“I mean conducting qualitative research in the context natural for the user and the potential product. Letʼs say youʼre developing an app for museum-goers in Berlin. Ethnographic research in that context would be a mix of research methods such as observation, user interviews and surveys. However, in such research you donʼt talk about your product, but only about the ways people visit museums in Berlin, whatʼs important to them, etc. That allows you to focus on the user needs and not get distracted by premature ideas you might have. … If you just start with design and then test, you will do this with all the conversations circling around your product, and not the user needs. It can lead you to a dangerous situation, where you have a great product, which doesnʼt solve the right problem.”
The benefits should outweigh the cost
One issue that some did raise, was do the benefits outweigh the cost of running a user test? i.e. if the testing proves too costly or cumbersome it might negatively impact or actually jeopardise the launch of the product or website. Dave Patterson (Digital Strategy and Solutions Director at CyberLogical) put it best:
“Interesting problem . . .A slightly different take on it for you to consider.
There is a cost to user-testing, there is also potentially a benefit. If the cost exceeds the benefit then there is actually a case for NOT doing user-testing — but I think this would be rare, but at the end of the day it should usually be a commercial decision, so there needs to be a clear commercial case made for doing it (which in most cases there is).”
Rico Chow (Experience Design formally at Sapient Razorfish now at Karmarama) also explained.
“Yeah, giving them (the UX agency) the benefit of the doubt, are you planning to release quickly and take a launch-and-learn approach? But even so, itʼs a long shot, even if you launch MVP I would still test the MVP before launch. Testing doesnʼt have to be expensive at all or time-consuming”.
Fast-prototyping testing with five people
The art is to find a quick and inexpensive way to run a user test. Make the test as small and lightweight as possible while ensuring that the results are still meaningful. I’m a big fan of working with the Design Sprint method, which puts emphasis on producing a ‘good enough’ prototype to test with real users.
As Tytti Kontulainen (Project & Product Consultant) explains
“Getting the first round of validation through fast-prototyping can give the team a green light for deeper exploration, analysis, and testing… or in the worst case: it’ll make you save a lot of money and time.”
Rico Chow also backed up the argument
“Maybe test with low-fi prototypes, wires and in a guerrilla format . . . test with 5 people. Chances are you will catch 80% of issues.”
Guerrilla testing is something Iʼm personally a fan of and have had the best results from. Various studies also suggest that five people are the magic number when it comes to users for a test. Anything more than five and the results tend to be the same and add no further value.
This is something also pointed out in Jake Knappʼs Sprint book (A must-read for any designers). In the book, Jake outlines the ‘best’ number of users to have in a test:
“How many interviews does it take to spot the most important patterns? Nielsen analyzed eighty-three of his own products studies. He plotted how many problems were discovered after ten interviews, twenty interviews, and so on. The results were both consistent and surprising: 85% of the problems were observed after just five people.
Testing with more people didn’t lead to many more insights–just a lot more work. “The number of findings quickly reaches the point of diminishing returns,” Nielsen concluded.”
Conduct quantitative and qualitative testing
Another point which others did raise and to quote Rob again:
“The other question should be qualitative vs. quantitative user-testing.”
When it comes to quantitative versus qualitative research the best approach is to combine the two for a more holistic perspective. Quantitative testing deals more with the factual side and statistics of a product or website, e.g. task duration, and completion rates. Qualitative testing, however, deals with the emotional side, e.g. how did the user feel, delighted, impressed, hooked or confused?
Combining the two allows the tester to gain not only hard facts but also an insight as to whether the experience was fun, frustrating or most importantly addictive. Fixing something from the quantitative testing can be as simple as moving a ʻBuy nowʼ button and can be relatively minor. However, gaining insights from qualitative testing into a personʼs emotional state can only come from interviewing actual users and not from a persona chart. Fixing an emotional ‘bug’ requires more time, investment and is best caught before launch.
Another quick point and one raised again by Rico Chow was the issue of A/B testing. If in doubt, build two versions, test and whichever one wins, e.g. has the most conversions, go with that design.
Conclusion and final thoughts
What I initially thought could have been a straight yes or no answer did, in fact, turn out to be more nuanced.
It was Dave Patterson who again best summed it up. Dave listed the relevant questions to consider before running a user test.
“What answers do you want the user-testing to provide — what are you testing to find out?
How much time and money will the user-testing cost?
If you delay going to market because of the user-testing, what impact will that have?
If it is a process which users need to complete, what happens if you increase the conversion during that process? Can you identify a per transaction benefit/saving/revenue?
What is the opportunity cost to the business if you donʼt do user-testing?
What are the less tangible costs of not testing (poor experience, frustration), can you quantify these in any way?”
All great questions to consider before heading with a clipboard in hand to a user-test scenario. What I’ve concluded from this little, albeit unscientific study, are a few simple but key takeaways.
- Do the benefits, of running a test, outweigh the time and cost?
- Design a good, but not perfect, prototype to take to the test.
- Test with five users.
- Conduct both qualitative and quantitative testing.
- Iterate and test once again.
That’s pretty much it.
Below are a few links sent by people I asked, that I certainly found useful.
Here is also a great article from Nick Babich, “The Art of the User Interview”, on how best to conduct a user test.
A massive thanks to everyone for taking the time to send me their feedback. Do you agree with the above? Am I talking total rubbsih? I’d love to hear from you either way. As mentioned, this is my conclusion but I’m always on the lookout for more opinions.
Gareth Fox is a multi-discipled designer & brand strategist at IM MAI Berlin. Follow him on twitter G23Fox or Instagram foxinberlin
