4 Main User Research Methods
I love e-learning and try to take new courses every month. One of my favorite e-learning platforms is the “Interaction Design Foundation”. There are many great courses about the psychology of interaction design, augmented & virtual reality, usability, emotional design and many more… In this article, I would like to share a summary of one of the courses I took: “User Research Methods”.
The course adopts the design thinking process. User research involves in every step.
If correctly done user research helps to maximize the impact of the overall development process. Here we will talk about 4 main research methods: usability testing, semi-structured qualitative interview, contextual inquiry, and user observations.
I found this diagram from the course particularly useful:
Usability testing:
The main question that the researcher tries to answer with this method is: “Do the user see the functionality”. During the research, user’s confusions and errors become design goals to accomplish for the next testing cycle. Therefore, the data gathered during usability testing can be quantitative and/or qualitative.
The researcher observes and records:
a) Behaviors such as task performance, speed, efficiency, goal fulfillment…
b) Opinions such as ideas on each task performed, thoughts…
c) Data such as the click path (eye-tracker), the pages visited…
The sample size for usability testing is fairly small because it is qualitative heavy: 8–15 users constitute a common sample size. It is generally accomplished in the lab. (If a field study is necessary, there are other research methods that can be used).
Have an idea on potential usability issues
In this method, prior to inviting the users to the lab, the researcher prepares the tasks that will be accomplished by the users. Therefore, it’s important that the researcher has an idea of potential usability issues before the actual study starts. The research might reveal many things that the designers and researchers haven’t thought of before but having a list of areas to concentrate on is a must. Especially given the fact that the user will have limited attention. The duration of the study varies from 60–90 min and each user can get around 12 tasks. For each task, the researcher should determine, the objective, the test scenario (the one that will be shared with the user), the end state and assets needed (ex: a cellphone with internet…)
How does the scenario sheet look like?
Each sheet should have one task that is written no less than 17 pt. The tasks can be questions that are directly addressed to the user: “using this app you want to book your next spa, according to the app is there a way to choose the location?”
The report:
The report is around 10–15 pages and contains an overview with the given tasks, task success rates, error/confusion rates, and relevant information gathered during the post-scenario interview. After the overview, there is a test walkthrough with the detailed information on each task, notes of the researcher and recommendations related to each task.
Recommendations for good usability testing:
This part of the course was prepared by Frank Spillers and here is some advice that he gives about good usability testing:
Ask the user to think aloud. Externalizing their thoughts helps in getting in the mind of the user.
You can have a post-scenario interview to get more thoughts and ask a question that you shouldn’t ask during the study to prevent bias.
Select participants on the based on the real end-user.
Many times, the user asks questions to understand how he/she is doing, to get approval, etc. The only right answer to give in this kind of situations is “you helped us identify what we were looking for”.
In the report, using expressions such as “user goals” instead of “our design” and “what users need” instead of “what team wants” help to communicate the message.
Semi-Structured Qualitative Interviews:
This method is used when an in-depth understanding of the user’s values, perceptions and experiences is needed. The semi-structured qualitative interview method isn’t used to understand the “what” but the “why”. What the user does is best understood through observation but why the user acts in a certain way can be explored by this method. The downside is that this method is quite subjective and dependent on the researcher’s interpretation. It’s best used for the concept exploration phase and in combination with other methods.
Question is king
The quality of this research method depends heavily on the questions asked, how they are asked and in which order. The introductory question should be easy to answer and concrete. For example, if we are researching online music listening habits, we can start by asking “Can you tell me which types of music platforms you use?”
The following questions should be about concrete experiences because users will have a hard time answering a general and abstract question such as “What are your general shopping habits?” Instead, this question is much easier to answer: “Describe what you did the last time you went shopping?” Asking questions about a specific experience on a specific time (should be recent) is called concrete experience method. Another way of gathering good quality input is by asking questions about a critical/memorable incident. For example, “Tell me about a time when you had a problem in finding the song that you wanted”.
Most commonly used method for analyzing semi-structured interviews is thematic analysis. It’s a process that transforms the messy data into themes. It involves familiarization with the data, giving codes to content, searching for patterns, reviewing and defining themes. It is very similar to affinity diagramming, however instead of post-it notes, generally a software is used to map the themes (NVivo, Atlas, Dedoose, etc.). Reporting involves personas, user scenarios resulting from the thematic analysis and description of themes.
Recommendations for good semi-structured qualitative interviews:
This part of the course was prepared by Ann Blandford and here is some advice that she gives about good semi-structured qualitative interview:
The interview should start with an introduction with the purpose of the interview and the topics that will be covered.
The researcher should accept pauses and let the participant think
Trying to explain a question can introduce the researcher’s bias.
Making eye contact with the participant enhance trust.
Contextual Inquiry:
Contextual inquiry is different from other methods because it requires researchers to go outside of their lab and observe users in their natural environment. It’s different from the naturalistic observation because the researcher can stop the user and ask questions on why he/she is doing something in a certain way. It is also different from semi-structured qualitative interviews not only by the context of the research but also by that fact that the researcher can share his/her interpretations with the user to build trust or to prevent misunderstandings. The attitude of the researcher here is called “master-apprentice” model. Here the user is the master, and the apprentice is the researcher.
Contextual inquiry is powerful yet an expensive research method. This method is best when the company is launching a brand-new product, or a re-design is needed. Most “usual” projects (ex: launching or re-designing a website) can be completed by faster and more “affordable” methods such as user testing. The sample size is relatively small and is around 10–12 participants (2 hours per each).
There are 3 main questions to answer while doing a contextual inquiry: “What do people do and what are their goals?”, “How much variations exist in these tasks and goals”, “What is a good and bad experience”. These questions are answered with the data gathered. The main method of data collection is taking notes, audio recordings and photos. Video recording isn’t recommended here since people act differently when they see a camera.
Once the field work is done, the data is broken into affinity notes. Each affinity note consists of an observation about one single concept and written in the first person. For example: “I use a minimal shopping list”. From affinity notes, researchers identify preliminary categories. Identifying the larger categories demands multiple iterations and sessions. Given the fact that there will be thousands of affinity notes coming out of a contextual inquiry, revisiting the notes and refining categories is a must for the affinity diagram. During this process, different stakeholders can be invited to increase the buy-in of the other departments and clients.
User Observations:
The main reason for directing an observation study is to learn what people actually do. It can be used in the pre-design phase to find out about the context and brainstorm on how people might use a product. Observation can be helpful in the ideation stage to test a prototype in its context.
While doing an observation study it’s important to use the user’s own device if it’s possible. The study can be run in a lab, in a similar way to the user testing but in a less controlled manner. The best use of this method, though, is to run it in the user’s own environment. While running the study the researcher can decide to have an active style where he/she interacts with the user. However, a passive style where the researcher is “invisible” is preferred to keep the environment as “natural” as possible. During the study, the researcher observes behaviors, words, faces, environment, and artifacts.
Recommendations for good user observations:
This part of the course was prepared by Alan Dix and here is some advice that he gives about good user observations:
Interpreting the culture in its own terms is the goal, even though achieving total enculturation is challenging.
“Estrangement” is a method of making normal things strange. This way, seeing the obvious can be possible. For example, if I ask you to write down all the steps you followed when you were making your coffee this morning, probably you’ll miss a lot of them that might be inspiring for design purposes. Because they were too obvious to see. If you observe each step you follow when you make your coffee tomorrow, in other words, “estrange” the everyday coffee making habit, you’ll be able to see some of the “obvious” acts and be quite surprised.
“Appropriation”, in other words, using things different from their design reason is very valuable.
Conclusion
The Interaction Design Foundation course provides a good summary of 4 main user research methods and recommends essential books to learn more about them. Having an idea about what each user research method can add to your project is important to collect the right input. This will prevent your business from going over budget when there is a more affordable method or under-investing during a big design decision process.
