The best thing to do when trying to improve a product is to put it in front of the people who are going to use it — your end-users. Put it in front of them, ask them to use it and observe; simple as that. This is called a usability test and you can use it to understand how real users will respond to your product, what they experience, and most importantly, how to improve user experience (UX). The problem, or rather the hindrance, is that this can lead to inaccurate results because your typical user isn’t likely to use your product in a lab (or maybe they are, but probably not). No worries, there’s an easy fix for this: remote usability testing.
What is Remote Usability Testing?
What do you do when you want the most authentic and natural feedback from end-users? You conduct a remote usability test (we’ll use RUT from now on). Long story short, it’s a method of usability testing in which users are in separate locations from the researcher. Apart from wanting realistic user environments, RUTs also help out when the researcher wants to test something that can’t be measured in a lab — this is called destination-based testing (E.g., a mobile app that needs to be used outside).
Let’s Do a Walkthrough
Sometimes the best way to learn is to go through a walkthrough, so let’s do just that.
What’s Your Objective?
Before anything else, you have to define an objective. Your first thought should be, “How will the results improve UX?” At the end of the day, the point of a remote usability test is to gain actual insights into what goes through an end-user’s mind as he/she uses your product, what challenges or complications they face, and most importantly, how to overcome them.
Moderated or Unmoderated Design?
Up next, choose between the two types of RUTs: moderated and unmoderated. As their names suggest, the researcher acts as a moderator in one of them but not in the other.
Trying out moderated user tests —
The first, moderated tests, are conducted with the researcher acting as a real-time moderator in a separate location from the participant. This real-time communication is made possible with web conferencing/screen sharing tools, and leads to a shared ‘virtual space’ where the researcher can observe and communicate with the participant at any time. For clarification purposes, the following is a possible interaction between a moderator and a participant:
- Moderator provides participant with tasks or activities to complete
- Participant goes through tasks while thinking out loud
- Moderator asks a question for clarification (only when necessary)
- Participant replies and moderator takes note of what was said
Or a tad more specific:
- Moderator provides participant with tasks or activities to complete
- Participant says, “Oh, I thought that would do something completely different.”
- Moderator responds, “What did you think was going to happen?”
- Participant replies and moderator takes note of what was said
This, the ability to interact with the participant, is one of the biggest draws of moderated RUTs because it gives the researcher the ability to flesh things out, explain unclear tasks, and steer the participant back on track if they get sidetracked. Keep in mind that this advantage can also be a disadvantage with an inexperienced moderator, as they can influence the participant’s behavior by asking leading or too many questions (this leads to skewed data).
With an idea of what a moderated RUT is, how about we cover some instances in which you’re likely to choose it over its unmoderated companion? Yes? Good. A moderated RUT is best when:
- The test includes tasks that require follow-up questions
- Tasks build off of one another → the participant needs to complete one task before moving on to another
- If the participant is stuck in a task in an unmoderated RUT, he/she can’t complete the test
- Security concerns make it impossible for the participant to access the product without a moderator’s supervision
Upon completion of your observations, think about implementing tips on creating a seamless user journey through UX.
Turning towards unmoderated tests —
With moderated tests under our belt, we turn to unmoderated tests. If you guessed that these differ from their moderated counterpart in that there’s no moderator (virtually) present, you’re right! Tackling this design means that you’ll have to use a program to automate the test and provide the participant with pre-determined tasks to complete and record (the participant typically thinks out loud as they work through the tasks).
As it’s unmoderated and there’s no real-time connection, participants can decide when and where to complete the tasks (part of the attraction of unmoderated RUTs). What the researcher can do, though, is ask follow up questions to clarify or further explain anything that the participant did/said.
So, why would you opt for this over a moderated test? Well, because the researcher need not be present, multiple sessions can be conducted simultaneously, hence a much larger sample size. There are also occasions where tasks are quite specific or direct and require no direction from a moderator, such as a study with an interaction design in which the participant is tasked with finding the specific price of a number of items.
What Tools Do You Need?
Besides the difference in real-time communication tools between moderated and unmoderated RUTs such as GoToMeeting, Google Hangouts and Skype, there’s not a lot to say about the tools because they can change wildly from test to test.
Let’s see, you might need some tools to recruit participants, others for recording, some to track analytics (real-time tracking if you’re using a moderated design), and so on. Don’t worry if you’re confused as to which you’ll need, I’ll tell you the one I use and trust the most when it comes to remote user tests — CanvasFlip. It covers most of the touch points anyone needs for user testing. Its suitable for user testing whether you have a existing user base or even when you don’t have it. It records all the user behaviour data you need, helps you track the analytics. When you’re on a budget, and when in doubt about your designs, CanvasFlip is the best tool you should opt for.
How Many Users and from Where Will You Recruit?
There’s no best number of participants to use, but a general best practice is ‘the more the merrier.’ More participant’s means more data, so always try for the largest sample size you can muster. An avenue you can pursue is to recruit users into separate target groups that differ based on demographics for well-rounded results.
Don’t forget about location either. It doesn’t make sense to recruit users from Miami if you have an app that has to do with cold weather (weird example, I know), so always think about who your actual end-users are.
How are You Going to Measure Results?
Time for results. There’s not a lot to say about this section either because your metrics will vary depending on the app/program/etc. you’re testing. All there is to say is pay attention to your participant(s) and focus on UX.
I use CanvasFlip and measuring results is quite simple because of the varied forms of user behaviour data I get [They have those listed here].
To Test or Not to Test
To test or not to test, that is the question. Now that we have Hamlet out of the way, let’s do a quick recap. RUTs are unbelievably good for capturing real responses from actual users in their natural environments. If live (moderated RUT), you can observe actual behaviour as it unfolds, and if your participant finds a flaw, you can switch gears and focus on it. Don’t know how? No problem, you can always go to an agency with UX experience to fix up any problems.
That being said, good luck for your testing!
