Masterful hypotheses

Learn how to construct better product design hypotheses.

Decisions often feel like shots in the dark in the fast-paced product design world. But what if there was a way to illuminate the path forward? Enter hypotheses: the guiding stars of product design & A/B testing. These concise statements inform design decisions, empowering us to measure and communicate design impact clearly and confidently. Join me to explore the art and science of hypothesis formation.

What is a hypothesis?

In product design and specifically, in A/B testing, a hypothesis is a tentative statement saying: If we change X, Y will happen. A hypothesis isn’t a random guess, it is usually based on evidence. It’s an informed prediction based on existing knowledge, insights and observations, a reasonable assumption and a low-risk bet.

The goal of a hypothesis is twofold. Like with life goals, you can either grow or learn so you can try growing better next time. So, product growth is the ultimate goal, but learning is always welcome, even if you did not achieve growth this time.

Why do we need to learn how to form better hypotheses?

1. Informed Decision Making: Hypotheses provide a structured framework for making design decisions based on evidence and insights rather than intuition alone.
2. Clarity and Focus: Crafting hypotheses helps to define clear objectives and focus efforts on specific design changes or improvements.
3. Work Prioritization: Hypotheses help us prioritise design initiatives and allocate resources effectively by focusing on areas with the highest potential for impact.
4. Measurable Outcomes: Hypotheses set clear success criteria, enabling us to measure the impact of design changes and validate their effectiveness through experimentation.
5. Continuous Learning: Through hypothesis testing, we gain valuable insights into user behaviour, preferences, and needs, fostering a culture of constant learning and improvement.
6. Cross-Functional Alignment: Hypotheses can be used as a common language for collaboration across cross-functional teams, aligning stakeholders around shared goals and objectives.
7. Risk Mitigation: By formulating hypotheses, we can identify and address potential risks or assumptions early in the design process, reducing the likelihood of costly mistakes.
8. Creativity and Innovation: Hypotheses might encourage us to think critically and creatively about design challenges, fostering the exploration of new ideas and solutions.

So, let’s find out how to create a useful hypothesis.

The anatomy of a hypothesis

A hypothesis consists of two elements: the description of the design change, the variant, and the predictable outcome, which can be a metric or set of metrics that this change might influence. So, in plain terms, the cause and effect.

Ex. Adding social proof on the upgrade flow equals more paid plan upgrades.

Often, the effect can consist of multiple results.

Ex. Adding social proof on the upgrade flow equals more paid plan upgrades and increased retention.

How to phrase a hypothesis

A hypothesis can be phrased in three ways:

• If… then

Ex. If we add social proof to the upgrade flow, then upgrades to the paid plans will increase.

• Correlation/Effect

Ex. Social proof on the upgrade flow will result in more paid plan upgrades.

• Comparison

Ex. Users who see the social proof on the upgrade flow will be more likely to upgrade to a paid plan than those who are not.

Characteristics of a Successful Hypothesis

• Clear: Articulate about the variable/s and the expected outcome.
• Specific: Focusing on a particular aspect of the design or user journey.
• Testable: Can be explored through experiments and be measured.
• Falsifiable: Can be proven or disproven.
• Insightful: Can produce findings that provide real value and/or learnings.

According to the above, our example hypothesis could turn into something like this:

Ex. Adding notable customer logos on our feature upgrade modals as social proof will result in users trusting the product more, and being more likely to upgrade to a paid plan.

This statement is clear; we can understand the design change and the expected result, and specific; we know where the design change will be applied. We can easily test it by creating a variant with the logos added and comparing it to the control, original design, measuring the average conversion in each group. Depending on the results, we can conclude if the change convinced more people to upgrade or not. If it did, then this would increase our profit/value, and we now know this is a successful strategy that we want to apply to other flows as well. If not, we learned that social proof might help with the product's first impressions, but it is not a deciding factor in paying for it.

Adding more context around the why behind the hypothesis can sometimes be helpful. For example, we know Z & T, so we believe if we do X, Y can happen. Or, when we did Z, Y happened, so we think doing X will also make Y happen.

Ex. In our new user interviews, people who saw the social proof logos on our website perceived our product as more trustworthy and appealing. We know social proof can impact user decisions to purchase a product, and we have also seen competitors using social proof in their upgrade flows. Thus, adding notable customer logos on our feature upgrade modals as social proof can result in more users trusting our product and upgrading to a paid plan.

This approach is convenient and effective when talking to stakeholders for presenting and justifying design decisions to get buy-in, as it provides a clear picture of the why behind the design. It also helps us find gaps or blind spots in our assumptions, indicating we might need to dig deeper for more evidence to build confidence in our design decisions and/or redirect our next steps.

Creating a hypothesis

Creating a hypothesis requires design thinking, reasoning and a little imagination. It can start with the question: What would happen if…?

The initial answer to the question should tap into what is already known about the topic, even if it is not that much yet. But if nothing is known, we should step back, and research the subject to have something more concrete to base our hypothesis on.

Here’s where to look for inspiration and evidence to create a solid product hypothesis or an informed assumption:

1. User Interviews and Observations: Everything starts with user needs and behaviours. Engaging directly with users, observing their interactions, and listening to their stories can reveal hidden pain points and aspirations for hypothesis generation.
2. Quantitative Usage Data: Numbers don’t lie. Analysing patterns, trends, and anomalies in usage data can point to unmet needs or unexpected user behaviours, serving as a solid foundation for a hypothesis.
3. Customer Support Feedback, Social Media and Forums: Interactions between users and customer support can reveal common issues, needs or desires that aren’t immediately obvious. Similarly, community forums or social media platforms where users express their thoughts can be a goldmine for hypothesis generation.
4. Competitor Analysis: Learning from similar products’ successes or mistakes. Your competitors’ design solutions can reveal opportunities for hypotheses that aim to match them or fill their gaps with innovative design solutions.
5. Theoretical Insights and Frameworks: Sometimes, the spark for a hypothesis lies in the basics, like best practices and design principles we use in our daily craft. Applying these lenses to a problem might unveil new angles and opportunities for exploration.
6. Market Research and Trends: Keeping a pulse on the broader market, emerging trends and growing demands can inspire forward-thinking hypotheses that tap into emerging needs and desires before they become mainstream, to help build a relevant and appealing product.
7. Technological Advances and Integrations: New technologies and the potential for integrating them into the existing product can lead to hypotheses around enhancing user experiences, solving problems more efficiently, or creating entirely new functionalities.
8. Behavioural Economics and Psychology: Insights from behavioural economics and psychology can offer a deep understanding of user decision-making processes, motivations, and barriers. This knowledge can inspire hypotheses aimed at designing more engaging and user-friendly products.
9. Creative Brainstorming Sessions: Collaboration with peers where diverse ideas and viewpoints meet can inspire hypothesis generation, producing unique and unexpected propositions. There are no bad ideas, just opportunities to explore, expand, and enhance them into better ones.
10. Cross-Functional Team Insights: Engaging with colleagues from different teams such as marketing, sales and engineering, can uncover insights not visible from the design viewpoint. This cross-pollination of perspectives can spark hypotheses more aligned with business objectives and technical feasibility.
11. Historical Data and Product Lifecycle Analysis: Analysing the lifecycle of similar products or services, including their rise, peak, and decline, can provide insights to form hypotheses related to feature launches, updates, or other product strategies.
12. Environmental and Societal Changes: Shifts in societal norms, ecological concerns and global events can alter user priorities and needs. Hypotheses can address these challenges and opportunities, ensuring designs remain relevant and impactful.

Turning hypotheses into design experiments

With a hypothesis as a guide, the next step is to design experiments to test these assumptions or bets. This phase is about creating controlled conditions to observe, collect data, and ultimately learn from the results. Each experiment is a story waiting to be told, with your hypothesis driving the narrative.

In a nutshell, this is the general process of creating a product experiment:

1. Define and create your variant: Identify what you will change and bring it to life as a design variant/s to test against the original. You have two options here. Incremental small changes take more time but produce more accurate results or a big bang where you change more things to see if their sum moves the needle.
2. Develop your methodology: Choose the most effective way to experiment, considering factors like user segment, sample size, duration and format like user testing, A/B testing or smoke testing, for example.
3. Define your metrics: Understand what you’d like to achieve and note what key metrics you want to move and in which direction.
4. Execute and observe: Carry out the experiment and observe users while measuring all the relevant metrics for the test duration. Document findings and any relevant feedback from customer support, or follow up with users for more qualitative insights.
5. Analyse and conclude: Assess the data against or in favour of your hypothesis. Did the results support your assumption/s? Did they reveal something new or unexpected? Or make no difference? Gather all insights, to either tweak your hypothesis or create a new one based on the learnings.
6. Rinse and repeat. As we learn and explore, we apply, tweak, pivot and grow. The process never ends; there is always a metric to move. So, in other words, never rest, keep challenging what you know, and accept that sometimes you can be wrong. Be humble and open to change.

Conclusion

In the ever-evolving landscape of product design, the ability to formulate impactful hypotheses is not just a skill but a mindset. It is about curiosity, experimentation, and the relentless pursuit of improvement. By embracing the iterative hypothesis process, we can unlock endless possibilities, driving innovation and delivering solutions that resonate with users. Refining this skill can empower us to make more informed decisions and communicate impact, driving meaningful change.