Shipping Your Product in Iterations: A Guide to Hypothesis Testing

A look at the App Store on any phone will reveal that most installed apps have had updates released within the last week. A website visit after a few weeks might show some changes in the layout, user experience, or copy.

Today, software is shipped in iterations to validate assumptions and the product hypothesis about what makes a better user experience. At any given time, companies like booking.com (where I worked before) run hundreds of A/B tests on their sites for this very purpose.

For applications delivered over the internet, there is no need to decide on every aspect of a product far in advance, and then build and eventually ship it. Instead, it is perfectly practical to release small changes that deliver value to users as they are being implemented, removing the need to make assumptions about user preferences and ideal solutions—for every assumption and hypothesis can be validated by designing a test to isolate the effect of each change.

In addition to delivering continuous value through improvements, this approach allows a product team to gather continuous feedback from users and then course-correct as needed. Regularly creating and testing hypotheses is a cheaper and easier way to build an iterative approach to creating product value.

What Is Hypothesis Testing in Product Management?

Throughout the process of discovering and delivering product improvements, it is imperative to validate assumptions about user needs, design, and features in order to understand their real-world impact. This helps teams determine whether proposed changes contribute to the desired user and business outcomes, instead of treating feature delivery as a measure of success in itself.

This validation is traditionally done through product hypothesis testing, during which the experimenter outlines a hypothesis for a change and then defines success. For instance, if a data product manager at a business analytics software company like Tableau has a hypothesis that adding an AI-generated performance summary to a dashboard will increase the number of users who explore the underlying report, then success is defined by a higher click-through rate from the dashboard to the report.

One of the key aspects of hypothesis testing is the isolation of different variables in the product experience in order to be able to attribute success (or failure) to the changes made. So, if our product manager had a further hypothesis that placing the AI-generated performance summary more prominently on the dashboard would increase the click-through rate to the underlying report, it would not be possible to test both hypotheses at the same time. Doing so would result in failure to properly attribute causes and effects; therefore, the two changes must be isolated and tested individually.

Thus, product decisions on features should be backed by hypothesis testing to validate their performance.

Different Types of Hypothesis Testing

A/B Testing

One of the most common use cases to achieve hypothesis validation is randomized A/B testing, in which a change or feature is released at random to one-half of users (A) and withheld from the other half (B). Returning to the hypothesis that an AI-generated performance summary would increase the number of users who explore the underlying report, one-half of users will be shown the summary, while the other half will see the dashboard as it was before. The click-through rate from the dashboard to the report will then be measured for each group (A and B) and compared. In case of a significant increase in the click-through rate for the group shown the AI-generated summary, the conclusion would be that the original hypothesis was correct, and the change can be rolled out to all users.

Multivariate Testing

Ideally, each variable should be isolated and tested separately so as to conclusively attribute changes. However, such a sequential approach to testing can be very slow, especially when there are several variables to test. To continue with the example, suppose the product manager wants to test both the placement of the AI-generated performance summary and the amount of detail it provides. Testing each possible combination separately would require several sequential experiments.

Instead, a multivariate test can be adopted, in which users are split into multiple variants that combine changes to these variables. For instance, four groups (A, B, C, D) could be made up of 25% of users each, with each group seeing a different combination of summary placement and level of detail. The click-through rate to the underlying report could then be compared across the groups to determine which combination performs best.

Before/After Testing

Sometimes, it is not possible to split the users in half (or into multiple variants) as there might be network effects in place. For example, if the test involves determining whether one logic for formulating surge prices on Uber is better than another, the drivers cannot be divided into different variants, as the logic takes into account the demand and supply mismatch of the entire city. In such cases, a test will have to compare the effects before the change and after the change in order to arrive at a conclusion.

However, the constraint here is the inability to isolate the effects of seasonality and externality that can differently affect the test and control periods. Suppose a change to the logic that determines surge pricing on Uber is made at time t, such that logic A is used before and logic B is used after. While the effects before and after time t can be compared, there is no guarantee that the effects are solely due to the change in logic. There could have been a difference in demand or other factors between the two time periods that resulted in a difference between the two.

Switchback Testing

The downsides of before/after testing can be overcome to a large extent by deploying switchback testing, a form of time-based on/off testing in which the change is introduced to all users for a certain period of time, turned off for an equal period of time, and then repeated for a longer duration.

For example, in the Uber use case, the change can be shown to drivers on Monday, withdrawn on Tuesday, shown again on Wednesday, and so on, continuing to alternate between the two conditions throughout the test. Switchback tests can also alternate over shorter periods, such as several times within a single day, to reduce the effects of changes in demand and other external factors.

While this method doesn’t fully remove the effects of seasonality and externality, it does reduce them significantly, making such tests more robust.

Test Design

Choosing the right test for the use case at hand is an essential step in validating a hypothesis in the quickest and most robust way. Once the choice is made, the details of the test design can be outlined.

The test design is simply a coherent outline of:

• The hypothesis to be tested: Adding an AI-generated performance summary to the dashboard will increase the number of users who explore the underlying report.
• Success metrics for the test: Click-through rate from the dashboard to the report.
• Decision-making criteria for the test: The test validates the hypothesis if users in the variant show a higher click-through rate than those in the control group.
• Metrics that need to be instrumented to learn from the test: Click-through rate to the underlying report, interactions with the AI-generated summary.

In the case of the product hypothesis example that an AI-generated performance summary will increase the number of users who explore the underlying report, the success metric is click-through rate and the decision criteria is an improvement in click-through rate.

After the right test is chosen and designed, and the success criteria and metrics are identified, the results must be analyzed. To do that, some statistical concepts are necessary.

Sampling

When running tests, it is important to ensure that the two variants picked for the test (A and B) do not have a bias with respect to the success metric. For instance, if the variant that sees the AI-generated performance summary already has a higher click-through rate than the variant that doesn’t see the change, then the test is biased and can lead to wrong conclusions.

In order to ensure no bias in sampling, one can observe the mean and variance for the success metric before the change is introduced.

Significance and Power

Once a difference between the two variants is observed, it is important to determine whether the change observed is an actual effect and not a random one. This can be done by computing the statistical significance of the change in the success metric.

In layman’s terms, the significance level represents the probability of concluding that a change had an effect when it actually did not. Power represents the probability of detecting an effect when one actually exists.

So, tests need to have sufficient power and an appropriate significance level in order to produce reliable results.

While an in-depth exploration of the statistical concepts involved in product management hypothesis testing is out of scope here, the following actions are recommended to enhance knowledge on this front:

• Data analysts and data engineers are usually adept at identifying the right test designs and can guide product managers, so make sure to utilize their expertise early in the process.
• There are numerous online courses on hypothesis testing, A/B testing, and related statistical concepts, such as Udemy, Udacity, and Coursera.
• Using feature management and experimentation platforms such as LaunchDarkly and Statsig can make the process easier, thanks to out-of-the-box capabilities for running tests and progressively rolling out features to selected groups of users.
• For remote and distributed teams, document the hypothesis, test design, results, and conclusions in a shared workspace. This allows everyone involved to review the results of each test and apply what was learned to future hypotheses.

Using Hypothesis Testing for Successful Product Management

In order to continuously deliver value to users, it is imperative to test various hypotheses throughout the discovery and delivery process. Several types of product hypothesis testing can be employed for this purpose, and each hypothesis needs to have an accompanying test design, as described above, in order to conclusively validate or invalidate it.

This approach helps teams continuously learn from users, measure whether new changes and features produce the desired outcomes, and use those learnings to deliver incremental iterations.