How to Design Experiments for Your Product

Google Jedi Marissa Mayer allegedly tested multiple variations of the color blue until she found the perfect shade of blue to increase conversions at Google. This kind of experimentation may be reasonable when you have tens of millions of visitors a day and can achieve statistically significant results in seconds, but many of us don’t.

Experiments can provide valuable insights for your product. However, many companies run experiments for the sake of running experiments without gaining any real insights which propel the product forward or achieve any interesting, insightful results. Merely uttering the words ‘A/B test’ is sometimes seen as an achievement in itself but without truly learning anything from your test, testing can be a waste of time and precious development effort.

In order to learn from your experiments, you need to spend time upfront creating experiments that will deliver conclusive results at the end. Changing button colors may give you incremental improvements but if you’re not Google or you don’t have the resources to conduct hundreds of concurrent experiments at the same time, you’re probably better off focusing on a few distinct experiments which answer interesting questions and provide you with real, tangible insights that can drive long term improvements or achieve your product goals.

How experiments fit into the overall picture

Product managers are busy people. And there’s a temptation to run experiments to merely demonstrate to people in your organisation that you’re running experiments. This is mostly a waste of your time. You’re better off not running any experiments than wasting precious planning / development time on irrelevant, vanity experimentation.

Since we’re such busy folks, experiments should be run as genuine pieces of scientific, investigative work to try and find insights into problems and answers to interesting questions which help you to achieve your goals.

Experiments should supplement not supplant your product OKRs, goals and roadmaps. They are a tool which assist you along the path to achieving goals and have little inherent value to your users by themselves.

Your OKRs, backlog and roadmap remain your focus and delivering a product which your customers love remains your primary purpose as a product manager.

Armed with results from a genuinely insightful experiment, you can inform your roadmap, your team and your stakeholders and reassess your goals with the insights you’ve gained.

For startups, your entire business is an experiment. You need to use experimental discovery methods and processes to find product / market fit. Often, startups do not have the traffic required to generate results that are statistically significant in any meaningful way so if you’re a product manager working at a startup it may be best to steer away from optimisation experiments such as changing text and button colors and instead focus on your overall goals and experimenting with your value proposition itself to ensure you’re delivering value and building a viable product.

For larger corporates, experimentation can be large and small scale at a departmental level and at a board level. Large companies have the luxury to experiment on a massive scale and test entire end to end products like Amazon Dash or Echo. Similarly, they also have the ability to conduct micro experiments at the departmental level. If you’re a product manager who is not operating at the board level, you’ll most likely be working on smaller scale experiments to optimise your product.

How to conduct experiments

There are several steps involved in the process of designing, prioritising, conducting and implementing experiments and the results of experimentation.

1. Ask questions and generate ideas
2. Create a testable hypothesis
3. Conduct your experiment
4. Communicate your results
5. Prioritise your results

1. Ask questions and generate ideas

All experiments start with a question. Your experiment is an attempt to answer your question through a series of hypotheses.

The question you ask should be linked to your product goals. For example, if your goal is to achieve a 10% visit to lead conversion rate on your sign up page, you could ask the question ‘what can we do to change the sign up form to increase the conversion rate?’.

If a product goal in an ecommerce product is to generate 15% more revenue year on year, your question could be ‘what is the best way to increase the average order value per customer from $20 to $30?’.

Questions are a form of ideation and which help to create hypotheses which can be tested.

The questions and ideas that you’re interested in exploring should ideally come not only from your own head but also from your stakeholders from across the business.

As a product manager, you’ll no doubt be in constant communication with many stakeholders across the business who (hopefully) know what’s on the roadmap and why.

How to involve stakeholders

If you do decide to conduct experiments, make sure you involve stakeholders at this stage and get them involved in the process of generating hypotheses by asking questions that are relevant to both them and to your product goals.

A simple way to do this is to create an experiment backlog which can either be a spreadsheet accessible by all or a Trello board where stakeholders can dump their ideas quickly and efficiently.

Decide on a structure for your experiment backlog and communicate the structure clearly to all your stakeholders. Make it clear that the structure must be adhered to for the experiment to be prioritised / understood effectively.

Stakeholders are busy too so ensure the structure you decide upon is easily understood by everyone. If you make the process of suggesting an experiment difficult you’re already setting yourself up for a fail. Keep it simple.

With a backlog prepared, you’ll need to prioritise.

Once your stakeholders have been involved in the idea generation and question generation process, you will then need to prioritise which questions / ideas to take forward.

You may have your own methods for prioritising experiments and there is no 1 definitive answer on the best methods for prioritising experiments for your product.

Here are some methods to help you prioritise your experiment ideas.

You’ll no doubt use similar prioritisation frameworks for prioritising your overall backlog, so use the same principles here. As I said, much of this is guesswork since that’s how scientific experimentation works, but try your best to put together a prioritisation process that works for you.
Involve stakeholders at this stage if you think it’s appropriate. Work with stakeholders to determine which experiments to tackle. If needs be, get everyone together on a call or in a meeting so that you can discuss which of the ideas will be prioritised first.

Whilst stakeholders should have a say in what goes into the backlog, the product manager should make the final decision on prioritisation. Stakeholders will be more interested in the prioritisation of your roadmap and less interested in your experimentation backlog so if you think involving them in this stage might just confuse things, just crack on yourself and ping around a note instead to explain which experiments will be prioritised and why.

2. Create a hypothesis

If you’re not from a particularly scientific background (and many product managers are not), you might feel slightly uncomfortable with some of the language bandied about around when creating experiments.

The most important term to understand fully is the concept of a hypothesis. A hypothesis is a testable statement which can be proved or disproved. It’s an educated guess or a prediction about the relationship between 2 variables.

There are a number of ways you can create a hypothesis using general statements ,analysing non-directional, directional, causal and relational propositions but unless your role is that of a full time data scientist and taking into consideration that we’re all busy bees I’d recommend sticking to the simplest way of creating hypotheses which is to use the ‘if then’ statement.

What’s an if then statement? Let’s look at some examples.

The if then statements clearly articulate your hypothesis and your prediction. In these examples we have 2 variables which will allow you to create your experiment. Variable 1 is known as the independent variable – since it’s not dependent on the other. The variable 2 is known as the dependent variable, since if you change the independent variable, this will change too. It is dependent on the independent variable.

One way to know whether you’ve created a strong hypothesis is to distill your hypothesis down into the 2 variables that will affect the outcome of your experiment.

Let’s look at an example again.

The 2 variables in this statement are the number of buttons on the homepage and the number of registrations. Your hypothesis is suggesting that the number of registrations is dependent upon the number of buttons on the homepage and is therefore the dependent variable.

You can also add a ‘because’ clause to your if statements to article the rationale which supports your hypothesis. An example could be:

You don’t have to use an ‘if then’ statement to form your hypothesis if you don’t want to; so long as your hypothesis is a statement which has variables that can be measured in experiments your hypothesis will still be valid.

At this point, you will have generated ideas and interesting questions to ask and created some hypotheses on the basis of those questions.

For example, you may start by asking this question:

This results in a number of hypotheses:

1. If the button was blue then we’d increase registrations
2. If there was only 1 button on the homepage then we’d increase the registration rate because the user would only have 1 choice
3. If there was a pop up form then we’d increase registrations because we’ve seen in previous experiments that pop up forms have a conversion rate of 10%

Once you have a list of hypotheses, these will then also need to be prioritised. You cannot and should not attempt to test every possible hypothesis. As with prioritising ideas / questions, there is no 1 single framework to use for prioritising hypotheses. You should consider:

Once you decide on a hypothesis to test, you’ll then start the fun of actually conducting the experiment.

3. Conduct your experiment

There are 2 common ways to run product experiments:

• A/B testing
• Multivariate testing

There are of course, plenty of other ways to run experiments, but these are the 2 most common so we’ll stick to these.

A/B testing involves 2 variations of the same page or feature: version A and version B. Multivariate testing involves testing multiple variations of the same page to identify which is the best performing.

Your experiment will consist of a control group, which is the group of people exposed to your existing page, and an experiment group(s) which is the group of people exposed to a new version or versions of your page or feature. The new version of will be created with your hypothesis in mind.

Aim to be as bold as you possibly can be with your experiment. Don’t always go for incremental, small changes that will result in 1 or 2% uplifts. Try completely changing the layout of your page; remove elements, add elements and be quite drastic to try and achieve drastic results.

You may find some resistance internally to experimenting with bold changes but in order to achieve radical results you often have to try radical experiments. Changing button colors alone is fine – and safe – if you’re looking for small optimizations across high volume sites but it’s through bold experimentation that you’ll learn more.

Let’s look at an example.

Your experiment would test your hypothesis and at the end of your experiment you would have a result. Your result would tell you whether your hypothesis was true or not.

There is a downside to running bolder experiments, which is that bold experiments mean more time, effort and resource since it’s not as simple as just changing a shade of button. If you do decide to run an experiment, it is worth throwing some resource behind putting together a real alternative to the current solution.

Your backlog and the features you build are all essentially experiments; you do not know what will happen when you build a new feature and it only through building the lightest touch version of what you believe to be a good idea that you ever find out whether it is or not. Whilst your stakeholders might be reluctant to throw time and effort into experimentation, remind them that everything you build is, to some extent, an experiment.

Some companies have their own experimentation platforms or indeed separate experimentation teams, which is nice.

If you don’t have the luxury of your own custom built experimentation platform here’s a selection of options to use for conducting your experiments:

• Optimizely – perhaps the most widely known experimentation platform, Optimizely only needs a javascript snippet to run. It has a detailed reporting dashboard and a WYSIWYG editor which makes making changes simple.
• Google Analytics – allows you to set up experiments for free. Fab.
• Omniture – a paid for analytics tool which has experiment functionality built in
• Traffic driven marketing – splitting your advertising budget across 2 different landing pages is a light weight approach to running experiments
• In house traffic splitting – whilst you might not have the time or resource to build your own experimentation platform, speak to your engineers anyway to see if there’s a light touch, creative way to split certain traffic across 2 different pages.
• Feature flagging – sometimes your engineers can enable a feature flag for a specific feature and enable it only to a specific percentage of your audience. This helps you to experiment with or test a feature before it goes live.

The key is to ensure you can measure all the metrics that matter to this experiment, which are typically one of the variables in your hypothesis. If you’ve created an experiment but you aren’t able to measure it, it isn’t an experiment. Make sure the tools you use to create your experiment allow you to measure its results!

How to test ideas without too much development effort

To reduce potential waste, try your best to keep the experiment as lean as possible, without dragging resources away from other important pieces of work to build things that might not actually work.

Some tips on how to do this include:

4. Communicate your findings and close the loops

Once your experiment demonstrates tangible, statistically significant results, it’s no use keeping the findings to yourself. Go back to your stakeholders and the wider business and articulate clearly what the result of the experiment was and what the next steps could be.

Share a few data points but don’t overwhelm everyone. Keep it clear, succinct and back it up with a graph or two. If possible, speak up in your cross functional meetings or weekly all hands if you have them.

Closing the loop of experiments is essential; your stakeholders need to know that their input is being listened to and that the experiments that have been run have led to tangible results which can be assessed and prioritised accordingly.

If you were running an A / B test between different pages, this would mean keeping the winning version. If you were testing a new feature on a segment of your audience, this would mean gradually rolling out the feature to the remaining users. If you were testing new pricing models, this would mean taking the plunge and changing the prices across the board.

It’s likely that before you can make a final decision on some of the experiments you’ve been running you’ll need to consult with at least one, if not many of the stakeholders you were engaged with at the beginning. This is not always the case but is for larger scale experiments which impact other parts of your business.

If you’re a plucky startup you can typically go ahead and roll out the better performing page or feature, but in larger organisations closing the loop with stakeholders by articulating what the next steps are is crucial.

You may need additional development work to implement the feature or page you were testing. If this is the case, prioritise the work as you would with any other work. Don’t be tempted to put it to the top of the backlog because you’ve uncovered something which you or your stakeholders personally find exciting or interesting or because it proves a hypothesis you created. Whilst you may have learned a lot from your experiment, that doesn’t mean you have to bypass the rest of your prioritised backlog and get cracking on the work immediately.

Keep an archive / record of your experiment results so that you avoid needlessly repeating the same experiment twice!