How to Get the Most out of a Product Beta Phase

There are typically 3 reasons why product teams decide to introduce a beta phase into their product development cycle:

1. To soft launch new products
2. To test new ideas
3. To migrate from existing platforms to new ones and sunset old products

Soft launching new products

Startups and large corporates alike will often release a version of a product with a beta label attached as a way to ‘soft launch’ a product and collect feedback with the understanding that the product is still in a testing phase. An MVP with a clear label attached helps to manage expectations.

Testing new ideas

Before fully committing to a new product, larger corporates may want to keep a product in a beta phase to test the new product or idea in a fully controlled setting.

Migrating from old platforms to new ones

In companies who struggle with tech debt and legacy platforms, migrating away from the existing tech stack and onto a new platform can often be the only way to eliminate the tech debt that’s been accrued. This typically means building a parallel version of the product and launching a beta to a set of customers before making a full switch later on.

If your answers to these questions suggest you need a beta, go for it. If not, there may be reasons to forget about the potential complexities of a beta and instead opt for something like a feature flag to give you control over who sees what.

OK, so let’s assume you’ve decided to go ahead with your plans to create a beta phase for your product. How can you and your product team get the most out of your beta phase? Here’s some ideas to help.

1. Make sure customers know what a beta is

First up, don’t assume customers know what the word beta actually means! Sure, if you’re a product manager at Github it’s likely that your users will understand the concept of a beta but if you’re an ecommerce product, your users are more interested in the contents of their shopping basket than your product strategy.

At the start of any beta phase, speak to customers clearly about what the beta phase means and provide clear definitions so that they understand what to expect.

Managing customer expectations

Creating a landing / information page which customers can easily refer to throughout your beta phase will help to manage their expectations.

Users might want to know why a specific feature is missing, for example, so spending the time to anticipate potential questions and popping these onto an information page beforehand will save you a bunch of time later on.

You’ll probably also want to explain how users can provide feedback throughout the beta phase – more on this later.

2. Select your beta customers strategically

One of the reasons product teams decide to launch a beta is because it provides them with an opportunity to collect feedback directly from customers.

However, users who opt in to take part in a beta program are typically happier with risk and are early adopters. This makes it easier to manage those customers since they may be used to taking part in beta programs.

What this also means however, is that these users are not necessarily a representative sample of your customer base and whilst they might be happy to live with a lack of features or a quirk, not all customers will.

That’s why if it’s possible, try to select your beta customers strategically. This means getting a broad representative sample of users who may or may not be happy with your beta product.

For example, if you’re working on a replatform of an existing product, look at your data and select a broad variety of customers based on their usage – and their likelihood to be happy with the new version that’s in beta.

Example sample of customers for a SaaS product

Here’s an example of the kind of users you could decide to select as a broad representative sample for a beta program:

1. Deep power users – make heavy use of >60% of all features available
2. Shallow power users – make regular use of <30% of all features but do so regularly
3. Active users – have activated their account but aren’t as engaged as power users
4. Lapsed users – did use your product but have since stopped using it as regularly as they used to
5. Inactive users – no longer use your product but are still registered

3. Collect structured metrics, feedback and engage with users

Metrics

Collecting behavioural metrics during your beta phase is essential. You’ll need to understand how users are actually using the beta version of your product.

Decide which metrics are important to measure and why.

If your beta product is a migration, you’ll need to run 2 separate sets of analytics platforms and understand which data refers to which version of your product is essential.

Loop in with your data scientists and analytics teams to ensure you have the tracking in place before your beta launch. Keep beta data separate from other data so that you have a clean way to track user behaviour.

Make collecting feedback easy but structured

Collecting feedback is an essential part of a product’s beta phase. You can use your beta phase to understand what’s working and what’s not.

Don’t make submitting feedback difficult.

One of the easiest ways to make feedback easier is to put a visible button on your product so that customers can click it and submit feedback.

Ease isn’t the only consideration though, as you’ll want to also keep feedback as structured as possible so that you can measure the metrics that will decide whether or not you’ll move out of beta.

Structure your data with forms but also provide plenty of opportunity for qualitative feedback.

Engage with users

Making it easy for customers to send you information about their beta feedback is one aspect of the feedback process but it’s also incumbent upon product teams to actively reach out to users during the beta phase. Not all users will want to take time out of their day to tell you what’s wrong with your product so finding a balance between incoming feedback and proactive feedback is essential.

During your beta phase, you might set up a call with a customer to understand how they’re getting on (and you can of course thank them for taking part by offering vouchers etc – just be careful not to let this distort their feedback too much!).

You could also create an interactive space such as a Slack channel or forum where beta users can send feedback and get regular updates.

If a bug was raised by a beta customer for example, using these channels to let them know it’s been fixed helps to ensure they feel valued and that they’re helping you to ultimately build a better experience for them as customers.

4. Grant parallel access for re-platforming beta phases

If the beta version of your product is the only version of your product then clearly this won’t be possible, but if your beta is the new version of an existing product then granting parallel access is essential.

What do we mean by granting parallel access?

Example – Google Analytics 4

Google recently rolled out the new version of their Analytics product, GA4 and to do this they’ve created 2 instances of Google Analytics – one which works with the existing ‘universal’ version of GA and one which is a separate instance of GA, named GA4.

During this transition phase, users are granted parallel access to both of their Google Analytics instances and they’re able to swap between the two at any time.

Giving users parallel access throughout your beta phase has a number of benefits, but perhaps the most powerful benefit is that in granting parallel access you’re essentially de-risking the leap from the existing product to trying out the beta product to zero.

In the worst case scenario, users know they can stick to the existing product for now, and that the beta product is essentially a trial.

This is likely to result in more users being willing to try your beta in the first place – and helps you to widen the scope of your beta to the different types of users we discussed earlier.

However, making parallel access easy means that users might take one look at the beta and then decide to stick to what they know best: the existing product! Adding a little bit of friction into your beta UX might help you to manage this more effectively, though.

Signpost

Speaking of UX, it’s very important during a beta phase to signpost a few things to help users navigate:

1. Whether or not they’ve opted into the beta
2. Whether they’re currently in the beta product or, in the case of a migration, on the existing product
3. How to opt out

The easiest way to let users know whether or not they’re in a beta version of a product is to attach the beta label clearly onto the logo of the name. This is a tried and tested way to help users understand where they are.

5. Put together your criteria for moving out of beta

Finally, a critical aspect of your beta is the ability to understand when you and your users might be able to transition a product out of beta and into a wider, official release.

As we’ve said, not all product releases need a beta – and plenty of products will be released as first iterations, but betas are helpful when you’re dealing with potentially high risk changes or launches that require additional input from customers before you’re happy to launch more widely.

In order to move out of beta you need to know what your criteria is for doing so.

You can put together your end of beta criteria before you launch your beta which helps you to structure customer feedback. For example, if one of your criteria is that X% of customers have activated on the new product and X% have said they’re happy to use it, these could be some success criteria that you decide beforehand.

However, one of the key benefits of running a beta phase is to understand how users interact with your product. This means pre-emptive criteria setting may not be appropriate since you don’t yet know what criteria matters.

For example, you launch a beta and you quickly find that more users are accessing the product on a mobile device than you expected. One criteria for moving out of beta could now be to ensure the site is responsive or to build a first iteration of a mobile app version to cater to these needs.

The perils of feature parity

A migration beta phase will inevitably lead to the question of ‘feature parity’. If you’re replatforming a product from one tech stack to another or building an entirely new proposition designed to replace the old one, it’s clearly sensible to consider feature parity as a criteria for ending your beta.

But, as with all things product, it pays to be sensible. Feature parity may not be relevant in your new product since you may have explicitly built the new version of your product without many of the features that existed before.

This is likely to annoy existing customers – and stakeholders, too. In a B2B SaaS context for example, if you plan to move customers over to a new version of your product and it’s missing core features, it could cause customers to leave.

Arm yourself with as much data as possible so that you know exactly which features in the previous version of the product were used and by who. A product manager’s role is to critically assess which features are essential, which ones are unnecessary bloat and which ones haven’t yet been created which could propel your product to even greater success.

Aiming only for ‘feature parity’ limits your thinking to what you previously built, so it’s wise to strike a balance.

Getting agreement

Once you’ve decided on sensible criteria for moving out of beta re-evaluate it at each step and when you think you’ve hit the targets, get agreement from stakeholders to move out of the beta and into a wider launch.

This can be tricky, of course, but product governance forums like the Product Council can help with this.

Bringing it all together

When managed effectively, running a product beta can be a lot of fun.

You get to create a low risk playground where users with managed expectations give you feedback and tell you what’s working and what’s not.

You get to try new ideas, work with new tech stacks and figure out how to keep customers happy.

Fundamentally, a beta is an opportunity to test hypotheses, learn how users interact with your product and build close relationships with your customers throughout the process.

And that’s ultimately what we all love most about product management, isn’t it?