Many of our customers will be familiar with running experiments like A/B tests on their websites, which can often be facilitated by fairly simple drop-in solutions. This has always been more difficult in-apps, due to the more complex nature of the code and expectation for a polished, consistent user experience. However as publishers are increasingly seeing their apps as important acquisition channels, rather than purely a retention tool, we've been hearing more and more interest.

As a result, Pugpig Bolt now supports A/B testing! This feature is in its early stages and we have plans to substantially expand its capabilities in terms of what can be experimented with. This doc lays out what's currently possible and how it works.

A/B testing in Bolt

We've built the A/B testing dashboard into Distribution, this means it leverages the exact same functionality we use to build your apps to create these experiments, but with a new UI that's designed to help you do so quickly and easily.

Also, a note on terminology: while we're generally referring to this as A/B testing, we actually support multivariate testing at launch (A/B/n). 

Our Product goals here were to build a fully-functional vertical slice and get customers involved early. As such, testing is currently limited to the metered paywall. We felt the question of how changes to a user's metered allowance can impact their chance to convert was a very typical (and potentially powerful) use case. This includes the ability to change the number of articles included, whether and how often it resets and the wording of the CTA.

Our plans are then to swiftly expand this to other elements of the app config, focusing on high-leverage elements first. This is where we'd love your feedback.

We've worked to make this as intuitive as possible, fields have tooltips explaining what they'll change and we'll warn you of potentially destructive behaviours, such as editing an inflight experiment. We'd love any feedback on this as well as we work to build a tool that you can pick up and use with little under-the-hood knowledge. Our support team are always on hand to help guide you, as ever.

User buckets and experiment roll out

Variant A of your test will always be the currently active config of your app, making this the control variant. At this stage, users will be split evenly between variants but we'd love to know if more fine-grained control over bucket sizes would be helpful or important.

Once an experiment is launched via the UI, separate config files for each variant will be created. Then, the next time a user opens the app they'll be placed into a bucket and download the relevant configuration. This will then take affect the next time the app is opened.

We're aware that this requirement means experimentation will only kick in after a user's next session and this can limit the efficacy of testing things that are more important/informative from a user's first session. We're looking to make a platform change to allow new configs to be downloaded and take affect immediately. Expect to hear more from us about this change early in the new year. 

Viewing the results

We've leveraged our existing analytics framework for A/B testing, this means right out of the gate you have access to any of our standard events and properties, enabling rich analysis post-facto, and not requiring you to box yourself into a particular metric pre-test.

Whilst most analytics providers can thus be used to analyse experiment results, we've primarily build this with Mixpanel in mind. All enterprise customers have access to Mixpanel, in the form of Pugpig Advanced Insights. Mixpanel is a best-in-class product analysis tool and provides a powerful, easy-to-use reporting suite that will allow you to see the results of your experiments, and drill down to understand what's driving those results.

We send both the experiment name (pugpigExperimentName) and the bucket the user is in (pugpigExperimentVariantName) as properties on all events, enabling easy segmentation of these users.

Going forwards

As we're still in the early stages of this toolset we'd love to hear from you if this is something you are interested in. Here are some questions we'd be keen to hear your thoughts on:

• What kinds of hypotheses/changes/areas of the app would you want to be able to test? 
• Do you already have expertise in-house on running experiments and analysing the outcomes? 

Note: The initial roadmap of this toolset is the ability A/B test app elements (e.g. paywall copy/button colours etc) rather than anything in your content. That's still an area we're interested in, and would like to hear your thoughts on, but is a notably different technical proposition that would require closer coordination with your CMS and workflows.