Skip to main content

A/B testing

Have you ever been sitting around deciding what to have for dinner, trapped with indecision between two amazing options — pizza or burgers (we are very American, we know)?

It’s the same with deciding what text to use in the Title of your nudge, or what Recommendation Set to use in Helphub. Sometimes you have two great ideas.

That’s where A/B testing comes in.

When should you A/B test using CommandBar​

CommandBar’s A/B testing feature exists for two primary reasons:

  • To allow you to optimize individual CommandBar experiences.
  • To allow you to measure the value an experience or experience configuration is creating.

In the first scenario, you’ll create two versions of the same experience, and show each to 50% of eligible users. This is called an A/B test.

In the second, you’ll make a change available to 50% of users, and observe how their behavior differs from the control group. This is called a hold-out test.

When should you NOT A/B test using CommandBar​

CommandBar’s A/B testing feature is not meant to replace a full-on A/B testing platform like Optimizely, LaunchDarkly, Eppo, or PostHog. In general, if you want to run experiments that involve both CommandBar and non-CommandBar experiences (like “I want to turn this feature on and show a nudge to introduce some of my users to it”), you should use a best-of-breed tool designed for that purpose.

How to run an A/B test using CommandBar​

To run an A/B test using CommandBar, you can use the “A/B test” rule type. You can either apply this to an individual experience in the Who section of the targeting tab. Or you can apply it to an Audience and then use that Audience in your experience (this route allows you to run A/B tests that involve multiple experiences — such as two nudges).

For the second half of your A/B test, you can apply the “B” group the same test group.

Let’s do an example. Say we want to see which nudge is more effective for achieving a goal. In the first of these nudges (it doesn’t matter which), open up the targeting tab and navigate to the Who (read more about our Who/Where/When targeting system tab. From here, add a custom condition to your audience. For it’s type, elect “A/B test”. Then, in the text box to the right, give your test group a name (don’t fret over what to call it — the name is only used to track A/B test groups between experiences, so you can make sure to use the right one). Then select the “A” group.

Targeting tab

If you’re testing two versions of the same nudge to determine which is more effective, you can do the following:

  1. Create the first version of the nudge you want to test
  2. Create an A/B testing group
  3. Apply group A to the Who section of the nudge
  4. Duplicate the nudge
  5. Change the A/B test group to group B

How to run a holdout test using CommandBar​

The process for running a holdout test in CommandBar is very similar to an A/B test. The only difference is that you will only use the A group for targeting. The B group will experience nothing (just a deep empty void).

Renaming test groups​

Once you’ve published a nudge whose targeting conditions involve a test group, you shouldn’t rename the test group. This will cause users to be reassigned randomly to the two halves of the group, which could ruin your test if it is already in progress.

Testing experiences that involve A/B testing​

When using simulate mode to test a nudge that has an A/B test applied to it, the audience opt-in will be toggled off by default (so that you don’t end up in a situation where you’re simulating a nudge but end up in the audience subgroup that isn’t supposed to receive it).