One method of determining the effectiveness of an A/B test platform is to perform an A/A test. This means that you create two or more identical variations and run an A/B test to see how the platform handles the variations. Successful results should show that both variations yield very similar results.
Conducting an A/A Test
- Create a new Custom Code campaign by going to Site Personalization › New Campaign › Custom Code.
- Give your campaign a name, do not change any other settings, and click Next.
- In the targeting tab, click Next without changing any settings.
- In the variation tab, click New Variation and select the Custom Code template.
- In the JS tab, add the following code.
console.log('A/A test variation A'); - Click Save Variation.
- Click New Variation and create a second variation with the following code in the JS tab:
console.log('A/A test variation B'); - After saving the second variation, use the Allocation column to allocate 50% of the traffic to each variation.
- Use the default settings for primary metric (purchases). Do not change the default advanced settings which are sticky for the user (multi-session), an attribution window that starts when the variation is served, and ends when the session ends.
- Click Next and set the experience status to Active.
- Click Save Experience and Publish.
- In the campaign list, click more options next to the campaign you just created and select Duplicate. Do this 9 times so that you have a total of 10 A/A tests. This will minimize any effect of randomness.
- Allow the test to run for a full week before assessing the results.
- After the test and the analysis are complete, archive the campaigns.
Analyzing A/A Test Results
After the test has run for a full week, analyze results according to the following parameters:
- Variation allocation: view the number of users that were allocated to each variation to verify that allocation is indeed random.
Good result: each variation was served to 48% – 52% of users.
Bad result: each variation was served to less than 48% or more than 52% of users. - Data collection: compare the number of purchases and revenue in the campaign report to your primary analytics platform.
Good result: less than 5% discrepancy in the number of purchases and revenue.
Bad result: more than 5% discrepancy in the number of purchases and revenue. - Probability to Be Best score: look at 2 metrics that did not affect each other directly. For eCommerce, we recommend add to cart and purchase. In each campaign, count how many variations have a score greater than 95% for one of these metrics. This is a false winner, but should occur around 5% of the time statistically due to random chance.
Good result: 8 or more campaigns did not have significant results (>95% score) for the 2 selected metrics.
Bad result: 2 or more campaign did have significant results (>95% score) for the 2 selected metrics.
FAQ
Why should I run 10 tests?
95% probability to be best score is considered reliable, however this also means that 5% of the time a false winner will be declared in an A/A test. If you run a single test that eventually reaches statistical significance, you will not know if it’s an outlier (1 of 20 chance) or a real issue with the platform. Looking at 3 metrics across 10 tests (overall 30 metrics), reduces the effect of randomness on the test.
What to do if an A/A test failed?
It depends on which metrics were problematic:
- Data collection: If there was a discrepancy between Dynamic Yield and your analytics platform of more than 5%, verify that your events are implemented correctly. For details, see Validating Events Implementation.
- Probability to be Best: If there were 4-6 metrics that had a variation with more a score greater than 95%, let the test run for a second week before determining the results. If more than 6 metrics has a variation with a score greater than 95%, contact Customer Support.
- Variation allocation: If the allocation of was not between 48-52% for most variations, contact Custom Support
What to do after an A/A test succeeded?
Simply archive all of the campaigns and start creating some real A/B tests!