To make sure A/B tests results are clean and reliable, each test is managed in versions. This allows you to make changes to a running test without mixing the pre and post-chage data in a way that would affect the result. When a new test version is created, the allocation is reset as is the Probability to Be Best result. The report will allow you to switch between the current version (default view) or any previous test version that ran in this experience.
Creating a New Test Version
The first test version is created when you launch an A/B test experience for the first time. If you edit the experience, you will be notified that a new test version will be created, or given the choice whether or not to create one.
We recommend creating a new test whenever the changes are expected to affect the test results (e.g. changing the image or copy). For smaller changes that probably won’t affect the results, you can continue using the same test version. For example:
- Changing the tracking URL parameter of a target URL
- Changing trigger from time on page of 15 seconds to time on page of 14 seconds
- Changing the backdrop color of an Overlay
- Changing the status of an experience (e.g. from active to draft)
In the following cases, a new test version is always created:
- Main metric changed
- At least one variation was removed, added, paused, unpaused
- Allocation of variations changed
- Stickiness was changed
- Campaign is archived