To make sure A/B tests results are clean and reliable, each test is managed in versions. This enables you to make changes to a running test without mixing the pre and post-change data in a way that affects the result. When a new test version is created, both the allocation and the Probability to Be Best result are reset. The report enables you to switch between the current version (default view) or any previous test version that ran in this experience.
Creating a new test version
The first test version is created when you launch an A/B test experience for the first time. If you edit the experience, you are notified that a new test version will be created, and you can select whether or not select to proceed with the new version.
We recommend creating a new test whenever the changes are expected to affect the test results (for example, changing the image or copy). For smaller changes that probably won’t affect the results, you can continue using the same test version. For example:
- Changing the tracking URL parameter of a target URL.
- Changing the trigger from time on page of 15 seconds to time on page of 14 seconds.
- Changing the backdrop color of an Overlay.
- Changing the status of an experience (say, from active to draft).
In the following cases, a new test version is always created:
- The main metric changed.
- At least one variation was removed, added, paused, unpaused.
- Allocation of variations changed.
- Stickiness was changed.
- The campaign is archived.