Dynamic Yield has two modes for determining how to allocate your variations across your traffic: A/B Tests and Dynamic Allocation. This article describes both modes, when to use each one, and how to use a control group to measure success.
Traffic Allocation Modes
A/B Test: You define what percentage of traffic to allocate to each variation. When enough data is collected, a winning variation is declared, at which point you can confidently serve the winning variation to all users.
Dynamic Allocation: Dynamic Yield decides what percentage of your users will receive each variation. It leverages a Multi-Arm Bandit strategy, which means that variations that performed better in the past are allocated more traffic. The best variation is gradually served to more and more of your visitors as data is collected. For more details, see Dynamic Allocation.
Choosing a Traffic Allocation Mode
A/B Test is better if you want to find one variation to use for the long run, or to test a hypothesis with the goal of learning information for future tests. The result of A/B tests is a statistically significant winner, so you can make confident decisions.
It is recommended to use this mode when testing a new page layout, the color of a call-to-action button, or a recommendation strategy for your product pages.
Dynamic Allocation is better if you want to optimize your conversions as fast as possible, or when variation lifespan is short. If you don't have time to collect data and wait for a winner, Dynamic Allocation allows you to direct more traffic to better performing variations within hours.
This mode is useful for items such as a hero banner that is updated weekly, or limited-time campaigns (e.g. "Black Friday" campaign).
Using the Control Group
A control group directs a portion of your traffic to a randomly chosen variation, or serves no variation, depending on your settings. Establishing a Control Group is essential for measuring uplift generated by any Dynamic Yield experience.
By default, all experiences include a control group set at 0% traffic allocation. To enable a control group, increase the percentage of traffic allocation. Normally, control groups of between 5-15% are sufficient.
For A/B tests, traffic in the control group is not served any variation.
For Dynamic Allocation, you can specify how the control group traffic is directed:
- No Experience: Visitors in the control group will not be served with the experience by Dynamic Yield. This option cannot be combined with a primary metric of CTR or Attribution Window that starts when a variation is served and clicked.
- Random Variation: Visitors in the control group will receive a random variation, not driven by an optimization or personalization algorithm.
