Dynamic Yield has two modes for determining how to allocate your variations across your traffic: A/B Tests and Dynamic Allocation. This article describes both modes, when to use each one, and how to use a control group to measure success.
Traffic allocation modes
A/B test: You define what percentage of traffic to allocate to each variation. When enough data is collected, a winning variation is declared, at which point you can confidently serve the winning variation to all users.
Dynamic allocation: Dynamic Yield decides what percentage of your users will receive each variation. It leverages a Multi-Arm Bandit strategy, which means that variations that performed better in the past are allocated more traffic. The best variation is gradually served to more and more of your visitors as data is collected. For more details, see Dynamic Allocation.
Choosing a traffic allocation mode
Use A/B test if you want to find one variation to use for the long run, or to test a hypothesis with the goal of learning information for future tests. The result of an A/B test is a statistically significant winner, so you can make confident decisions.
It is recommended to use this mode when testing a new page layout, the color of a call-to-action button, or a recommendation strategy for your product pages.
Use Dynamic Allocation if you want to optimize your conversions as quickly as possible, or when the variation lifespan is short. If you don't have time to collect data and wait for a winner, Dynamic Allocation enables you to direct more traffic to better-performing variations within hours.
This mode is useful for items like a hero banner that is updated weekly, or limited-time campaigns (such as a Black Friday sale).
Using a control group
A control group directs some of your traffic to a randomly chosen variation, or serves no variation, depending on your settings. You'll need a control group if you want to measure the uplift generated by any Dynamic Yield experience.
By default, all experiences include a control group set at 0% traffic allocation. To enable a control group, increase the percentage of traffic allocation. Usually, control groups of between 5-15% are sufficient.
For A/B tests, traffic in the control group is not served any variation.
For Dynamic Allocation, you can specify how the control group traffic is directed:
- No experience: Visitors in the control group are not served the Dynamic Yield experience. This option can't be combined with a primary metric of CTR or Attribution Window that starts when a variation is served and clicked.
- Random Variation: Visitors in the control group will receive a random variation, not driven by an optimization or personalization algorithm.