Dynamic Yield has three modes for determining how to allocate your variations across your traffic: A/B tests, dynamic allocation, and affinity allocation. This article describes the available modes, when to use them, and how to use a control group to measure success.
Traffic allocation modes
A/B test: You define what percentage of traffic to allocate to each variation. When enough data is collected, a winning variation is declared, and you can confidently serve the winning variation to all users.
Dynamic allocation: Dynamic Yield decides what percentage of your users will receive each variation. It leverages a multi-armed bandit strategy, which means that variations that performed better in the past are allocated more traffic. The best variation is gradually served to more and more of your visitors as data is collected. For more details, see Dynamic Allocation.
User affinity: Dynamic Yield automatically determines which variation to serve to each user based on their current, real-time preferences and affinity profile, and the affinity attributes each variation is tagged with.
Choosing a traffic allocation mode
Use A/B test if you want to find one variation to use for the long run, or to test a hypothesis with the goal of learning information for future tests. The result of an A/B test is a statistically significant winner, so you can make confident decisions.
It is recommended to use this mode when testing a new page layout, the color of a call-to-action button, or a recommendation strategy for your product pages.
Use dynamic allocation if you want to optimize your conversions as quickly as possible, or when the variation lifespan is short. If you don't have time to collect data and wait for a winner, Dynamic Allocation enables you to direct more traffic to better-performing variations within hours.
This mode is useful for items like a hero banner that is updated weekly, or limited-time campaigns (such as a Black Friday sale).
Use user affinity for areas that are fairly evergreen in terms of real estate on the website or app and content type. Because user affinity allocation serves the variation that is optimal to each user, there is no inherent test between variations, but rather between the allocation method and the control group.
Using a control group
A control group directs some of your traffic to a randomly chosen variation, or serves no variation, depending on your settings. You'll need a control group if you want to measure the uplift generated by any Dynamic Yield experience.
By default, all experiences include a control group set at 0% traffic allocation. To enable a control group, increase the percentage of traffic allocation. Usually, control groups of between 5-15% are sufficient.
In A/B tests, no variation is served to traffic in the control group.
In dynamic allocation and user affinity allocation mode, you can specify how the control group traffic is directed:
- No experience: Visitors in the control group are not served the Dynamic Yield experience. This option can't be combined with a primary metric of CTR or an attribution window that starts when a variation is served and clicked.
- Random variation: Visitors in the control group receive a random variation, not driven by an optimization or personalization algorithm.