Dynamic Allocation is one of the traffic allocation modes that determine how to allocate variations of experiences across your traffic. The Dynamic Allocation mode automatically determines what percentage of traffic each variation will serve. It leverages a Multi-Arm Bandit strategy, which means that variations that performed better in the past, are allocated more traffic. The best variation is gradually served to more and more of your visitors as data is collected. Over time, the leading variation may be replaced if a different variation begins performing better.
Unlike A/B tests, dynamic allocation aims to maximize the performance of an experience as fast as possible, with less focus on reaching statistical significance and more focus on optimization and speed.
This allocation mode is ideal for promotions that are changed frequently, as it quickly shifts traffic to better performing variations automatically.
Dynamic Allocation Algorithm
Variations are initially allocated equal traffic. As users interact with the variations, we use the primary metric to calculate the probability to best score, and over time increase exposure of the variations that are more successful. The more time has elapsed, the more heavily the score is weighted.
Every 40 minutes, Dynamic Yield recalculates the scores and assigns allocations (globally for all experiences, not 40 minutes after each create/edit). This ensures that optimization is done quickly, to make the most of the limited lifespan of creatives.
Measuring the Effectiveness of Dynamic Allocation Campaigns
For campaigns using A/B Test allocation, the Probability to be Best score typically reaches statistical significance and declares one “winning” variation. However, campaigns using Dynamic Allocation are designed to constantly test the results, and this typically prevents the Probability to be Best from clearly defining a winner. This is by design, and it means that the Probability to be Best score should not be used to determine the effectiveness of a Dynamic Allocation campaign.
Instead, we recommend allocating some traffic to a control group that selects a random variation. This will allow you to view a comparison of sessions with and without Dynamic Allocation in the Experience Reports.
As long as the control group is assigned a random variation, and is assigned 10% of your traffic or less, the traffic is taken from the “hidden” control group. Therefore we recommend allocating 10% of your traffic to such a control group, as it does not affect your experience negatively in any way and has the benefit of allowing you to measure your campaign.
For details about how to define a control group, see Using the Control Group.
Effects of Editing Experiences
As Dynamic Allocation is often used for campaigns that are frequently edited, we do not reset the algorithm every time a change is made. If a variation is edited, added, or deleted, we give it a relative Probability to be Best score according to the number of variations (e.g. 25% if 4, 33% if 3). The existing variations are allocated the remaining traffic according to their previous performance.
For example, let’s say you have three variations: A,B, and C. A is performing best, followed by B and then C.
- If you delete C (the worst variation), A will still get more exposure than variation B.
- If you edit variation C, A will still get more exposure than B, but C will be served to 33% of traffic in the first hours, allowing the system to evaluate the new version of variation C.
- If you add a new variation (D), it will initially be allocated 25% of the traffic, allowing the system to evaluate it, while still allocating A more than B, and B more than C.
In short, when using Dynamic Allocation, you can and should edit your variations to weed out losing variations and try new ones. The algorithm is designed to constantly adjust and optimize the allocation so you get the best possible results.
Note: The Probability to be Best is always calculated from the last edit of a variation until the current time, regardless of the time selected in an experience report. This may cause seeming discrepancies between the Probability to be Best and the primary metric when viewing reports in different time ranges, especially after editing an experience.
Explore vs. Exploit
Explore and Exploit are terms that define how much is "wasted" on learning, and how much on taking advantage of what is learned. In Dynamic Allocation, the rate of explore and exploit is 10/90. Meaning, 10% of traffic will be served a random variation, and 90% will be served with the better performing variation. This allows Dynamic Yield to direct most traffic to the leading variation, but still learn about "losing" variations, and allow losing variations to bounce back.
When using a control group that serves a random variation, Dynamic Yield reduces the "Explore" rate, as it can use the control group for this purpose.