Predictive Targeting

Follow

When you run an A/B test, Dynamic Yield's predictive analytics engine runs in the background searching for hidden personalization opportunities.

The predictive analytics engine solves one of the biggest problems of A/B testing: tests optimize for the average user. That is, if most users significantly prefer variation A, when the test concludes it will be served to all users. However, you can improve your results if you serve a different variation to a set of users who prefer a less popular option. 

Predictive Targeting ensures that even tests that conclude without a winner for the overall population, may have a chance to generate uplift by serving a “losing” variation to an audience that actually prefers it.

In the image below, we see a test that resulted in an overall uplift of -51.7%. Predictive Targeting suggests a new course of action: if variation A is served to the Referral Traffic audience, while all other users receive the Control Group, the expected uplift is +11.4%.

Predictive_Targeting.png

How it works

  1. Predictive Analytics runs in the background of every A/B test you run – scanning all audiences, and surfacing those who prefer a “losing” variation. Only courses of action that have enough significance are evaluated, and prioritized in terms of significance and expected uplift.
  2. Once an opportunity is found, you’ll be notified in the report and via email. The table displays the targeting groups, the variations which should be served to each, and the expected uplift the course of action would generate.
  3. You may accept the personalization opportunity by clicking the APPLY button if you want to take advantage of this opportunity. Clicking APPLY does one of the following:
    • If the Control Group is one of the variations:
      A new targeting condition is added, limiting the serving of the experience, so those who preferred the control group will not be served with this experience.
    • If the Control Group is not one of the variations:
      The experience is duplicated, and a new targeting condition is added to the new experience. All variations but one are paused in one experience, and all variation but (a different) one are paused in the other experience, so one targeting group is served with one variation, and the rest with the other variation.
  4. Click the 3 dots to see the following options:
    • Review before publishing: redirects you to the form with the setup that clicking on APPLY would create. This enables you to double-check the setup and make any changes before publishing.
    • Breakdown report: Enables you to see how the relevant audiences performed. By doing so, you will see that there’s a different leader when filtering the report.
  5. Clicking More Options under the table displays other courses of action (if found), and the common A/B testing course of action (serving the leading variation to all users).
     

Statistical significance of personalization opportunities

Personalization opportunities require high significance to ensure valid suggestions. The suggested course of action must be better than serving the leading variation to all users by at least 1%, with a confidence of at least 95%.