Global Control Test is a method for measuring the overall impact of your experimentation program. The measurement is done using an A/B test that splits your overall traffic into two groups (users who receive personalized Dynamic Yield experiences and those who don’t) and compares them.
For example:
95% of users get Dynamic Yield Experiences These users are served multiple experiences you want to measure the impact of. |
5% of users are the Control Group These users receive, to the extent possible, a non-Dynamic Yield experience. |
Experience A: PDP Recommendations | – |
Experience B: Exit Intent | – |
Experience C: Personalized HP Banners | Experience C (baseline): Static HP Banners |
By comparing the performance of the Dynamic Yield Experiences and Control Group variations, we can estimate the total impact of experiences A, B, and C.
How does it work?
Step 1:
(One-time setup) |
Step 2: The evaluator reads the stored user assignments to each variation, and then surfaces them to use for targeting. (One-time setup) |
Step 3: These are the experiences whose impact you want to measure. Use the evaluator value to target the experiences to a specific variation of the Global Control Test. |
Setting up the Global Control Test
Step 1: Create the Global Control Test
Create a Custom Code campaign using the following settings:
- Name the campaign Global Control Test.
- Set the trigger as Page Load.
- Set the frequency to Session.
- Create an experience with the name Global Control Test.
- Leave targeting conditions as they are, targeted to all users.
- Create a variation called Dynamic Yield Experiences with the following JavaScript code:
cleanGcgStorage(); window.DYO.StorageUtils.set('_dy_cs_dygroup', 'Dynamic Yield Experiences', ['localStorage']); function cleanGcgStorage() { var GCG_STORAGE_NAME = '_dy_cs_gcg'; var DYG_STORAGE_NAME = '_dy_cs_dygroup'; var CG_VALUE = 'Control'; var DY_VALUE = 'Dynamic Yield Experiences'; var STORAGE_TYPE = 'localStorage'; if (DYO.StorageUtils.get(GCG_STORAGE_NAME, [STORAGE_TYPE]) === CG_VALUE) { DYO.StorageUtils.remove(GCG_STORAGE_NAME, [STORAGE_TYPE]); } if (DYO.StorageUtils.get(DYG_STORAGE_NAME, [STORAGE_TYPE]) === DY_VALUE) { DYO.StorageUtils.remove(DYG_STORAGE_NAME, [STORAGE_TYPE]); } }
- Create another variation called Control Group with the following JavaScript code:
cleanGcgStorage(); window.DYO.StorageUtils.set('_dy_cs_gcg', 'Control', ['localStorage']); function cleanGcgStorage() { var GCG_STORAGE_NAME = '_dy_cs_gcg'; var DYG_STORAGE_NAME = '_dy_cs_dygroup'; var CG_VALUE = 'Control'; var DY_VALUE = 'Dynamic Yield Experiences'; var STORAGE_TYPE = 'localStorage'; if (DYO.StorageUtils.get(GCG_STORAGE_NAME, [STORAGE_TYPE]) === CG_VALUE) { DYO.StorageUtils.remove(GCG_STORAGE_NAME, [STORAGE_TYPE]); } if (DYO.StorageUtils.get(DYG_STORAGE_NAME, [STORAGE_TYPE]) === DY_VALUE) { DYO.StorageUtils.remove(DYG_STORAGE_NAME, [STORAGE_TYPE]); } }
- Set the allocation to 95% for Dynamic Yield Experiences and 5% for Control Group.
- Ensure Variation Stickiness is set to Sticky for the User (multi-session).
- Publish the campaign.
Note: While the setup works regardless of the campaign and experience names you choose, we strongly recommend that you use the naming conventions outlined in this procedure.
Step 2: Create the evaluator
The evaluator is used to target experiences to the Dynamic Yield Experience group or the Control Group.
- Create an Evaluator with the name Global Control Test Group.
- Set the value type to String.
- Set Expected Values to Dynamic Yield Experiences and Control Group.
- Insert the following code:
(function() { window.DYCS = window.DYCS || {}; var deferred = DYO.Q.defer(); if (DYCS.inGlobalControl) return DYCS.inGlobalControl; getGCCookie(); return deferred.promise; function getGCCookie () { var GCValue = DYO.StorageUtils.get('_dy_cs_dygroup', ['localStorage']) || DYO.StorageUtils.get('_dy_cs_gcg', ['localStorage']); if (GCValue) { DYCS.inGlobalControl = GCValue; deferred.resolve(GCValue); } else { setTimeout(function() { getGCCookie(); }, 10); } } })();
- Save the evaluator.
Step 3: Add experiences to the Global Control Test
Adding experiences to the Global Test is an ongoing process. If you have a personalized experience, target it (using the evaluator you just created) to the Dynamic Yield Experiences group. This ensures that only users who are part of the 95% population receive this experience.
If the campaign required a default experience (served to all traffic, including the 5% control group), create the experience and serve it to the Control Group using the same evaluator.
Best practices: What should be measured?
- Recommendations, personalized experiences, and experiences running dynamic allocations should be served to the Dynamic Yield Experiences group.
- A/B tests, during the testing period, should not be served as part of the Global Test, as this is an exploration mode. However, after a winner is declared, you should pause all losing variations and serve the winning variation to the Dynamic Yield Experiences group.
- If you must serve content to the Global Control group (for example, if you have a personalized hero banner, but can’t afford to serve an empty slot) create an experience with a single variation as a fallback.
- Experiences that do not modify the user experience (fire an event or implement a cookie) should not be included in the Global Test.
- Experiences that must be served and are not optimized (say, cookie consent notification) should not be included in the Global Test.
Note: As with any A/B test, there is a small opportunity cost inherent to reserving a small group that is not served with the Dynamic Yield Experiences. However, the tradeoff of the cost is that you can at the same time measure the impact of your Dynamic Yield campaigns.