Personalization Impact Report

Measure the overall impact of your personalization program by running a global test (previously known as Global Control Group test) comparing users who receive personalized Dynamic Yield experiences and users who don’t.

For example:

By comparing the performance of the Dynamic Yield Experiences and Control Group variations, you can estimate the total impact of experiences A, B, and C.

Start measuring the impact of your personalization

Step 1: Add experiences to the test

Adding experiences to the global test is an ongoing process. If you have personalized experiences, target them (using the built-in Global Test targeting condition) to the Dynamic Yield Experiences group. This ensures that only users in the 95% population group receive those experiences.

If you're using active cookie consent, consider creating a second fallback experience with no targeting conditions that can be served to opted-out users.

Best practices: What should you measure?

• Consult your Customer Success Manager regarding personalization and testing efforts that best measure the value of your personalization program.
• Don't include experiences that have no effect on the user's experience.
  For example, firing an event or implementing a cookie.
• Don't include experiences that must be served and are not optimized.
  For example, cookie consent notification.

Step 2: Activate the Personalization Impact report

Go to the Experience OS dashboard, and in the Personalization Impact widget, click Set Up.

The first time you enter the report, an onboarding page appears. The instructions direct you to update your campaign and target them as described in Step 1. When your campaigns are targeted correctly, click Yes, start running report. The report will be available the next morning (by 9 AM).

Important: Don’t activate the report until after you target the relevant campaigns. Activating the report prematurely will affect your measurement because the result will reflect an A/A test and not the true impact of your personalization efforts.

Step 3: Start measuring the impact of your personalization efforts

The Personalization Impact widget on your dashboard is your entry point to the Personalization Impact report. Get a quick view of both your cumulative gains and the monthly impact of your selected primary metric. You can view the overall results since activating the report, or view the current or previous year. 

Measuring impact is an ongoing process: Add campaigns as needed

When adding new campaigns and experiences in Experience OS, follow the guidelines in Step 1 and consider adding them to the global test. You can use the campaign list in the Personalization Impact report to get an overview of which campaigns are part of the test and which are not.

The Personalization Impact report

Working with timeframes

The report initially covers the entire test duration ("Overall"). To focus on a specific timeframe, adjust the selector at the top of the report. You can select any period within the test duration by selecting Custom Range, or use the predefined periods of Year to Date or Last Year.

In contrast to the Experience Report, custom timeframes include all users who visit your site and their activity, rather than only new users. This provides results based on all site traffic.

Note: The Probability to Be Best value is available only when you select the first day of the test.

The Over Time bar chart

Analyze the gain and uplift of each metric in smaller time increments and see the changes over time. Select the time unit that best fits your requirements and group the data by week, month, or quarter.

The bars in the graph represent the metric gains added during each period. The summary of the entire period appears above the chart. Note that the overall metric gain is the sum of all the periods, but the overall uplift can't be summed up the same way. 

Understand which campaigns are measured in the global test

In the report, all your campaigns are divided into three groups:

• Dynamic Yield Experiences: Campaigns that are part of the global test and served only to users who belong to the Dynamic Yield Experiences group.
• Global Control: Campaigns that are part of the global test and served only to users who belong to the Global Control group.
• Unmeasured: Campaigns that aren’t part of the global test and are served to all users (according to their targeting condition).

Use this overview to ensure all the campaigns in your personalization program are measured and see which campaigns are influencing the global test performance.

Executive spotlight

Easily share a summary of the global test results for the chosen primary metric. Changing the selected timeframe and the primary metric generates a new summary.

Export results

Click to export the report data to a CSV file to further analyze it in a different platform or share it externally.

The export includes the daily cumulative data of all the metrics in the report. Each row represents a date and the snapshot of the results on that date, based on the data collected from the first day of the test.

The export file structure:

Upgrade from the Global Control Group setup to Personalization Impact

If you're currently using an evaluator to split all users into the Dynamic Yield Experiences and Global Control groups to measure your personalization program, the basic and recommended procedure to upgrade to Personalization Impact is to go over all the campaigns that are currently part of your Global Control Group test and replace the existing evaluator-based targeting condition with the built-in Global Test targeting condition.

However, we highly recommend that you update your evaluator code with the following JavaScript to make sure that the logic of the new targeting condition is replicated in ALL your campaigns. This is important because it ensures that the targeting remains consistent even if you happen to overlook migrating some of your measured experiences. 

Updating your evaluator code is also a fast and effective solution if you want to start using the Personalization Impact report immediately, while you work on updating your campaigns with the built-in Global Test targeting. But don't forget to update the targeting condition in all measured campaigns to the Global Test targeting condition, as it will enable you to:

• Start measuring users' performance beginning with their first pageview.
• Measure the impact of API campaigns.

// DYO.Q
  function gcgCheck() 
  {
    const HIGH_BIT = 0xffffffff;
    const MACHINE_MASK = 0xff;
    const DY_EXP_GROUP = 'Dynamic Yield Experiences';
    const GLOBAL_CONTROL_GROUP = 'Control Group';
    const GCG_GROUP_VALUE = 4;
    const CGC_TOTAL_GROUPS = 100;

    const id = DYO.UserUtils.getDyid();
    return new Promise((resolve, reject) => {
        let highBytes;
        if (window.BigInt) { 
            const i64 = BigInt(id);
            const n32 = BigInt(32);
            highBytes = i64 >> n32;
            lowBytes = i64 - (highBytes << n32);
            highBytes = Number(highBytes);
            resolve(highBytes);
        } else {
            const l = DYO.Long.fromString(id);
            highBytes = l.high;
            resolve(highBytes);
        }
    }).then(highBytes => {
        const cycle = highBytes >>> 20;
        const group = cycle % CGC_TOTAL_GROUPS;
        const groupName = group > GCG_GROUP_VALUE? DY_EXP_GROUP : GLOBAL_CONTROL_GROUP;
        return groupName;
    });
}

gcgCheck()

Verify the evaluator update

To make sure your updated evaluator is working properly, do one or both of the following: 

• Test your evaluator in the console 
• Check the engagement with your experiences to make sure they're being served (report impressions)