The Experience Report enables you to assess the results of experiments and draw conclusions, while giving you the flexibility to uncover additional insights by applying segmentation and exploring secondary metrics.
You can view this report by clicking the icon on any page that lists campaigns or experiences, or from the Dashboard tiles.
Experience report types
The Experience Report layout changes depending on the allocation method (A/B test, Dynamic, or Affinity Allocation), the number of variations, the existence (or lack) of a control group, and the campaign type (some campaign types have dedicated layouts, such as Multi-Touch, Experience Email, and Reconnect).
A/B tests
A report for experiences that use A/B test allocation and have at least two active variations. This report enables you to assess the results of a test between multiple variations and draw conclusions.
Dynamic and Affinity Allocation
Provides results for experiences that use Dynamic or Affinity allocation and have at least two active variations. The report includes the results at the entire experience level to measure the algorithm's performance against a control group, as well as the individual performance of each of the variations.
Learn more about Dynamic Allocation and Affinity Allocation reports.
Single variation experiences (without a control group)
Provides results for experiences that have 1 variation that receives 100% of the traffic, regardless of the allocation method. This type is experience is generally used to manage content on your site (CMS) by harvesting Dynamic Yield capabilities. This experience report displays the usage attributed to the variation without testing it against an alternative.
Multi-touch, Experience Email, and Reconnect
Reports for these unique campaign types with a single experience provide results at the campaign level to measure its performance (and also test multiple variations in multi-touch and Reconnect campaigns), and also drill down to the individual performance of each touchpoint (multi-touch), channel (Reconnect), or block (Experience Email).
Learn more about Multi-Touch, Reconnect, and Experience Email reports.
Selecting a report version
By default, the report presents the results of the latest or current version of the experience. You can select a previous version from the Version dropdown:
A summary of the results for the selected version is displayed in the Summary pane:
The summary includes:
- The dates during which the selected version ran.
- When running a test between multiple variations (A/B test or Dynamic or Affinity Allocation with a control group):
- The test results for the overall traffic. A version is highlighted if it was declared a winner.
- The results of the test for primary audiences. A version is highlighted if it was declared a winner for the audience.
- Personalization opportunities detected by Predictive Targeting (available only for A/B tests).
Comparing variation performance
Visualizations in the report enables you to compare the performance of individual variations.
The Variation Performance table
The table displays the test results by variation and metric, with results based on the primary metric as the default. Click Metrics to add secondary metrics to your analysis. These are displayed in additional tables under the primary metric table.
The structure of each table is similar across primary and secondary metrics, with one row per variation and the fields described in the following table:
Field | Definition |
Normalization unit Examples: Users, Sessions |
Depending on stickiness, this is the unit by which the chosen metric is normalized:
This count is located under the name of each variation. |
Metric totals Examples: Purchases, Revenue |
If the selected metric is an event or goal, such as Purchases, this is the number of purchases attributed to the variation. If the selected metric is the value of an event, such as purchase revenue, this is the total value of the event that's attributed to the variation. Other common metrics not related to events are clicks and pageviews. |
Normalized metric Examples: Purchases/User, Revenue/User |
The metric total divided by the normalization unit of the test. This metric enables you to compare the performance of different variations fairly, normalizing results by the actual exposure they received. |
Uplift | The ratio of the normalized metric of each variation vs the control group's, minus 1. This metric appears only if there is a control group or a baseline variation has been selected. You can choose which variation to use as a baseline above the Variation Performance table. |
Probability to Be Best |
The chance of a variation to outperform all other variations in the test. It's a calculation that takes into account the difference in the performance of each variation and the statistical confidence we have in the results. This is the most actionable metric of your A/B test results, as it defines when results are conclusive, and you can apply the winner to all traffic. If a variation has been declared a winner, a trophy cup appears next to Probability to be Best, and uplifts are colored either green or red. A variation is declared a winner if:
|
Probability to Beat Control | The chance of a variation to outperform the baseline. Probability to beat control is equivalent to probability to be best, but each variation competes only against the baseline rather than against all other variations. It's useful in tests with more than two variations, where multiple variations might outperform the control, but perform similarly to each other. This means that no single variation can have a high probability to be best, but each can have a high probability to beat the control. |
Credible intervals |
A credible interval is a range that captures the likely true value of a metric with a certain probability. Credible intervals are the Bayesian counterpart to frequentist confidence intervals, but unlike the latter, they can be interpreted at face value: A 95% credible interval contains the likely true value of the metric with 95% probability. Credible intervals of 95% and 50% probability are available for both the metric (primary or secondary) and the uplift. |
Selecting a baseline variation
If a test doesn't have a default control group, but you want to use a specific variation as a baseline for the calculations of uplift and probability to beat control, you can select a variation from the Baseline dropdown.
Working with credible intervals
By default, the reports show text-based credible intervals, representing the 95% credible interval for both normalized metric and uplift.
Use the toggle in the Additional Options dropdown to display graphical credible intervals instead.
The graphical intervals are a great way to visually compare the variations, and in addition to the 95% intervals (thin horizontal line) also display the 50% intervals (thick horizontal line)
The interval for the selected baseline is colored grey and displays dashed lines for easier comparison with other variations (note that the interval for the uplift of the baseline does not exist).
Hovering over any interval displays a tooltip that contains the interval values.
Over Time graph
Click the Over Time Data button in the Variation Performance table to display daily results for the selected metric, for each variation.
You can plot the absolute or normalized metric totals for the selected metric as both daily or cumulative results.
While daily results are useful to inspect daily fluctuations, cumulative results are useful to see how the estimate for the normalized metric converges over time as more data is collected.
Working with time ranges
By default, reports display all data for the selected version (the Time Frame button displays Overall). Optionally, click the Time Frame button and filter for a specific time frame within the selected version:
Note that you can't select a time period that spans multiple versions; that is, blend data of multiple versions. This is designed to ensure that variations are comparable with each other.
Time range behavior with user stickiness
User stickiness is available only on A/B tests with multiple variations.
When filtering for a time range that doesn't include the start of the version, all users who were exposed to the test before the time frame are excluded from the results, including their eventual activity in the selected time frame.
This is designed to ensure that variations are comparable and prevent issues with survivorship bias, a form of selection bias, as the observed population of users in the selected time frame could otherwise influenced by their exposure to the test before the time frame.
For example, if the full version data includes two users and two purchases:
Selecting the following time frame would exclude user 1 and their purchase, because their first exposure to the test was before the selected time frame, and their purchase could thus be influenced by something outside the selected time frame.
A convenient application of this behavior is that for any time range selected, the results are based on completely fresh users who experienced your campaign for the first time in the given time frame. This might be useful, for example, to exclude the effect of a promotion that happened in week 1 of a test running multiple weeks.
While filtering for specific time frames might be useful depending on the context, we strongly recommend to generally:
- Draw conclusions using all the data collected.
- Use subsets of the entire data that contain full weeks.
Time range behavior with session stickiness
The behavior described in the previous section applies conceptually but not practically to experiment versions with session stickiness. This is because for the purpose of the Experience reports, sessions end at midnight, and so any individual date of any selected time frame contains full sessions and full attributed events.
Today (real-time data)
By selecting Today in the time range filter, you can access a simplified version of the report that shows all activity for the current day in real time. Note that this view is designed for monitoring the liveliness and correct serving of the experiment and does not include report functionality like audience breakdown or outlier handling. Note that due to the potentially large amount of live data being queried, this view might take longer to populate than past time frames.
Audience breakdown
Use the Audience Breakdown dropdown to segment results by audience.
After selecting an audience from the list, all tables are broken down by audience and indicate the percentage of total users or sessions that belong to the audience.
All table functionality available without audience breakdown also exists for audience breakdown, with the exception of the data over time, which is available only for the overall traffic.
Note: Audience breakdown considers users to be part of an audience only if they were in that audience at the moment of their first interaction with the experiment version. If a user enters the audience after interacting with the variation for the first time, the user is not considered part of that audience in the audience breakdown.
Outlier handling
Outliers are anomalous observations that, however rare, can carry enough weight to distort the results of a test. Dynamic Yield detects and handles two types of outliers:
- Extreme event values: Applied to every event or goal with value.
- Users with an extreme number of events: Applied to every event or goal (as of July 1, 2023).
If Dynamic Yield detects that the value of a revenue-based event or the number of events done by a specific user exceeds a threshold, the event or the user is labeled as an outlier and is replaced or excluded from the results, depending on the type of outlier. Learn more about our Outlier Handling.
By default, experience reports exclude outliers, but you can include them by turning off the toggle in the Additional Options dropdown.
Both types of outliers – extreme event values and users with an extreme number of events – are excluded via the same selector. All numbers in the reports are affected by this selector, with the exception of Predictive Targeting, which always computes results without outliers.
You can export a log of all events with values that were marked as outliers for a specific report, and how their value was handled by using the Revenue event log export described in the next section.
Exporting report data
You can export report data to a CSV file to further analyze it in a different platform or to share externally by clicking Export.
Two export options are available:
- Full report: A CSV version of the report, including all secondary metrics and audience breakdown (if applied).
- Revenue event log: A log that includes all events with event value (such as Purchase), with information about whether they were flagged as outliers due to an extreme event value or because the user who performed them was flagged as a user with an extreme number of events.
Update frequency
The experience report is computed nightly shortly after midnight, depending on the time zone selected in the section's General Settings. The results for the previous day should be available by 9 AM in the selected time zone.
Frequently Asked Questions
Which Pageviews are counted in the Pageviews metric?
The pageviews metric counts pageviews that are attributable to a given variation, that is, that happened after the variation was served to the user.
How is Uplift calculated?
The uplift compares the performance of each variation to the control group. It's relevant only if a control group or a baseline variation is defined. The calculation is as follows: (variation normalized metric / control group or baseline normalized metric) -1. For example, if Revenue/User for a variation is $5, but only $4 for the control group, the calculation is as follows: (5 / 4) -1 = 0.25, or an uplift of 25%.
Why don't I see information about clicks or CTR?
The CTR metric only appears in campaign types that render HTML (such as Recommendations, Dynamic Content, and Overlay). This is because CTR measures clicks on the HTML element that's rendered. Therefore, campaign types that don't render any HTML (such as Custom Code or Visual Edit) can't track clicks. If you want to measure clicks for one of these campaign types, you can fire a custom event upon clicking the element you want to track clicks for.
In multi-touch campaigns, clicks are tracked only to the touchpoints (which might have an HTML element) and not to the parent variation.
Why am I missing users when comparing Audience Breakdown vs. the overall population?
In A/B test reports, when you break down results by audiences, you might notice that some users were not segmented into any audience. This happens because users must be part of the selected audience at the moment of their first interaction with the variation. What can cause this:
- The variation redirects the user to another page immediately after it's served. This can cause missing data in audience breakdown, up to 100% of the data. Learn how to avoid this using this guide to Split/Redirect Tests.
- Users left the site or the page immediately after receiving the variation.
- In device type audiences: If the device is unknown (not a smartphone, tablet, or desktop).
Note: All Users data includes the data of all the users, even if their data doesn't appear in the audience breakdown.