Outliers are revenue-based events that have unusually high or low values. A/B tests attempt to predict future user behavior and patterns so their results should not be overly affected by a few big purchases. Therefore, we filter these values out of important calculations to ensure that A/B tests reflect the true winners and are not skewed by one huge purchases (e.g. one user purchasing 25 items).
Viewing Outliers
By default, A/B test report will not include outliers, but you can include them by clicking the dropdown above the variation performance table and changing it to Include Outliers.
Revenue Events Log
To export a log of all revenue events that were fired by users of the A/B test, click the Export button at the top-right of the experience report page, and select “Revenue event log”. This will include all revenue events as well as indicate if these events were outliers or not. The revenue events log is available for all A/B tests (campaigns using A/B test allocation that have at least 2 variations or 1 variation with a control group).
Calculating Outliers
For every event that has a goal attached to it with a revenue value based on the event value (e.g. purchase, add to cart), an outlier threshold is calculated.
The median, average, and standard deviation are calculated daily based on data from the past 30 days of each event (even by users who haven't participated in any tests). The threshold for defining an outlier is 3 standard deviations from the median (above and below).
Note: outlier threshold is only calculated for events that had revenue in 100 event hits in the past 30 days.
In A/B test reports, in "Outlier Excluded" view (which is the default view), every event with a smaller or higher value than the outlier threshold is replaced with the average value of this event (based on the past 30 days or the entire population).