Outliers are revenue-based events that have unusually high or low values. A/B tests attempt to predict future user behavior and patterns so their results should not be overly affected by a few big purchases. Therefore, we filter these values out of important calculations to ensure that A/B tests reflect the true winners and are not skewed by one huge purchases (e.g. one user purchasing 25 items).
By default, A/B test report will not include outliers, but you can include them by clicking the dropdown above the variation performance table and changing it to Include Outliers.
Revenue Events Log
To export a log of all revenue events that were fired in a A/B test, click the Export button at the top-right of the each report, and select “Revenue event log”. This will include all revenue events as well as indicate if these events were outliers or not. The revenue events log is available for all A/B tests (campaigns using A/B test allocation that have at least 2 variations or 1 variation with a control group).
In A/B test reports, Dynamic Yield calculates outliers according to each revenue-based event type, in each variation, for every test version. After the median is calculated, we mark any event whose value is +/- 3 standard deviations from the median as an outlier. When calculating the Probability to be Best, the winner of an A/B test, and for predictive targeting, the value of the outlier is replaced with the average value of the event.