The ability to test variations is built into Experience OS, and Search is no exception. Experience Search offers seamless A/B testing, with no additional effort.
Set up an A/B test
In your experience, go to the Variation tab and select A/B Test:
- Configure the primary metric.
Select the main metric you want to use to evaluate test performance. - Click Settings (gear icon) to configure test settings.
- Variation stickiness: Sticky for user (multi-session) or sticky for session.
-
Start Attribution Window and End Attribution Window:
Recommendation: Use Variation is served and clicked attribution to reduce noise and ensure cleaner results.
- Create a new test variation.
Ensure that the assumption you're testing is the only difference between variations. Examples:- With Algorithm Studio boosting vs. no boosting
- Algorithm Studio algorithm A vs. Algorithm Studio algorithm B
- Different combinations of relevancy vs. Algorithm Studio factors
- With merchandising rules vs. no merchandising rules
- With JSON content vs. without JSON content
- With a redirect vs. no redirect
- Name the variation.
Use a descriptive name that clearly identifies the difference being tested. - Set traffic allocation.
Decide how much traffic to send to each variation. - Save and publish the test.
Test against other vendors
You can also test Experience Search against other vendors using a Dynamic Yield Experience OS API Custom Code campaign.
The idea is simple: Use Dynamic Yield as a decision layer to determine which search API should respond to the user query (Experience Search or another vendor).
How:
- Create an A/B test within an API Custom Code campaign.
- Set each variation to return a JSON response, where each response corresponds to a different vendor. For example:
- {"vendor":"a"}
- {"vendor":"b"}
Experience OS then generates a report showing how each variation impacted your selected metrics, enabling you to compare performance across vendors.