When comparing two variations against each other, I would find it useful to know what percentage of users interacted with the widget/recommendations/element served.
Often, the element placed on screen would require more than one click, yet the CTR shown in reports counts each click meaning that CTR is not always a good measure of interaction.
Example test and hypothesis
My sale clothing category is heavily fragmented in terms of sizing. Driving visitors to this category has a low success rate (low CR, high bounce, high exit) as customers quickly click on items that are unavailable in their size and navigate away. I create an interactive widget to AB test against my simple banner which asks users to select their size first, and hypothesise that these users will find items more easily in their size and add to basket.
Variation 1 (click to browse)
- 10 users see a simple banner inviting them to browse the sale category
- 7 click it
- CTR is 70% (7 clicks total, 7 users interacted)
- Revenue is low because most users got frustrated and left
Variation 2 (choose size, and then browse)
- 10 users see an interactive banner which asks them to choose their size, and then invites them to browse the sale category
- 4 users click to select their size, 2 of these users click to browse the category
- CTR is 60% (6 clicks total, 4 users interacted)
- Revenue is high because these 2 users both found something straight away and checked out
With DY default reports, I would see various metrics such as revenue, AOV, CR, but I would have no idea how many users or what percentage of users interacted with what was served. Indeed, a badly designed widget which users had to click on a lot would register a high CTR (most often a good sign) whilst not leading to an uplift in other KPIs.
In the above example, revenue would be up for Variation 2 despite a lower CTR but I would have no idea how many users used it, how often on average each user was clicking it, etc.
Interested to hear whether others have wanted to measure rate of interaction and how they've managed to do so.
Please sign in to leave a comment.