I had just started working on an established freemium web game targeting children in the 6 - 10 year old range when, senior leadership became interested in getting more users to check their notifications page. This page was both a vector for marketing our paid subscription offers and where subscriber benefits were delivered to the players. I was personally interested in seeing if there was something we could do to meaningfully affect player engagement patterns, since many of our new features launched to relatively lacklustre engagement. While our most active cohorts always checked out the new updates, we always seemed to have significant issues with feature discover in our core users.
The product manager wanted to test out a badge on the notification button to indicate that there were new notifications for the user. This was a very basic MVP (minimum viable product) test -- just a red dot placed as an overlay on the button to see if it would draw players' mouse clicks. I wanted to look at the whole player engagement funnel for this: the percentage of active players that were exposed to the badge, the percentage that clicked on the badge after it was displayed, whether players scrolled through notifications after opening the page, and how many notifications were clicked on. Leadership wanted to look at higher level metrics, such as subscription conversion rate (CvR), subscription retention rate, and product retention rate.
We ran a single variate A/B test over a span of 2 - 3 weeks -- necessary to get a good bead on CvR and retention -- which gave us exceptionally clean data on the engagement figures. We saw a very large jump in the percentage of users that actually visited the notification centre, and a big increase in the number who actually opened a notification. Time on the page saw a significant bump, too, but that was a factor of users clicking into notifications (there were no differences in scrolling, and the lift in average notifications opened shifted from 0 towards 1).
Expectedly, there was no movement on the conversion and retention figures, as there was no mechanism to actually drive those values.
To me, this experiment was a major success on multiple levels. Not only was it the first time we had managed to demonstrate the ability to direct user focus and modify user behaviour to our preference, it also went on, eventually, to have a major impact on the product's design. While the follow-up to this experiment was very lagged due to senior leadership discounting all of the results save for conversions -- I pointed various PMs to the report for over 4 years after it was filed -- the findings were eventually the foundation of the product's UI refresh in late last, which included an "attention manager" feature that used visual highlighting to draw users' attention toward parts of the UI that led to features the design and product teams wanted to pull users into engaging with.