Assessing an A/B Test With No Meaningful Information

As part of a take-home technical assessment, I was given two monetization metrics from an A/B test:

Article Summary

Domain: Game Economy Optimization & A/B Test Interpretation

Core Problem: Variant B produced a marginal increase in ARPDAU ($0.15 -> $0.16) alongside a large (~47%) drop in ARPPU ($30 ->$16), presenting a classic monetization trade-off.

Methodology: Isolation of statistical/sample size gaps, behavioral decomposition of monetization metrics across multi-day experiment windows, and distribution diagnostics (bimodal spend distributions vs. whale cannibalization).

Key Insights:

1. 1. 1. 1. Metric Mechanics: ARPDAU divides total revenue by aggregate user-days (i.e. sum of DAU), making it sensitive to both daily spender proportion and player login frequency. A steady ARPDAU paired with a halved cumulative ARPPU points to a substantial expansion in unique payers relative to average active days.

         2. Strategic Valuation: Converting non-payers into spenders is traditionally the hardest hurdle in F2P economies. However, evaluating Variant B requires confirming that existing "whales" weren't simply cannibalized by lower-priced SKUs.

         3. Root Mechanics: Hypothesized features triggering this behaviour include entry-level starter packs, battle passes, or piggy bank mechanics that unlock high-volume, low-price purchases.

Here, ARPDAU (Average Revenue per Daily Active User) is calculated as the sum of the revenue generated over the experiment divided by the sum of the daily users. ARPPU (Average Revenue per Paying User) is the sum of the revenue over the experiment divided by the total number of unique payers in that window. 

1. What scenario would you pick to be the winner? Why?

Without more data, I would not crown a winner. The presented data does not have p-values, standard errors, or even sample size included, so it’s not clear that this isn’t a statistical tie. It’s also not at all clear how long the experiment ran for. More than that, many -- if not most -- features that would impact ARPPU this severely (assuming statistical significance) would also potentially impact player retention, playtime, progression rate, content consumption rate, and other measures of interest and import to stakeholders and game health.

Assuming that the differences between ARPPU and ARPDAU for these groups are significant, and that other metrics of interest have been measured and no significant difference between groups found, I would lean towards suggesting Group B. If no change was observed in DAU, then the lack of substantive change in ARPDAU with ARPPU nearly cut in half means the number of unique spenders in Group B was nearly double that seen in the control group.

Convincing players to spend money on a game once is a significantly harder task than to convince someone who has already spent to spend again. A feature or change that approximately doubles the number of players who have opened their wallets creates a more stable situation for the game and the business. A larger, more diverse payer base produces more and more diverse possibilities for future monetization.

That said, who these spenders are, and what has happened to the users who seemed happy spending $30 are questions of interest, and some exploration of them should be done before a final decision is made. If Group A's high-ARPPU spenders are whales who are more stable/predictable revenue, and Group B's larger, lower-value payer base could include price-sensitive players who are more likely to complain about pay-to-win or churn once the promotional/test conditions end, A might actually be the safer long-term bet.

It's also possible the conversion rate didn’t really change, and Group B just suppressed or removed high-spend outlier transactions (e.g. a large currency bundle got removed or de-emphasized in the test variant), pulling the average down without meaningfully changing payer count, in which case it would be prudent to stay with the control group.

2. Why do you think the ARPDAU and ARPPU values are behaving this way? What is influencing them?

There are a couple of mechanisms that could cause this behaviour:

a) A significant decrease in ARPPU coupled with a small change in ARPDAU is often a sign of conversion rate increasing on the back of globally lower purchase prices. If prices are reduced, more price-sensitive players can be enticed to purchase, but those who were happy spending more don’t have to.

b) It’s also something that could result from a bimodal distribution in ARPPU, if a new low-cost price point is introduced. With a large influx of new purchasers buying something at a very low price, as this could drag down the mean average spend per payer while only lifting ARPDAU by a tiny amount.

In order to say for sure which, if either, of these are driving this, I would have to look at the ARPPU and sales price distribution data.

3. What feature could we have tested to get these results?

Any number of features could generate these kinds of results:

a) A monthly battle pass could add a significant number of recurring purchases at a low-enough price point to entice a significant number of new buyers, while possibly cannibalizing some existing in-game store purchases.

b) An inexpensive starter pack could generate a significant number of new sales, while not appealing to (or not being available to) the players who are more likely to spend larger sums of money in the store.

c) Similar to a starter pack, the introduction of new, lower-priced packs or bundles in the in-game store could entice new, price-sensitive buyers to purchase, while being uninteresting to bigger spenders. 

d) A wholesale decrease in the cost of store packs could do this as well. Cutting prices in half could make purchases significantly more enticing to price-sensitive players, while also bringing down the costs for players who were going to make purchases anyway.

e) A piggy bank mechanic, where bonus rewards are held back from players behind a low-cost purchase and the players themselves determine the purchase’s value (by letting the bank collect as much as they want before pulling the trigger on a purchase) could create a large number of low-spending payers, just like a starter pack or entry-level SKU would.