I was working on a large educational game targeted at young children. At the time, the developer had discovered issues with with tutorial or FTUE (First Time User Experience) completion rates and I asked if I could look more deeply into the issue. From everyone's experience at the time, this issue had been around for some time, but they didn't have good tracking of the onboarding funnel, so they couldn't see what the underlying issues were.
The way the game worked, you could explore everything quasi-freely without engaging in any educational content, but if you wanted to progress you had to engage in combat encounters, and to do anything in an encounter you needed to answer educational questions.
Correctly.
Often over and over again.
And the educational content was mathematics.
For those unaware, math is an incredibly heavy subject for students, and carries a lot of baggage. It's such a big issue that there's a whole area of pedagological research and concern called "math attitudes" which examines how peoples feelings about and relationship to the subject of mathematics change over time, and which groups have more or less positive feelings about the subject. Why is this such a finely studied area? Because math is both a requirement for most deeply technical jobs these days, and because math is a social signifier of intelligence. Not an actual signifier of intelligence, mind you, just a social one.
People who are good at math are seen as highly intelligent, while people who are not are often seen as less intelligent. The inverse is also true: People who aren't "supposed" to be smart are assumed to be good at math, and people who are "supposed" to be dumb are assumed to be bad at it. This has a lot of effects, but some well known ones are that A) math ability is widely viewed as an innate quality of a person, B) girls, black, and indigenous children are often overlooked or given less support during math class, and C) people who underperform in math end up with low self-esteem around the subject (and are noted to have bad "math attitudes").
Educational games and gamified educational applications often position themselves as a way to get students to engage with subjects that have a lot of friction or carry a lot of baggage. This is exactly what this developer had done with their game, and they had interviews, referrals, and even published partnered research showing that it worked. Giving students a goal that they care about and gatekeeping it behind educational content can encourage students who are struggling to push through, discover that they can rise to the challenge, and develop better attitudes towards math.
The way the game was set up, however, you could not progress if you answered questions incorrectly. Failing to get the correct answers could, hypothetically, stonewall you.
This is what I decided to look into: Were the students who struggled the most with math, and who may not have had an explicitly supportive environment, being locked out of participation in the game? These were the students who would use the support and assistance of the product the most, and I suspected that they were being actively dissuaded from getting that support by being locked out and humiliated right out of the gate. I also suspected that players who were the best at math would have the best retention and conversion figures, since the game actively rewarded them for the skills they already had.
Mathematically, this was a fairly simple investigation. I looked at the question and response data from the first N or so questions players answered (where it took N-n questions to get through the FTUE, if you had a perfect score), and then looked to see what the retention and conversion rates were after 1, 2, and 4 weeks. I binned students by how many questions they completed and then plotted success rate vs completion rate. For students that completed n+ questions, I also looked at success rate vs retention/conversion rate at the different weekly thresholds. The responses were interesting.
For players that completed fewer than n questions, their success rate was very low. Many of these students got nowhere near completing the FTUE, with only a handful of questions answered, and of the questions that were answered, success rates of less than 25% were common. Immediately we could see that students were not completing the FTUE. Many of them weren't even getting past the first hurdle! They were being shut down almost immediately.
For players that completed n-or-more questions, I found that students with success rates of less than 50% were highly unlikely to be retained even into the next session. On the other side of the distribution, players with success rates of over 90% also showed a downward trend with respect to retention and conversion. The peak, it turned out, was around 70% success rate. Significantly higher than that, and students found it too easy and churned after a couple of weeks, while significantly lower than that and they found it too difficult and discouraging and churned (basically immediately).
I spoke to producers and the educational team about this, and found out that they had set up a placement test (to determine the student's math skills and help tailor the kinds of questions they were being fed) right at the very start of their play experience, i.e. right at the start of the FTUE. Placement tests work by providing students with questions of varying difficulty and seeing where the threshold is between success and failure, and that meant players during the FTUE were being hit with questions that may have been significantly above their actual skill level, right at the same time as they were trying to learn and enjoy a new video game.
Ultimately, the education team agreed to delay the placement test until after the FTUE and spread the questions out over a longer period of time so that players could learn, acclimate to, and get invested in the game experience before being overwhelmed by questions that would harm their attitudes towards math.