Christopher Cooke
Senior Product Data Scientist | Knowledge Generation through Experimentation, Eventing, User Behaviour Analysis, and Predictive Analytics

Main Menu

  • Home
  • About Me
  • CV/Resume
  • Portfolio
  • LinkedIn Profile
  • Contact Me

Login Form

  • Forgot your password?
  • Forgot your username?
  1. You are here:  
  2. Home
  3. Portfolio

Gender Bias in On-Site Interview Outcomes

I was asked to look into whether there was evidence of gender bias in interview outcomes. In particular, I was asked to provide:

1. A summary report for the executive team. 

2. A detailed report for the data team, covering analysis details, possible follow-up tasks, open questions, etc.

 

Article Summary

Domain: Statistical Auditing, People Analytics & DE&I Data Integrity

Core Objective: Evaluate gender bias in on-site interview decisions across 4,538 candidates, distinguish stage-specific pipeline bias from final selection outcomes, and uncover data pipeline anomalies.

Methodology: Confounding variable isolation (Simpson’s Paradox identification), p-value hypothesis testing for decision rates and time-to-decision, and data quality forensics (data provenance tracking via schema case-sensitivity).

Key Analytical Breakthroughs:

        1. Simpson’s Paradox Isolation: Aggregate company data skewed male (p < 1.57 x 10-21), but department-level disaggregation revealed zero male-favorable bias at the on-site stage. Engineering exhibited a statistically significant selection bias toward female candidates (82% vs. 61%, p < 8.44 x 10-5).

        2. Upstream Pipeline Imbalance: The overall headcount skew stems entirely from top-of-funnel interview pool composition (Engineering and Product invited men at 7.5x and 20.7x the rate of women).

        3. Data Forensic Discovery: Identified an ingestion bug where lowercase gender tags ("male", "female") isolated a specific inbound source with a $2.7\times$ lower conversion rate (p < 0.004), alongside a 1.6-day time-to-decision penalty for women in Engineering (p < 2.37 x 10-6).

 

Read more: Gender Bias in On-Site Interview Outcomes

Article Review: Are referred candidates more likely to get hired?

Article Summary

Domain: Methodological Review, Data Journalism Critiques & Talent Acquisition Analytics

Core Objective: Deconstruct a flawed recruitment trends report to highlight statistical blind spots, compositional biases, and unaddressed confounding variables.

Methodology: Systematic interrogation of data definitions, identifying composition/denominator shifts, diagnosing potential Simpson's Paradox, and verifying whether conclusions directly answer the headline premise.

Key Analytical Vulnerabilities Identified:

        1. Proportional Distortion vs. Absolute Volume: Demonstrating that a drop in the proportion of referrals (from 2% to <1%) is primarily a mathematical artifact of surging inbound volume rather than an absolute decline in referral counts.

        2. Denominator & Stage Confusion: Uncovering ambiguity between "offer rate per applicant" vs. "offer rate per interviewee," which conceals whether conversion shifts stem from changing candidate quality or talent acquisition capacity constraints.

        3. Confounding & Aggregation Bias: Pointing out that pooling 38M applications across heterogeneous company sizes, industries, and job levels without controlling for role distribution creates severe risk of Simpson’s Paradox.

 

I had been asked to provide feedback on this article. This was my response:

 


 

Who is the audience for this article? Hiring managers? Job seekers? Talent acquisition specialists?



With a massive increase in inbound application volumes

Are “inbound” applications the only ones that have increased? The linked article only discusses “business and tech” roles. Is this trend seen in other roles? How are “business” and “tech” defined?

Read more: Article Review: Are referred candidates more likely to get hired?

Player Session Data SQL Examples

As part of a technical take-home assessment, I was presented with the following three tables and asked to manipulate them using SQL.

Article Summary

Domain: Gaming Data Engineering & Technical SQL Assessment

Core Objective: Demonstrate production-grade SQL data manipulation, analytical CTE structuring, timestamp/epoch arithmetic, window function navigation, and dynamic Jinja/dbt metaprogramming across player telemetry tables.

Methodology: Explicit type casting (::INT, TO_TIMESTAMP), defensive joining (LEFT JOIN to identify orphan records and unobserved churn buckets), windowed LEAD() event intervals, and generate_series() array expansions for dynamic unpivoting.

Key Solutions:

        1. Data Cleaning & Aggregation: Materialized cleaned timestamp views and computed active session lengths using epoch difference extraction (EXTRACT(EPOCH FROM ...)).

        2. Windowed Inter-Event Timing: Deployed LEAD(ts, 1) OVER (PARTITION BY player_id, session_id) to isolate intra-session action latencies while explicitly flagging stakeholder ambiguity regarding grouping granularities.

        3. Dynamic Pivot & Metaprogramming: Handled sparse dimensional data (missing Level 3 churn) via COALESCE with generate_series(), and provided a scalable dbt/Jinja macro solution to dynamically unpivot rows to columns based on target table invariants.

Read more: Player Session Data SQL Examples

Assessing an A/B Test With No Meaningful Information

As part of a take-home technical assessment, I was given two monetization metrics from an A/B test:

 

GROUP A (Control)

GROUP B

ARPDAU $

0.15

0.16

ARPPU $

30

16

 

Article Summary

Domain: Game Economy Optimization & A/B Test Interpretation

Core Problem: Variant B produced a marginal increase in ARPDAU ($0.15 -> $0.16) alongside a large (~47%) drop in ARPPU ($30 ->$16), presenting a classic monetization trade-off.

Methodology: Isolation of statistical/sample size gaps, behavioral decomposition of monetization metrics across multi-day experiment windows, and distribution diagnostics (bimodal spend distributions vs. whale cannibalization).

Key Insights:

        1. Metric Mechanics: ARPDAU divides total revenue by aggregate user-days (i.e. sum of DAU), making it sensitive to both daily spender proportion and player login frequency. A steady ARPDAU paired with a halved cumulative ARPPU points to a substantial expansion in unique payers relative to average active days.

        2. Strategic Valuation: Converting non-payers into spenders is traditionally the hardest hurdle in F2P economies. However, evaluating Variant B requires confirming that existing "whales" weren't simply cannibalized by lower-priced SKUs.

        3. Root Mechanics: Hypothesized features triggering this behaviour include entry-level starter packs, battle passes, or piggy bank mechanics that unlock high-volume, low-price purchases.

Here, ARPDAU (Average Revenue per Daily Active User) is calculated as the sum of the revenue generated over the experiment divided by the sum of the daily users. ARPPU (Average Revenue per Paying User) is the sum of the revenue over the experiment divided by the total number of unique payers in that window. 

Read more: Assessing an A/B Test With No Meaningful Information

Revenue, Retention, and Recommendations for a Live Operations Event

This was a test given to me as part of a technical assessment from a video game studio. It asks the data scientist to create a report on a Live Event given mission completion and skip data aggregated by player level. In order to progress through the event, players either have to complete a series of missions -- broken up into 5 separate chapters -- or they can pay premium currency to skip missions.

Premium currency can be purchased at a rate of 100 for $4.99.

The original report was written as a PowerPoint slide deck. Stakeholders requested:

  1. A Churn Analysis that included:
    • Identifying the missions with the largest player drop-off
    • Splits by player-level groupings
  2. A Revenue Analysis that included:
    • Total Revenue earned from the event
    • Revenue by level
  3. Recommendations to improve future events

 

Article Summary

Domain: LiveOps Game Economy & Monetization Analytics

Core Problem: Severe early-game churn (78% drop in Chapter 1), broken difficulty pacing (loss of the "sawtooth" curve in Chapters 4–5), and erratic skip-cost scaling that alienated players and concentrated late-game revenue entirely on whale-tier final missions.

Methodology: Full-funnel cohort telemetry analysis, cross-tabulation of premium currency spend (skips) by mission and player level, and behavioral proxy modeling (using skip rates as a metric for real-time difficulty friction).

Key Findings:

        1. Monetization Concentration: Chapter 1 accounted for 39.5% of total event revenue ($224,111 total), while Chapter 5 was an extreme "revenue desert" where 91.3% of chapter spend came from the final mission alone.

        2. Pacing Collapse: The classic design pattern—a sawtooth difficulty curve building toward a rewarding climax—collapsed entirely in the back half of the event, rendering mid-chapter progression flat and unengaging.

        3. Pricing Distortion: Inconsistent skip cost multipliers ($45 event finale skip price) signaled a "pay-to-win/whales-only" economy that risked long-term retention and new-player acquisition.

Outcome:

        • Delivered actionable design recommendations to restructure early-game onboarding friction, reintroduce multi-wave pacing in long chapters, smooth out cost multipliers, and reposition penultimate vs. finale mission balancing.

Read more: Revenue, Retention, and Recommendations for a Live Operations Event

  1. Directing Focus: How a Basic UI Test Shaped an Attention Framework
  2. Fun and Folly with Microtransactions
  3. Educational Gaming and the Impact of Existing Knowledge
  4. Modelling the NS COVID-19 Outbreak - 12 May, 2020

Page 1 of 2

  • 1
  • 2