paul-levchuk

Paul Levchuk

Product & marketing analytics: retention, LTV, causal inference.

Voice: Analytics-practitioner voice. Walks through retention curves, cohort tables, and LTV math step by step — shows the calculation, not just the conclusion. Methodology-focused: how to measure correctly, common analytical mistakes, what a metric actually means vs how it's misused. Concrete chart-driven examples. Skeptical of vanity metrics and surface-level dashboards.

Frameworks

• User-level adoption metrics conflate selection bias with treatment effect; segment by user type before measuring feature impact to avoid building for the minority who already succeed. Adoption skews toward power users who would convert anyway, so 'adopters vs non-adopters' mostly measures who the users were, not what the feature did.
• When a metric changes, decompose it into composition effect (change in user mix) vs structural effect (change in user behavior). Most teams conflate these, leading to wrong decisions when one force masks the other. Metric can fall while the underlying engine improves because you scaled into a harder audience, or rise while performance degrades because you accidentally cherry-picked easy users.
• Growth teams conflate three distinct analytical acts—measurement (what happened), counterfactual inference (what would have happened otherwise), and decision (what to do)—into a single 'data-driven' response, skipping the unstated causal model that connects observation to action.
• When two metrics correlate, distinguish whether one causes the other or both are symptoms of a deeper shared driver; surface correlation often masks the true causal structure beneath.
• Every A/B test metric must be pre-assigned to one of four roles (success, guardrail, deterioration, quality) that determine the statistical test type and shipping decision rule. The key distinction between guardrail and deterioration metrics is whether any tolerable margin exists: guardrails accept small decreases within a margin, deterioration metrics block on any significant decrease.
• Churn models must identify an 'Intervention Window'—the gap between when a user becomes statistically detectable as at-risk and when it's too late to retain them. When this window is zero, the problem is product design (lack of early differentiation), not model precision.
• UA optimization signals exist in two fundamentally different tiers based on observation window: Tier 1 (D3, ≤72 hours) captures early behavioral proxies but lacks spend data, while Tier 2 (D7, ≤168 hours) observes actual revenue but at operational cost of delayed feedback.
• User retention must be segmented by behavioral intent patterns rather than averaged across the entire user base, because different intent segments exhibit dramatically different churn rates requiring distinct intervention strategies. A single churn model for your entire user base averages across groups whose behaviors have almost nothing in common.
• Aggregate retention metrics like CAC Payback Period mask critical heterogeneity in user value; decompose cohorts using five primitives (survivor rate, tail half-life, cliff intensity, headroom, payback period) to understand which user segments drive value and route insights to the right operational teams.
• Churn intervention requires splitting into two separate decisions that mature on different timelines: early exclusion of safe users (high precision immediately) and delayed targeting of at-risk users (precision matures later). Firing retention campaigns on early signals wastes budget by targeting users who would have stayed anyway.