product-analysis

Product Analysis

Purpose

Understand how a product is actually used and decide what to do about it. The failure mode is a dashboard full of numbers that go up, none of which are connected to whether the product is working.

When to Use

• Deciding what to build next.
• A metric moved and nobody knows why.
• Assessing whether a feature worked.
• Setting up product analytics.

Capabilities

• Metric selection: the one that matters versus the ones that flatter.
• Funnel analysis and drop-off diagnosis.
• Retention and cohort analysis.
• Feature-adoption measurement.
• Prioritization on evidence.

Inputs

• Usage data, at the event level.
• What the product is meant to do for the user.
• The decision this analysis informs.

Outputs

• The metric that actually reflects value, and where it stands.
• The specific point of failure in the funnel, or the specific cohort that churns.
• A prioritized recommendation.

Workflow

1. Choose the metric that reflects value received — Not signups, not page views, not "engagement". What is the action that means the user got what they came for? That is the metric.
2. Look at retention before acquisition — A product with a leaking bucket does not need more water. If week-4 retention is 8%, acquisition spend is being poured into a hole.
3. Segment before concluding — An aggregate number hides everything. A flat retention curve can be two cohorts: one that retains at 60% and one at 2%. Those require completely different responses.
4. Find the drop-off, then find out why — The funnel tells you where users leave. It never tells you why. That requires session recordings, support tickets, or asking them.
5. Distinguish a movement from noise — A 6% week-on-week change on a small base is noise. Before declaring a trend, check whether the change exceeds the normal variance.
6. Recommend something specific — With the expected impact and how you will know if it worked.

Best Practices

• Vanity metrics go up regardless of whether the product works. Total registered users, cumulative page views, and total revenue since launch can only increase. If a metric cannot go down, it cannot tell you anything.
• Retention is the product metric. Everything else — acquisition, activation, revenue — is downstream of whether people come back.
• A cohort retention curve that flattens has found product-market fit for that cohort. One that goes to zero has not, regardless of how good the early numbers look.
• The aggregate hides the answer. Always segment: by acquisition channel, by cohort, by use case, by company size.
• A funnel identifies where users leave. It cannot tell you why, and guessing at the why is how teams ship the wrong fix.
• Before acting on a change, check whether it is larger than the week-to-week noise. Most "the metric moved" investigations are investigations of noise.