ai-roi-audit

AI ROI Audit Skill

Every org now spends real money on AI tools, and most justify it with adoption counts ("80% weekly active!") — which measure enthusiasm, not return. This skill audits what the spend returned, using methods that survive a sceptical CFO: baselines, counterfactuals, and quality deltas, with "we can't know yet" said out loud where it's true.

What This Skill Produces

• A per-tool verdict table: keep / consolidate / renegotiate / cut, each with its evidence
• The measurement behind each number — method, baseline, confidence — so the audit is checkable
• A hidden-cost ledger (the part vendor ROI decks omit)
• A baseline plan for every "unknown", so next year's audit has data

Required Inputs

Ask for (if not already provided):

• The AI tool inventory with costs: subscriptions, API spend, seats — and utilisation if known
• What each tool was bought to do (the promised outcome, from the original business case if it exists)
• Available evidence: usage data, before/after metrics, time studies, quality data, anecdotes (labelled as anecdotes)
• The decision at stake: renewal? consolidation? budget defence? (calibrates depth)

Audit Method

1. Reconstruct the promise. Per tool: what outcome justified the purchase — time saved, quality improved, headcount avoided, revenue created? A tool without a stated outcome gets audited against the best-fit guess, flagged as retrofitted.
2. Score with the strongest method the evidence allows, in descending order of credibility:
   • Natural experiment — teams/periods with vs without the tool, same work (best available in most orgs)
   • Before/after with baseline — the metric before adoption vs after, seasonality noted
   • Task-level time study — 10-20 real tasks timed with/without (cheap to run during the audit — do it rather than skip to tier 4)
   • Structured self-report — users estimating time saved, discounted (self-reported AI savings run ~2× actuals; say so) Never present a tier-4 number with tier-1 confidence. Every figure carries its method and a confidence label.
3. Count the hidden costs. Verification time (humans checking AI output), rework from AI errors that shipped, licence sprawl (seats bought > seats active), integration/prompt-maintenance time, and training time. These come off the gross benefit — an ROI audit that skips them is a vendor deck.
4. Convert honestly. Time saved → money only via a stated loaded rate and a stated assumption about what the time became (more output? earlier finishes? — different values). "Saved 400 hours" that nobody redeployed is capacity, not cash; label which one you're claiming.
5. Verdict per tool. Keep (positive with tier ≤2 evidence) · Consolidate (positive but duplicative — name the overlap) · Renegotiate (positive but mispriced vs utilisation) · Cut (negative or unmeasurable after a fair baseline attempt). Ties break toward the tool with a measurement plan.
6. Leave the audit better than you found it. Every "unknown" verdict gets a baseline plan: the metric, how it's instrumented, and the review date. The first audit is mostly this; that's a finding, not a failure.