spot-ai-mistakes

Spot AI Mistakes

AI fails in patterned, learnable ways — it invents citations, states outdated facts with total confidence, agrees with whatever you imply, and fumbles arithmetic while sounding certain. Once you know the patterns and their tells, you catch most errors on sight. This maps the failure modes most relevant to how you use AI, the signs that give each away, and a fast check for the ones that would actually hurt — building the instinct so confident-wrong stops catching you off guard.

What This Skill Produces

• The failure patterns that matter for you — the specific ways AI goes wrong in your use (hallucinated facts, fabricated citations/quotes, outdated info, sycophancy, arithmetic slips, false precision, missed nuance, confident guessing)
• The tells for each — the signals that give a mistake away (suspiciously specific sources, confidence on recent/niche topics, agreeing too readily, round-number math)
• A fast check for the dangerous ones — a quick way to catch the errors that would actually cost you, without over-checking everything
• A calibrated trust level — where AI is reliable for your uses and where it isn't, so trust is earned per-domain not blanket
• The instinct, built — the habit of pattern-matching for these tells as you read AI output

Required Inputs

Ask for these if not provided:

• How you use AI — the domains and tasks (points at which failure modes matter most)
• A past miss — a time AI got something wrong on you, if you have one (great teacher)
• The stakes — what a missed error would cost in your use
• Your trust level now — where you're too trusting or too skeptical

Framework: Know The Patterns, Read The Tells

1. Map failures to your use. The failure modes that matter depend on how you use AI — a coder cares about wrong APIs, a researcher about fake citations, a student about outdated facts. Focus on yours.
2. Learn the tells. Each failure has signals: fabricated citations look oddly specific and un-Google-able; hallucinations spike on recent/niche topics; sycophancy shows as agreeing right after you hint at a preference; math errors hide in confident round numbers.
3. Watch the confidence trap. AI's tone is uniform whether it's right or inventing — so confidence is not a signal of correctness. Judge by pattern and verification, never by how sure it sounds.
4. Check the dangerous ones fast. For the errors that would actually hurt, a quick independent check (a source, a second tool, testing it) — don't over-verify the low-stakes stuff.
5. Calibrate trust per domain. Trust AI more where it's reliable for you and less where it isn't — calibrated, not blanket trust or blanket suspicion.