letta-failure-modes

Letta 8-Failure-Modes Diagnostic

Overview

The Letta Leaderboard for benchmarking agentic memory names eight distinct failure modes that tend to co-occur (Ch4):

1. No-retrieval-when-available — model fails to recognize when relevant info is already in memory; issues unnecessary searches.
2. Hierarchy-collapse — trivia in prime memory; critical facts archived or dropped.
3. In-conversation-misses — agent misses key pieces of info even when present in the immediate context.
4. Volume-degradation — retrieval accuracy degrades as data volume grows; performance drops at scale.
5. Silent-overwrite — new info overwrites old facts instead of being layered; system cannot explain how or why things changed.
6. Cross-reference-failure — related info isolated in separate silos; no pattern recognition.
7. Temporal-blur — event timelines blur; agent loses temporal coherence.
8. Threshold-collapse — works at hundreds of facts, quietly collapses at thousands.

This skill takes a snapshot of an agent's memory state (or a structural description) and reports which of these 8 are present, with evidence and recommended fixes. It runs static analysis — no agent inference loop required.

When to Use

• Pre-launch review of a new memory implementation
• Root-cause analysis when an agent in production is "forgetting" or "drifting"
• Periodic audit (weekly / monthly) on long-running agents
• Code review of a colleague's memory layer

Phrases: "audit my memory architecture", "why is my agent forgetting", "is my memory production-ready", "Letta Leaderboard", "memory diagnostic".

When NOT to Use

• Single failure mode you've already identified. If you know it's silent-overwrite, just go fix it; this skill's value is the cross-cutting audit, not the depth on one mode.
• Runtime monitor. This is a static diagnostic. Production needs metrics + alerts, not periodic full-audit invocation.
• Benchmark/eval. This does not produce a comparable accuracy number. For Letta Leaderboard scoring, run the actual benchmark suite.