hierarchical-memory

Hierarchical Memory (Letta / MemGPT)

Overview

Letta popularizes a three-tier memory architecture that mirrors human cognition and CPU memory hierarchy. The architectural decomposition (Ch4):

• Core memory (cache layer): small, fast, structured. The agent's active reasoning context. Bounded (core_limit parameter). Once full, eviction is forced — you cannot have everything in core. This is the forcing function that pushes the agent to declare what is durably important.
• Recall memory (raw interaction layer): the literal conversation history. Append-only. Answers "what did we talk about yesterday."
• Archival memory (persistent layer): effectively unlimited storage for evicted-but-still-relevant facts. Searchable. Not deleted.

Eviction is the key discipline. Naive FIFO (oldest goes first) loses high-value durable facts that were learned early. Naive LRU (least-recently- used) loses background-but-relevant context. The chapter recommends a combined score: access frequency × recency, with explicit handling for "durable attributes" (peanut allergy) vs "short-lived states" (having coffee right now).

When to Use

• Long-running personal assistant agents — multi-session, must feel consistent over time
• Multi-day DevOps incident investigation where some facts (production region, on-call rotation) are durably important and others (current shell history) rotate fast
• Customer-support agents that need both "what we know about this customer" (core) and "what was said in last week's tickets" (archival)

Phrases that should invoke this skill: "the agent needs memory across sessions", "core context", "evict old memory", "MemGPT", "Letta hierarchy", "working memory vs long-term memory".

When NOT to Use

• One-shot agents. Single-prompt, no persistence — flat context is correct. The eviction overhead pays for nothing.
• Event logs / audit trails. Use append-only kafka-style logs. The recall layer here is interaction-oriented, not event-oriented.
• Every fact equally important. Then a flat KV store is right. The hierarchy exists because some facts are more important than others; if that gradient doesn't exist, the hierarchy is overhead.
• Hard real-time eviction is too slow. Default impl is O(n) on eviction. Production needs a heap for evict_least_used. Swap the internal index at the seam noted in lib.py.