cached-system-instructions

Cached System Instructions (KV-Cache Prefix Architecture)

Overview

In Large Language Models, processing input tokens requires running dense matrix multiplications across the entire input sequence on every turn. In a 50-turn agent session with a 4,000-token system prompt, the provider re-processes the exact same 4,000 tokens 50 separate times ($200,000\text{ redundant token computations}$).

Prompt Caching (Anthropic Prompt Caching, OpenAI Automatic Prefix Caching, DeepSeek Context Caching) saves the transformer's Key-Value (KV) Activation States in GPU memory. Re-sending an identical prefix reuses the precomputed KV cache, cutting Time-to-First-Token (TTFT) latency by 80% and applying an automatic 90% cost discount.

The Hierarchical System Caching Protocol organizes instructions to guarantee uninterrupted prefix matching.

---

Uncached Re-computation vs. KV-Cache Hit

┌─────────────────────────────────────────────────────────────┐
│                 KV-Cache Mechanics Comparison               │
│                                                             │
│  Uncached Execution (Every Turn):                           │
│  • Re-computes Attention Matrices for 4,000 System Tokens   │
│  • TTFT Latency: 2.8 seconds                                │
│  • Full Input Rate: $3.00 / M tokens                        │
│                                                             │
│  KV-Cache Hit (Unbroken Static Prefix):                     │
│  • Directly loads KV-tensors from GPU VRAM                  │
│  • TTFT Latency: 0.35 seconds (8x Faster!)                  │
│  • Cached Input Rate: $0.30 / M tokens (90% Cost Discount)  │
└─────────────────────────────────────────────────────────────┘

---

The 4-Layer Caching Hierarchy

To prevent accidental cache invalidation, place instructions in order of decreasing stability:

┌───────────────────────────────────────────────────────────────────────────┐
│ LAYER 1: Base Agent Identity & Tool Schemas (100% Immutable) ──► CACHED   │
│ LAYER 2: Repository Architecture Blueprint (Static for Session)──► CACHED │
│ LAYER 3: MCP Tool Definitions & Skill Instructions         ──► CACHED     │
│ ───────────────────────────────────────────────────────────────────────── │
│ LAYER 4: Dynamic User Requests & Shell Outputs (Dynamic Tail)──► UNCACHED │
└───────────────────────────────────────────────────────────────────────────┘