deduplicated-file-caching

In-Memory File Ingestion Deduplication (mtime Cache Guard)

Overview

During multi-step debugging and refactoring sessions, AI agents frequently call view_file on the exact same file path 3 to 5 times (e.g., re-reading src/routes.ts at Turn 2, Turn 7, and Turn 14 without any intervening edits on disk).

Each redundant file read re-injects hundreds or thousands of identical tokens into the transcript. In a 20-turn session, redundant file reads account for up to 40% of total input token waste.

The File Ingestion Deduplication Protocol tracks file path access, modification timestamps (mtime), and SHA-256 hashes in the agent runtime - intercepting duplicate read requests and returning a lightweight reference token if the file is already resident in active context and has not changed on disk.

---

Redundant Re-Reading vs. In-Memory Deduplication

┌─────────────────────────────────────────────────────────────┐
│                 File Read Deduplication Flow                │
│                                                             │
│  Uncached Redundant Reads (Anti-Pattern):                   │
│  • Turn 2: `view_file("src/config.ts")` (1,200 tokens)      │
│  • Turn 7: `view_file("src/config.ts")` (1,200 tokens)      │
│  • Turn 12: `view_file("src/config.ts")` (1,200 tokens)     │
│  ↳ 3,600 tokens billed for the exact same file content!     │
│                                                             │
│  Deduplication Cache Guard:                                 │
│  • Turn 2: `view_file("src/config.ts")` $\rightarrow$ Ingests (1,200)│
│  • Turn 7: `view_file("src/config.ts")` $\rightarrow$ Intercepted!  │
│    ↳ Return: `[CACHE HIT: src/config.ts unchanged (Turn 2)]`│
│  ↳ 12 tokens billed, 99% savings on subsequent reads!       │
└─────────────────────────────────────────────────────────────┘

---

The Deduplication State Machine

┌───────────────────────────────────────────────────────────────────────────┐
│ 1. INSPECT REQUEST: Agent calls `view_file(path)`                         │
│ 2. CHECK RUNTIME CACHE: Is `path` in active context session?              │
│    • NO $\rightarrow$ Read from disk $\rightarrow$ Record `mtime` & `hash` $\rightarrow$ Ingest │
│    • YES $\rightarrow$ Check disk `mtime`:                                 │
│        • Unchanged $\rightarrow$ Intercept & return 1-line Cache Reference │
│        • Changed on disk $\rightarrow$ Ingest updated content $\rightarrow$ Update hash  │
└───────────────────────────────────────────────────────────────────────────┘