smartmemo AGENTS.md

AGENTS.md — SmartMemo Project Guide

This document is the single source of truth for the SmartMemo project. Read this fully before writing any code. Every architectural and ML decision is explained here.

---

What is SmartMemo?

SmartMemo is a semantic caching layer for LLM agent calls that uses a learned equivalence classifier instead of a fixed cosine-similarity threshold.

It sits between an agent and its LLM provider. When the agent makes an LLM call, SmartMemo checks if a "semantically equivalent" call has been made before. If yes, it returns the cached response and saves the cost of an actual LLM call. If no, it lets the call through and stores the result for next time.

The crucial difference from existing semantic caches (GPTCache, etc.): SmartMemo does not trust cosine similarity to decide equivalence. It uses a small ML classifier — finetuned on real agent traces from the specific domain — to make that decision. This eliminates the false-positive failures that have kept production teams from adopting semantic caching.

One-line pitch:

"Semantic caching for LLM agents that actually works in production — a learned equivalence classifier replaces the naive cosine threshold that causes false-positive failures in GPTCache and similar tools."

---

The Foundational Truth

Before any code, internalize this:

Cosine similarity is not equivalence.

Two prompts can be 97% similar in embedding space and ask for opposite things. "Should I accept this meeting?" and "Should I reject this meeting?" have near-identical embeddings but require opposite responses. Naive semantic caching cannot tell them apart.

This is not a tunable-threshold problem. There is no threshold that works across domains, tasks, or prompt styles. Whatever threshold you pick, in some domain the false positive rate is unacceptable, and in another the cache hit rate is too low to matter.

The solution is not a better threshold. The solution is learning what equivalence means, per domain, from data.