context-optimization

Context Optimization

The Law

PROMPT COST IS NOT OPTIMISED BY GUESSING.
"I already have a short prompt" is a guess about token count.
"Reducing context will hurt quality" is a guess about the quality/cost curve.
Measure first. Apply the hierarchy. Measure again. THEN claim improvement.

When to Use

Trigger when:

• Prompt + context exceeds 60% of the model's context window
• p95 latency is above target
• Cost per 1k calls exceeds budget
• RAG pipeline is stuffing too many chunks into each call
• Output quality is degrading on long inputs (long context dilution)

When NOT to Use

• The feature has < 100 calls/month and total monthly cost is < $20 — optimisation ROI is negative at this volume
• Quality is currently below threshold — fix quality first; optimising a broken pipeline only makes it cheaper to be wrong
• The system prompt changes every call — prompt caching (Level 5) has no effect; skip straight to Level 1–3

The Reduction Hierarchy

Apply in order. Stop when the target is met. Do not apply all steps preemptively.

Level 1 — Trim the System Prompt

Audit every sentence:

• Run the prompt without each instruction — does output quality change?
• Remove any instruction the model follows without it
• Remove few-shot examples that duplicate knowledge the model already has
• Replace verbose explanations with single directives

Target: system prompt under 500 tokens for most tasks. Measure token count precisely:

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
print(len(enc.encode(system_prompt)))

Level 2 — Reduce Retrieved Context

If using RAG, before any other change:

1. Reduce top_k by 1 and run eval-before-ship — recall often holds at lower top_k
2. Add or tighten a similarity threshold (filter chunks below 0.75 cosine similarity)
3. Set a hard context budget: MAX_CONTEXT_TOKENS = context_window × 0.4

Level 3 — Compress Retrieved Content

Before injecting:

• Strip document headers, footers, and repeated boilerplate
• Normalise whitespace and remove formatting artefacts
• Convert verbose JSON to compact tabular format
• For chunks over 600 tokens: LLM-summarise to 1/3 length with a fast model