foundations-information-theory

Information Theory Foundations

When to Apply

Apply information-theory when:

• Compressing prompts, retrieval contexts, logs, or feature sets
• Drift detection — distribution shift from baseline (KL, JS divergence)
• Feature selection by mutual information with target
• Retrieval re-ranking, MMR, or diversity-aware candidate selection
• Prompt-quality diagnosis via output-conditional entropy / Fano bound
• Hallucination / abstention gating via semantic entropy over meaning-clustered samples (#1)
• RL post-training diagnostics — policy-entropy collapse is the dominant failure mode in RLVR (#1)
• Agent-to-agent message budgets and KV-cache handoff sizing, framed as a bottleneck/rate problem (#6, #8)

Skip and use simpler alternatives when:

• Question is about causation, not information — use foundations-causal-inference
• Single-feature linear correlation is sufficient — Pearson r is cheaper than MI for monotonic continuous data
• Streaming data with hard latency budget — full MI/KL is too slow; use sketches or sampled approximations
• N samples too small for stable entropy estimate (rule of thumb n > 5 × #bins per variable)
• Problem is system-stability or feedback control — use foundations-control-theory
• Bits/nats unit doesn't map to a business decision — risk of treating it as decoration, not signal

11 applied information-theory primitives for quantifying uncertainty, signal, and compression, backed by a formal theory map. Each primitive solves a specific measurement problem. Primitives are domain-agnostic: the same entropy calculation that budgets a context window also bounds a lossless compressor; mutual information that scores retrieval also measures feature relevance in ML.

Contents

• Quick Reference
• Primitive Index
• Formal Supporting Theory
• Anti-Patterns
• Misuse Boundaries
• Decision Checklist
• Composition Recipes
• Workflow
• ASCII Flow
• Practitioner Judgment
• Navigation
• Related Skills
• Fact-Checking