llm-patterns

LLM Patterns

Patterns for building, evaluating, streaming, and testing LLM-powered applications.

Evaluation

Quality assessment and validation of LLM outputs using LLM-as-judge, multi-dimension scoring, pairwise comparison, hallucination detection, and batch evaluation. Integrates with Langfuse for score tracking.

• Use a separate judge model (e.g., GPT-4o-mini, Claude Haiku) to avoid self-evaluation bias
• Score across 3-5 dimensions: relevance, accuracy, completeness, coherence
• Quality threshold: 0.7 for production, 0.6 for drafts

Full patterns and code: See references/evaluation.md

Autonomous Evaluation Loop

Use /experiment to autonomously iterate on evaluator quality (judge prompts, scoring rubrics) against golden datasets, tracking agreement via Langfuse Experiments API. See references/autonomous-evaluation.md for the full pattern and a worked example.

---

Streaming

Real-time token delivery via Server-Sent Events (SSE) for better UX. Covers OpenAI sync/async streaming, FastAPI SSE endpoints, TypeScript frontend consumers, streaming with tool calls, and backpressure handling.

• Use SSE for web, WebSocket for bidirectional communication
• Buffer 50-200 tokens; set 30-60s timeouts for long responses
• Always close connections properly and handle stream errors

Full patterns and code: See references/streaming.md

---

Testing

Deterministic testing patterns for non-deterministic LLM outputs. Covers quality gate tests, template rendering, edge cases (empty/long inputs, partial responses), and VCR.py for recording API responses in CI.

• Use VCR for integration tests, mocks for unit tests
• Always test with < 1s timeout; validate both valid and invalid schemas
• Record mode "none" in CI to avoid live API calls

Full patterns and code: See references/testing.md