agentsop-domain-eval-set

domain-eval-set — Your Held-Out Domain Benchmark

"Compiled program beats baseline on a held-out test set (not the val set used in optimization)." — DSPy SOP exit criterion [dspy.ai/learn/optimization/overview/]

"Build the eval loop before optimizing anything. Every subsequent change must be gated on these numbers." — LlamaIndex SOP Stage 2

This is an ENHANCE overlay skill. It produces one artifact — a versioned, sealed, human-labeled set of 50–200 examples drawn from your domain — that other skills consume: [[agentsop-regression-gate]] enforces it on every PR, [[agentsop-metric-design]] defines the scoring function applied to each example, and [[lm-evaluation-harness]] runs the complementary public-capability axis. The core claim: public benchmarks tell you the model is smart in general; only a held-out domain set tells you it works on your task. The latter is the one that predicts production.

---

1. 何时激活 (When to Activate)

Activate when any of these is true:

• "Does THIS system work on OUR data?" — someone is about to ship or trust an LLM/RAG/agent system and the only evidence is vibes, a demo, or a public benchmark number. You need a quantitative answer on the real distribution.
• A public-benchmark number is being used as a deployment gate. Someone cites "92% on MMLU" or "passes HumanEval" to justify go-live. That measures general capability, not your task fit (AP-1). Force a domain set into the decision.
• A model / prompt / retriever / chunking change needs a regression gate and no domain test set exists yet to gate against. You must build the set before [[agentsop-regression-gate]] can do its job.
• Switching models (GPT-4o → a cheaper or local model). The public-bench gap may be small while the domain gap is large, or vice versa. Only your held-out set tells you which.
• Production complaints don't match your eval scores. Either the set is stale (refresh, OP-DE06) or it never reflected the domain (rebuild from real traffic).