create-evaluation

Create Evaluation

Run this skill when a user asks to create evals for a task, workflow, or output type.

Outcome

Produce all of the following in one flow:

1. Scoped evaluation dimensions with clear pass/fail boundaries
2. Deployed live eval endpoints
3. Full runnable cURL per endpoint (must include exact live eval ID and exact API key)
4. A generated companion skill that explains how to use the evals in the user's workflow

Default behavior

• Prioritize non-technical scoping first.
• Use binary evaluations by default.
• Create separate evals per dimension by default.
• Avoid asking implementation-detail questions unless they change product intent.
• Infer technical defaults and execute.

Interactive Q&A protocol (mandatory)

Anti-pattern: "This is obvious, skip questions"

Do not skip the interactive scoping loop, even when the use case seems simple. Fast assumption-heavy execution creates weak criteria and poor downstream behavior. Keep the dialogue short when possible, but do not skip it.

Checklist (complete in order)

You MUST complete each item in order:

1. Initial framing. Restate the use case and intended operator outcome.
2. Clarifying dialogue. Ask one question at a time; prefer multiple-choice when possible.
3. Approach options. Propose 2-3 decomposition options with trade-offs and recommendation.
4. Design approval loop. Present these sections and get approval after each section:
   • Quality dimensions
   • Pass/fail boundaries and strictness
   • Operational usage pattern (gate, rank, revise loop, monitor)
5. Seed labeling. Have the user label a small sample of traces to calibrate the LLM judge (see Seed labeling section below).
6. Build authorization checkpoint. Ask for explicit go-ahead before any MCP build or deploy action.
7. Implementation and verification. Execute from-scratch flow, verify, then deliver artifacts.