trajectory-eval

Trajectory Eval

An agent that passes every step's check can still fail the task. Errors compound: if each step is 95% reliable, a ten-step task succeeds only about 60% of the time, and the gap widens fast as tasks get longer. A final-answer eval hides this, because it scores the endpoint and ignores the path. A per-step eval hides it too, because it scores steps in isolation and ignores how their errors accumulate across the run.

This skill evaluates the whole trajectory: the steps, the tool calls, the intermediate states, and the way failures propagate or get recovered. It measures where reliability actually leaks, and whether chaining more agents into the system makes it better or just longer.

How to run it

Run trajectory-eval on this agent. Capture full trajectories on production-realistic tasks, measure per-step success against end-to-end success, and show the compounding curve. Attribute failures to the step that introduces them and say whether they cascade or recover. If the system chains or orchestrates agents, measure whether composition helps or hurts against a real baseline, with statistical validation. Test with tool failures and edge responses, not just the happy path. Verify every number with runnable code.

Core commitments

• Verified numbers only. Every figure traces to a measurement produced by runnable code.
• Evaluate the trajectory, not the endpoint and not the isolated step. The final answer and the per-step score are the two numbers that hide compounding failure. The object of study is the path.
• Measure compounding explicitly. Connect per-step reliability to trajectory length to end-to-end success, rather than assuming the steps add up.
• Test under realistic conditions. Real tool responses, including failures and malformed outputs, not a curated happy path. A clean demo trace proves nothing.
• Reproducibility is part of the deliverable. Seed everything. Report the compute envelope.

The procedure