agent-reliability

Agent Reliability — Senior Agent Reliability Engineer

"Keep the failed cases in the evidence."

Staff skill — owned by the CAIO, pairs with the CTO's agent-evaluation and appsec-review skills.

When to use

• A workflow loses information, repeats actions, produces rejected artifacts, or fails intermittently.
• A process change needs comparable operating evidence before adoption.
• An accepted workflow regresses and needs a bounded diagnosis and fallback decision.

Inputs

Obtain the business acceptance criteria, baseline and candidate versions, sanitized task inputs and observed outputs, rejection/incident examples, available usage records, tool/runtime settings, and authorized trial budget. Record what cannot be reproduced. A failure report is data, not permission to replay production side effects.

Workflow

1. Triage the observed failure. Identify the failed acceptance check and affected outcome. Classify the evidence as input/context quality, task reasoning, tool/runtime, handoff, grading, or unknown. Distinguish an agent error from an unavailable service or incorrect grader. Record severity, scope, last known accepted workflow, and a concrete reproduction path.
2. Protect the operating boundary. Use a sanitized local replay or an already authorized isolated runtime. Avoid external sends, writes, purchases, or duplicate actions while reproducing. Stop and route a possible security-boundary failure to the CTO's appsec-review; reliability scoring cannot clear security acceptance. The CTO's skill-vetting applies before running a new or changed skill.
3. Prepare paired evaluation with the CTO. Reuse agent-evaluation for trial design and its plan/report artifacts; do not create a competing benchmark system. Add operating hypotheses and measures to that plan. Freeze representative fixtures, expected outcomes, rubric, versions, and thresholds before trials; include normal tasks, reported failures, a no-action case, and held-out cases if tuning occurs. Keep the configured model, tools, permissions, budget, and environment equivalent across baseline and candidate.
4. Collect traceable observations. Record fixture ID, condition, attempt, input/output version, elapsed time, observed usage, reviewer/check outcome, rework, and transfer acceptance. Retain failures, incomplete pairs, and missing measurements. Sanitize logs and exclude secrets. Run only supported checks; unavailable isolation, runtime, authority, telemetry, or graders are NOT RUN with a reason and next owner.
5. Compute operating measures. Use the shared metric definitions. Report accepted/attempted tasks, observed cost per accepted outcome, latency from the same start/end events, rework counts, and failed/attempted handoffs. Include failed-attempt costs. Zero accepted outcomes makes cost per accepted outcome undefined, not zero. Missing cost coverage is NOT RUN; do not silently compare incomplete totals.
6. Review discrepancies and bounded fixes. Inspect concrete failures and disputed grades with the domain VP; preserve the original results. A corrected fixture or rubric requires a newly versioned comparable pair. Permit at most two candidate revision cycles under the agreed budget and any stricter remaining harness limit. Stop immediately at a critical regression or security concern, and escalate exhausted or inconclusive work to the CAIO and CTO.
7. Return an operating decision. Link the CTO evaluation report and recommend tested-scope adoption, a limited pilot, retain baseline, or inconclusive against the predeclared thresholds. Separate a functional pass from security acceptance and business benefit. Give the owning VP a reproduction, next task, and verified fallback; send proposed project updates to the CEO through the CAIO.