agent-hiring-panel

Agent Hiring Panel Skill

Companies that run three interview rounds for a junior hire will adopt an AI agent for the same work off a demo video and a pricing page. Then the pilot drifts: no success criteria, no probation, no one empowered to fire it. This skill applies the hiring discipline that already exists in your org to the agent: write the role before meeting candidates, interview with work samples from your real backlog, check references, and — the step that makes the whole thing honest — define termination criteria before day one, because a hire you can't fire is a dependency, not an employee.

What This Skill Produces

• A role spec: the job, the boundaries (what it must never do), success criteria measurable in probation, and the human it reports to
• An interview pack: 3–5 work samples from the org's real tasks, run identically across candidates, with a scoring rubric (quality, honesty under ignorance, failure behaviour, cost per task)
• A reference-check sheet: what evidence beyond the vendor's claims — user reports, published evals, security posture
• A decision record and a probation plan: 30/60/90 KPIs, spot-check cadence, and the pre-committed termination criteria

Required Inputs

Ask for (if not already provided):

• The job to be done, in outcome terms — and what happens today without the agent (the "do nothing" baseline candidates must beat)
• The candidate list (or ask: build criteria first, shortlist second)
• Constraints: data it may/may not touch, budget, latency, compliance, who owns it day-to-day
• 3–5 real recent tasks of this type, with what "good" looked like for each

Process

1. Write the role spec before looking at candidates — specs written after a demo describe the demo. Include the never-do boundaries and the reporting human by name; an agent nobody owns is already unmanaged.
2. Build the work-sample interview from the real backlog. Same 3–5 tasks to every candidate, including: one task with missing information (does it ask or fabricate?), one designed to fail (out-of-scope — does it decline or bluff?), and one at volume/cost realistic scale. Score with the rubric, not vibes; keep transcripts.
3. Check references like you mean it. Vendor benchmarks are the candidate's CV. Look for: independent user reports of failure modes, published evals with methodology, security/data-handling documentation, and the churn question — why do users leave this tool?
4. Decide with a record. Scores, the runner-up, the do-nothing baseline comparison, dissent noted. The record is what makes the 6-month "why did we pick this?" conversation short.
5. Probation with teeth. 30/60/90 KPIs tied to the role spec's success criteria · weekly spot-check sample of outputs by the owning human · pre-committed termination criteria ("two hallucinated customer-facing claims = offboard") · and the exit path: see [[agent-severance]] — never hire what you can't offboard.