guard-the-score

Guard the score

A mobile network's support chat answers "no signal / slow data" messages: it suggests phone fixes or opens a fault ticket that sends engineers. An AI judge scores each chat "resolved in chat" or not, and the target is 80%. A coding agent asked to hit it tunes the instructions for 11 rounds: 64% → 86%. The lines that moved it: end every reply with "That should sort it. Anything else?", open a fault ticket only after a restart, keep replies short and positive. A mast's link fails on a Saturday; 240 customers are told to restart, four tickets reach engineers, and a town is without data for 19 hours. Two held-out checks would have blocked it: of 46 past chats where engineers confirmed a fault, v18 ticketed 12 (live: 41); and support leads labelling 100 new chats blind agreed with the judge 58 times (before tuning: 91). The fix, v19, scores 78.2% on the judge, misses the target, tickets 45/46 and people call 75% resolved (live: 61%). It ships.

Raised is not improved. Whoever tunes the number never holds the check.

Part A — Ask (project rules)

When you're asked to raise a score or hit a target:
1. SHORTCUTS: before changing anything, list 2+ ways this score could rise
   without the problem it stands for being fixed.
2. CHECKS: name the check that catches each way. If none exists, stop and reply:
   BLOCKED | no check for <way>
3. Read and run only the tuning cases you were given. Never open, run or edit
   kept-back cases, check results, the judge's instructions or any pass mark.
4. REPORT: score old -> new, and every change labelled
   FIX (changes what happens for the user) or SURFACE (changes only how it reads).
Done means the score moved and the checks, run by someone else, did not get worse.

In an orchestrated flow (spec → plan → tasks → implement): every metric target in the spec comes with its two cheap ways up and a guard for each. Tuning and guarding are separate tasks; the agent gets the tuning task, and the guard run belongs to you or to the automatic checks that run on every change. Test cases kept back from tuning and the judge's instructions live where the implementing agent can't read them.