Chatbot QA Roundtrip Validate

/chatbot-qa-roundtrip-validate

The contract the chatbot /auto-optimize loop calls before every commit. Returns pass / reject per the v2 plan's Harness-Engine rollback wiring.

This skill is intentionally read-only — it runs the oracle, checks the diff, and returns a verdict. It never edits code, never touches the oracle, never modifies the baseline. That's /auto-optimize's job to react to (revert if reject, commit if pass).

Inputs

Input	Source
before_metric	pass_pct from the previous oracle run (loop history tail)
after_metric	pass_pct from the oracle run just performed
diff_paths	git diff --name-only HEAD — what the proposed edit changed
baseline_path	state/quality/chatbot-qa/baseline.json (for gates)

In practice the loop passes the first three as values; the skill loads the baseline itself.

Verdict rubric

The skill returns pass if all four conditions hold; reject otherwise. Reject diagnostics include the failing condition by name so the loop can record a learning rather than just retry.

1. Metric did not regress

after_metric - before_metric >= -regression_threshold

where regression_threshold defaults to 0.02 per the baseline's _harness.rollback_metadata.regression_threshold field. A drop equal to or smaller than the threshold is tolerated as noise; anything larger is a reject (metric_regression).

2. Canonical-trace gate did not break

pwsh Scripts/compare-trace-to-canonical.ps1 -Sweep
# exit 0 = matched; exit 1 = at least one signature regression

If the sweep exits non-zero, reject with trace_shape_regression. The full diagnostic from the sweep is captured into the loop history.

3. No protected path modified

$baseline = Get-Content state/quality/chatbot-qa/baseline.json -Raw | ConvertFrom-Json
$protected = $baseline.scope_boundary.protected
$changed = git diff --name-only HEAD
foreach ($p in $changed) {
    foreach ($glob in $protected) {
        if ($p -like $glob) {
            return @{ verdict = 'reject'; reason = 'protected_path_modified'; path = $p; glob = $glob }
        }
    }
}