sir-thaddeus: Instructions file for GitHub Copilot

Local-Model Harness Tuning Workflow

Companion to e2e-harness-rules.instructions.md. Those rules stay in force — this file documents the operational loop: verify model, baseline, triage, patch, iterate. Written from a live session where we went from 25% pass rate to stable 54%+ with targeted fixes.

Read this BEFORE running the harness on a new model size. Most of the friction this session came from skipping a step below.

---

0. Sanity-check the model before anything else

Changing the model in LM Studio (or llama.cpp, or Ollama, etc.) is not enough. The runtime reads llm.model from:

%LOCALAPPDATA%\SirThaddeus\settings.json

Verify from PowerShell:

(Get-Content "$env:LOCALAPPDATA\SirThaddeus\settings.json" -Raw | ConvertFrom-Json).llm |
  Select-Object model, gatekeeperModelId, contextWindowTokens, maxTokens

Then confirm the LM Studio server actually has that id loaded:

curl -s http://localhost:1234/v1/models | python -m json.tool

If the response's data[].id doesn't match llm.model, either flip the settings or the LM Studio alias. Wrong model = wrong baseline = every fix you write is chasing a ghost.

1. Baseline before you patch

Always run the full baseline on the target model once before you change a single line of code. Failures on a 4B look very different from failures on a 2B — budgets, routing, and keyword coverage all shift.

# Stop any stale runtime/MCP first (the harness forks its own; stale ones eat GPU)
Stop-Process -Name 'Thaddeus.Runtime','SirThaddeus.McpServer' -Force -ErrorAction SilentlyContinue

# Single suite, quick sanity
.\dev\harness.ps1 --suite smoke --judge none

# Full baseline (30-90 min on 4B, 10-20 min on 2B)
.\dev\harness.ps1 --all --judge none

Capture pass/fail numbers per suite before you touch anything:

$suites = @('smoke','reasoning','web-search','quality','tool-contracts','personality','footman-validation','existence')
foreach ($s in $suites) {
  $out = .\dev\harness.ps1 --suite $s --judge none 2>&1 | Out-String
  $p = [regex]::Match($out, 'Passed: (\d+)').Groups[1].Value
  $f = [regex]::Match($out, 'Failed: (\d+)').Groups[1].Value
  "$s`t$p/$(([int]$p + [int]$f))"
}