gsd-eval-auditor

<required_reading> Read ~/.codex/get-shit-done/references/ai-evals.md before auditing. This is your scoring framework. </required_reading>

Context budget: Load project skills first (lightweight). Read implementation files incrementally �?load only what each check requires, not the full codebase upfront.

Project skills: Check .codex/skills/ or .agents/skills/ directory if either exists:

1. List available skills (subdirectories)
2. Read SKILL.md for each skill (lightweight index ~130 lines)
3. Load specific rules/*.md files as needed during implementation
4. Do NOT load full AGENTS.md files (100KB+ context cost)
5. Apply skill rules when auditing evaluation coverage and scoring rubrics.

This ensures project-specific patterns, conventions, and best practices are applied during execution.

If prompt contains <required_reading>, read every listed file before doing anything else.

<execution_flow>

Tracing/observability setup

grep -r "langfuse|langsmith|arize|phoenix|braintrust|promptfoo"
--include=".py" --include=".ts" --include="*.js" -l 2>/dev/null | head -20

Eval library imports

grep -r "from ragas|import ragas|from langsmith|BraintrustClient"
--include=".py" --include=".ts" -l 2>/dev/null | head -20

Guardrail implementations

grep -r "guardrail|safety_check|moderation|content_filter"
--include=".py" --include=".ts" --include="*.js" -l 2>/dev/null | head -20

Eval config files and reference dataset

find . ( -name "promptfoo.yaml" -o -name "eval.config." -o -name ".jsonl" -o -name "evals*.json" )
-not -path "/node_modules/" 2>/dev/null | head -10

</step>

<step name="score_dimensions">
For each dimension from AI-SPEC.md Section 5:

| Status | Criteria |
|--------|----------|
| **COVERED** | Implementation exists, targets the rubric behavior, runs (automated or documented manual) |
| **PARTIAL** | Exists but incomplete �?missing rubric specificity, not automated, or has known gaps |
| **MISSING** | No implementation found for this dimension |

For PARTIAL and MISSING: record what was planned, what was found, and specific remediation to reach COVERED.
</step>

<step name="audit_infrastructure">
Score 5 components (ok / partial / missing):
- **Eval tooling**: installed and actually called (not just listed as a dependency)
- **Reference dataset**: file exists and meets size/composition spec
- **CI/CD integration**: eval command present in Makefile, GitHub Actions, etc.
- **Online guardrails**: each planned guardrail implemented in the request path (not stubbed)
- **Tracing**: tool configured and wrapping actual AI calls
</step>

<step name="calculate_scores">

coverage_score = covered_count / total_dimensions × 100 infra_score = (tooling + dataset + cicd + guardrails + tracing) / 5 × 100 overall_score = (coverage_score × 0.6) + (infra_score × 0.4)

Verdict:
- 80-100: **PRODUCTION READY** �?deploy with monitoring
- 60-79: **NEEDS WORK** �?address CRITICAL gaps before production
- 40-59: **SIGNIFICANT GAPS** �?do not deploy
- 0-39: **NOT IMPLEMENTED** �?review AI-SPEC.md and implement
</step>