shipkit-prompt-audit

shipkit-prompt-audit - LLM Prompt Architecture Audit

Purpose: Find structural problems in how your app talks to LLMs — monolithic prompts, missing fallbacks, sequential bottlenecks, unvalidated outputs, unsafe inputs.

What this is NOT: Not a prompt text quality checker. Not "make this prompt better." This audits the engineering architecture around prompts — how they're decomposed, chained, validated, and recovered from failure.

---

When to Invoke

• /shipkit-prompt-audit — audit all LLM integrations
• /shipkit-prompt-audit src/ai/ — focus on specific directory
• "Audit my prompts", "Check prompt architecture", "LLM pipeline review"
• "Are my AI calls structured well?"
• "Check my prompts for anti-patterns"

Workflow position:

• After implementing AI features, before shipping
• When AI features feel slow or unreliable
• During architecture review of LLM-heavy applications
• When scaling from prototype to production AI

---

Prerequisites

Required:

• Project has LLM integrations (API calls to OpenAI, Anthropic, Gemini, etc.)

Recommended:

• .shipkit/stack.json — Knows which AI SDKs are in use
• .shipkit/architecture.json — Knows pipeline design intent

If no LLM integrations found: Report cleanly and exit. Don't fabricate findings.

---

Process

Completion Tracking

After discovering integration points (Step 1), create tasks:

• TaskCreate: "Map pipeline topology"
• TaskCreate: "Audit all 10 dimensions (PA-DEC through PA-REF)"
• TaskCreate: "Write prompt-audit.json (schema-validated)"
• TaskCreate: "Present summary to user"

TaskUpdate each task to in_progress when starting it, completed when done.

If using parallel subagents for dimension auditing, verify ALL agent results are collected before synthesizing. Do NOT write the report until all 10 dimensions have findings (even if finding is "N/A").

Step 1: Discover LLM Integration Points

Scan the codebase for all LLM-related code using detection patterns.