ai-output-validation

Overview

AI models produce unstructured text by default. In production pipelines, unstructured outputs cause brittle parsing, unexpected behavior, and silent failures. This skill enforces structured output generation and validation at every AI system boundary.

When to Use

• Any AI pipeline where output is used programmatically (not just displayed to a user)
• When AI output feeds into another system, database, or agent
• When building agentic systems that make decisions based on AI output
• When AI generates code, SQL, JSON, or other structured formats

Process

Step 1: Define Output Schema Before Prompting

1. Define the exact structure you need BEFORE writing the prompt.
2. Use JSON Schema or Pydantic/Zod models to formalize the expected output.
3. Example schema:

   {
     "type": "object",
     "required": ["summary", "action", "confidence"],
     "properties": {
       "summary": {"type": "string", "maxLength": 200},
       "action": {"type": "string", "enum": ["approve", "reject", "review"]},
       "confidence": {"type": "number", "minimum": 0, "maximum": 1}
     }
   }
   

4. Design the schema to be minimal — only what you actually need.

Verify: Schema is defined and versioned before any prompt is written.

Step 2: Prompt for Structured Output

1. Explicitly instruct the model to output in your defined format.
2. Include the schema or an example in the prompt.
3. Use models/APIs that support structured output natively where available (OpenAI structured outputs, Gemini JSON mode, Anthropic tool use).
4. Prompt pattern:

   Respond ONLY with valid JSON matching this schema:
   {schema}
   
   Do not include explanation or markdown. Output raw JSON only.
   

Verify: Prompt explicitly requests structured output with schema reference.

Step 3: Validate and Parse Output

1. Parse the output against your schema — never use raw AI output directly.
2. If parsing fails:
   • Log the raw output and the parse error
   • Retry with a clarification prompt (max 2 retries)
   • After 2 failures: return a structured error, not a crash
3. Validate semantic constraints beyond the schema:
   • Is the confidence score consistent with the action?
   • Are referenced IDs in the database?
   • Are dates in the valid range?