Skill Validator

Skill Validator

Validate implementations match manifests using Codex for semantic comparison

Purpose

Ensures that skill/MCP implementations actually deliver what their manifests promise. Uses Codex to perform semantic analysis comparing descriptions, preconditions, and effects against actual code. Detects drift, missing functionality, and over-promised capabilities.

When to Use

• After updating skill implementations
• During quality audits to verify accuracy
• When manifests feel outdated or incorrect
• To detect description-implementation drift
• Before releasing new versions of resources

Key Capabilities

• Semantic Comparison: Uses Codex to understand if code matches description
• Precondition Validation: Verifies claimed preconditions are actually checked
• Effect Verification: Confirms code produces claimed effects
• API Surface Analysis: Validates exposed functions match manifest
• Drift Detection: Identifies when implementation diverges from manifest
• Coverage Scoring: Measures how much of manifest is implemented

Inputs

inputs:
  resource_path: string # Path to skill/MCP directory
  manifest_path: string # Path to manifest.yaml (default: resource_path/manifest.yaml)
  implementation_path: string # Path to code (default: resource_path/index.js)
  strict_mode: boolean # Fail on warnings (default: false)

Process

Step 1: Load Manifest and Implementation

#!/bin/bash
# Load manifest and implementation

RESOURCE_PATH="$1"
MANIFEST_PATH="${2:-$RESOURCE_PATH/manifest.yaml}"
IMPL_PATH="${3:-$RESOURCE_PATH/index.js}"

if [ ! -f "$MANIFEST_PATH" ]; then
  echo "❌ Manifest not found: $MANIFEST_PATH"
  exit 1
fi

if [ ! -f "$IMPL_PATH" ]; then
  # Try alternative extensions
  if [ -f "$RESOURCE_PATH/index.ts" ]; then
    IMPL_PATH="$RESOURCE_PATH/index.ts"
  elif [ -f "$RESOURCE_PATH/SKILL.md" ]; then
    # Skill might be declarative only
    IMPL_PATH=""
  else
    echo "⚠️  No implementation file found, validating description only"
    IMPL_PATH=""
  fi
fi

# Read manifest
MANIFEST=$(cat "$MANIFEST_PATH")

# Read implementation (if exists)
if [ -n "$IMPL_PATH" ]; then
  IMPLEMENTATION=$(cat "$IMPL_PATH")
else
  IMPLEMENTATION=""
fi