runtime-validator

Runtime Validator

Execute runtime validation steps against the golden truth and each migration attempt to capture qualitative evidence that goes beyond static code analysis.

Inputs

• Golden truth directory path
• Attempt directories: a list of (name, path) pairs
• Validation config: either user-provided validation steps or a target runtime config file (e.g., targets/patternfly_runtime.yaml)
• Output directory: where to write runtime-validation.json

Validation Config

The validation config describes how to verify the migration works at runtime. It can come from:

1. Target runtime config file (preferred for known frameworks): YAML files in targets/ with default commands and routes
2. User-provided steps: natural language from the user interview (e.g., "run npm run dev, screenshot the dashboard page")

A validation config contains:

• setup_commands: commands to install dependencies (e.g., npm install)
• launch_command: command to start the application (e.g., npm run dev)
• launch_ready_signal: string to watch for in stdout indicating the app is ready (e.g., "ready in", "Compiled successfully")
• launch_timeout: seconds to wait for ready signal (default: 120)
• health_check: optional URL to verify the app responds (e.g., http://localhost:3000)
• test_command: optional test suite command (e.g., npm test -- --watchAll=false)
• routes_to_screenshot: list of {path, name} for visual comparison (e.g., [{path: "/", name: "home"}, {path: "/settings", name: "settings"}])
• cleanup_command: command to stop the app after validation (e.g., killing the dev server process)

Process

1. Load Validation Config

If a target runtime config file path is provided, read the YAML file. Otherwise, use the user-provided steps parsed into the config fields above.

2. Validate Golden Truth First

Run the validation against the golden truth directory to establish the baseline:

1. cd into the golden truth directory
2. Run setup_commands (e.g., npm install). If setup fails, report the error and skip runtime validation for this directory.
3. Run launch_command in the background. Watch for launch_ready_signal in stdout/stderr.
4. If health_check URL is provided, poll it until it responds (up to launch_timeout).
5. If test_command is provided, run it and capture stdout/stderr as test evidence.
6. If routes_to_screenshot are provided and Chrome MCP is available, screenshot each route. Save screenshots to $OUTPUT_DIR/screenshots/golden/<name>.png.
7. Stop the dev server (kill the background process).