Runtime Validation

Runtime Validation

Unit tests assert on mocked inputs and isolated functions. Runtime validation asserts on the actual running system — logs, real HTTP responses, and cross-layer data. A suite can be 100% green while the product is broken, because the mocks never hit the code path that fails. This skill is the discipline that closes that gap.

Why this exists (real forge incident, not hypothetical)

hooks/scripts/smoke-test-reminder.sh in this plugin carries the origin verbatim:

On 2026-02-06, an agent spent an entire day writing 2326 mock unit tests while the actual product was broken. A duplicate placeholder route intercepted all real requests. Nobody caught it because nobody started the actual server.

That is the whole thesis: a passing mock suite is not evidence the running system works. The three techniques below are how you get that evidence.

The three techniques

Runtime validation composes with your existing tests — it does not replace them. Run the unit suite for logic, then observe the running system for everything the mocks hid.

1. Log monitoring during test/server runs

The cheapest, highest-yield technique: watch the app's own logs for error signatures while you exercise it, and fail the run if any appear — even when every assertion passed.

# Tail logs while a test or smoke run executes; fail on known error signatures.
: > logs/app.log                      # truncate so you only see this run
npx vitest run                        # or: cargo test / pytest / your smoke script
if grep -Eiq 'validation error|panicked at|Input should be|type=\w+_error|Unhandled' logs/app.log; then
  echo "RUNTIME ERRORS in logs despite green suite:" >&2
  grep -Ei 'validation error|panicked at|Input should be|type=\w+_error|Unhandled' logs/app.log >&2
  exit 1
fi