verifying-external-behavior

Verifying External Behavior

Most integration bugs are not coding errors. They are a belief about someone else's system — a status code, a parameter name, a response shape, a build backend's scoping rule — that was never checked and turned out to be wrong. The code is written correctly against a contract that does not exist.

The fix is not more care. It is a probe: a throwaway command that asks the real system the exact question, before the code is written. Probes cost seconds. The bugs they prevent are silent, ship green, and are found months later.

Probe the exact call you are about to write

Not a similar call, not the documented example — the same endpoint form, the same library version, the same argument spelling.

# A library's real defaults / return shapes, with no venv to create or clean up
uv run --no-project --with somelib python -c "import somelib; print(somelib.Thing())"

# The exact URL, including the collection-vs-item distinction
curl -s -o /dev/null -w '%{http_code}\n' https://api.example.com/v1/things
curl -s -o /dev/null -w '%{http_code}\n' https://api.example.com/v1/things/1

# What a build backend actually put in the artifact
uv build && unzip -l dist/*.whl

# A real service instead of a fake, for one minute
docker run --rm -p 6390:6379 redis:7-alpine

Two rules make probes worth the minute they cost:

• Probe before you design around the answer. A probe that confirms a parameter tweak "should" fix a discrepancy is worth more than the tweak.
• Paste the probe command and its output into the PR, comment, or test. A finding with no reproduction decays into folklore, and the next person re-derives it — or, worse, trusts it after it has gone stale.

Permissive clients don't reject wrong arguments — they ignore them

This is the highest-severity class, because the failure returns plausible data. Many HTTP client wrappers forward keyword arguments verbatim into the query string without validating them against their own documented parameter list. The remote service then drops the unknown parameter and applies its default — often "the authenticated user". A misspelled selector (owner= for owner_screen_name=, id_= for id=) does not raise; it silently returns someone else's records.