webhook-dx-audit

Webhook DX Audit

Audit how a platform's customers experience its outbound webhooks and event destinations, end to end, from discovery through to production, and produce a scored YAML audit file with specific, prioritized recommendations.

The subject is any company that sends events to its developers (Stripe, Shopify, Paddle, or a smaller platform). You evaluate what their integrating developers actually hit: docs, dashboard, signing, retries, observability, and tooling, using only what is public or already exposed in product.

Scope: webhooks AND event destinations. Treat "outbound webhooks" and "event destinations" as the same audit. The industry terminology is in flux: Stripe popularized "event destinations" (and now delivers directly to Amazon EventBridge and Azure Event Grid alongside webhooks), Shopify ships HTTP webhooks + EventBridge + Pub/Sub destinations and is rolling out "Event Subscriptions" branding, and others still call the whole thing "webhooks". The benchmark for what a modern offering should include is the Event Destinations initiative at https://eventdestinations.org. Score against that broader concept regardless of the platform's chosen label. For a webhook-only platform, criteria that target other destination types are Not Applicable (the destination type breadth criterion in category 6 still scores 0 because the breadth gap is real).

Three states + two scores. Each criterion ends up at 0/1/2, Not Supported (= 0 with intent labeled), Not Applicable (logical exclusion, dropped from math), or Not Assessed (couldn't reach, e.g. dashboard-gated in a Pass 1 run). Pass 1 produces two roll-ups from the same data: a Public-scope grade (what's reachable now) and a Provisional minimum (the floor if human-in-the-loop (HITL) verification never runs). See references/rubric.md for definitions and references/scoring.md for the math.

Audience matters. Declare the platform's intended audience at audit start: developer-platform (where integrators are software engineers), no-code-saas (where integrators are power users wiring up automations through a UI), or mixed (multiple audiences with the webhook surface serving a specific tier). Verify the designation by fetching the platform's homepage and citing specific signals (hero copy, nav structure, customer testimonials, pricing tiers, API prominence); see references/methodology.md step 0 for the checklist. The audience-driven N/A logic in rubric.md removes criteria that don't apply (e.g. IaC and local-dev workflow simulation are N/A for a pure no-code SaaS; under mixed you score by judgment per criterion). Default to developer-platform only as a Pass-1 fallback if the homepage cannot be reached; Pass 2 must revisit with HITL verification.

Perspective: this is a human developer's experience. Categories 1 through 11 score what a person integrating with the platform encounters, so read docs as a human reads them: the rendered HTML pages a developer visits, not .md or llms.txt exports. Whether those machine-readable doc formats exist is an AI-readiness signal scored only in category 12. Keep all AI and agent assessment inside category 12; do not let it bleed into the other eleven. (Fetching a formal API/event spec like OpenAPI for category 4 is fine; that serves human codegen and validation, and is not the same as reading a machine doc export in place of the human docs.)

Fetching the .md export of a page to extract a quote or speed up evidence collection is fine; the rule is that you score what the rendered HTML page presents to a human, not what the .md contains. If the two diverge, treat it as an evidence gap, not a free pass to use whichever is better.