backstage-incident-debug

Backstage Incident Debugging

Narrow a failing Backstage deployment to one layer with evidence before proposing a cause. Read-only by default; anything that mutates a shared environment stops for authorization.

Preconditions

• The exact symptom, its first-seen timestamp, and which environment. Without a timestamp you cannot correlate with deploys, and correlation is most of the diagnosis.
• Read access to the failing environment's merged config, not the repo's app-config.yaml. app-config.production.yaml overrides nearly everything that matters.
• Backend reachability plus a token if auth is enforced. A 401 with Missing credentials from every endpoint is your missing token, not the incident. External callers use backend.auth.externalAccess — type: static (with token, subject, optional accessRestrictions) or type: jwks.
• Release line from backstage.json, and yarn backstage-cli info --format json for Node, CLI, and resolved @backstage/* versions.
• Generation, because half the diagnostics below do not exist in the other one. Backend: createBackend() in packages/backend/src/index.ts is the new backend system; createRouter files under packages/backend/src/plugins/ are legacy. Frontend: createApp from @backstage/frontend-defaults is NFS (default since v1.49); @backstage/app-defaults plus <FlatRoutes> is legacy.

Procedure

1. Record symptom and blast radius before forming any hypothesis. One entity, one user, one plugin, or everyone. This single fact eliminates most layers: one entity is data or processing; one user is auth or permissions; one plugin is that plugin's config, backend, or upstream; everyone is process, config, or database.
2. Establish whether the backend is up. GET /.backstage/health/v1/liveness and GET /.backstage/health/v1/readiness (new backend system, v1.29.0+; legacy backends expose /healthcheck). Liveness OK with readiness failing means the process is alive and a dependency — usually the database — is not. Neither answering plus a restarting pod is a crash loop; jump to step 5.
3. Build the change timeline covering the 48h before first-seen: application deploy, changes to any app-config.* file, credential or secret rotation, a backstage.json bump (backstage-upgrade), permission policy change, and catalog-info.yaml changes in target repos. "Nothing changed" almost always means an expiring credential, a scheduled provider run, or an upstream quota reset.
4. Reproduce against the failing environment's merged config, never the local default. yarn backstage-cli config:print --lax --format yaml, passing --config for each file in the same order the deployment does; --frontend prints exactly what the browser receives; --with-secrets only if authorized. Validate with yarn backstage-cli config:check --lax --deprecated; --strict additionally rejects keys no schema declares. config:schema shows which keys are even known. BACKSTAGE_ENV takes comma-separated values, so you can stack the deployed config layers locally.
5. Split frontend from backend with the browser network tab, not with logs.
   • /api/* returning 4xx/5xx → backend; go to that plugin's layer below.
   • Request 200 but the data is wrong → data or processing, not transport.
   • Blank page with no failing request → the bundle got the wrong config. Frontend config is injected at container start by the nginx entrypoint from environment variables, not baked at build time, and a single missing variable makes the whole ${VAR} value evaluate to undefined. Diff config:print --frontend against what the deployment sets.
   • Origin and CORS errors are app.baseUrl / backend.baseUrl disagreeing with the real hostname, not a Backstage bug.
6. Raise log level surgically. LOG_LEVEL=debug (env var, takes precedence over config) is fine locally and a firehose in production. Prefer backend.logger.overrides with matchers: { plugin: catalog } and level: debug to raise one plugin, and backend.logger.meta to stamp every line with the environment. Levels are error, warn, info (default), debug.
7. Read the logs for the right lines. Every line carries the plugin id from the plugin-scoped logger — filter on it first. The useful signals are the startup sequence for the failing plugin, and the last successful scheduled-task line for the relevant provider. Catalog processing errors are not logged by default, so an absence of catalog errors in the log is not evidence of a healthy catalog.
8. Catalog layer. Query the catalog's own belief rather than reading provider code (backstage-catalog):
   • GET /api/catalog/entity-facets?facet=kind — a census. A whole integration that stopped ingesting is visible in one request.
   • GET /api/catalog/entities/by-query?filter=metadata.annotations.backstage.io/orphan=true — the orphan set.
   • GET /api/catalog/entities/by-name/<kind>/<ns>/<name> — read status.items; that is where processing errors actually live.
   • .../by-name/<kind>/<ns>/<name>/ancestry — which root is keeping the entity alive, or which one stopped.
   • GET /api/catalog/locations — registered roots. POST /api/catalog/refresh with { entityRef } forces one cycle instead of waiting out catalog.processingInterval.
9. Scaffolder layer. Find the task id from the task list page under /create/tasks, then read its status and event log. Read the exact backend route paths and client method names from the installed @backstage/plugin-scaffolder-backend router and @backstage/plugin-scaffolder-react types — they are version-sensitive. Tasks are database rows with a heartbeat, so a task claimed by a backend instance that died stays claimed.
10. TechDocs layer. Establish which of the four stages broke — annotation, mkdocs source, generator, publisher — by listing the storage bucket under the entity's lowercased <namespace>/<kind>/<name>/ prefix before touching anything else (backstage-techdocs).
11. Auth layer. Drive the flow by hand at /api/auth/<provider>/start?env=<auth.environment> and watch the callback. Watch /api/auth/<provider>/refresh in the network tab for session-persistence problems. Decode the issued token in the console with atob(token.split('.')[1]) and check sub and ent claims against the catalog.
12. Permissions layer. Confirm permission.enabled in the merged config for that environment — permissions are frequently on in production and off locally, which alone explains "works on my machine". Then confirm whether the policy sees the identity you think it does: an empty ownership claim set from the sign-in resolver denies everything a policy conditions on group membership.
13. Database layer. With no backend.database config, each plugin gets an in-memory SQLite database that is discarded on restart — that is the cause of "everything is empty again after every deploy". Otherwise check connection-pool saturation, which presents as readiness failing and every plugin slowing at once rather than one plugin breaking.
14. Integration layer. Rate-limit exhaustion is partial, not total: discovery returns fewer repos than it did, refreshes stall, 403s arrive in bursts, and it recovers on its own at the quota reset. GitHub Apps (integrations.github.apps) get substantially higher limits than a PAT; note that This endpoint requires you to be authenticated from a correctly configured app usually means the app is not installed on that organization, not that the credential is wrong.
15. Stop after three hypotheses. Form at most three candidate causes, each with a stated disproof test, and run them cheapest-to-disprove first. If all three are disproved, return a BLOCKED report containing:
    • the symptom, first-seen timestamp, and blast radius;
    • the change timeline you built in step 3;
    • the three hypotheses and the specific evidence that killed each;
    • the layer you narrowed to and what remains ambiguous within it;
    • the exact access, log, or authorization you would need to continue. Do not start a fourth round, and do not apply a speculative fix to see what happens.
16. Stop for authorization before any production mutation. Restarting a pod, re-running a provider, deleting or re-registering a location, cancelling or retrying a scaffolder task, editing deployed config, rotating a credential, and running a migration all change shared state and most of them destroy evidence. Propose the exact command, its blast radius, and how you would undo it, then wait.