resilient-io-and-jobs

Resilient I/O and Background Work

Overview

Everything outside your process can be slow, down, or duplicate its own delivery. The recurring defects: an HTTP client with no timeout (Go's default client has none), a retry on a non-idempotent call, a goroutine nobody waits for, and a consumer that isn't safe to redeliver to.

Core principle: every call out has a timeout, every retry is on an idempotent operation, and every background task is tracked, not fired and forgotten.

Outbound HTTP

• One shared *http.Client{Timeout: ...} per downstream — the zero-value http.DefaultClient has no timeout and will hang a goroutine forever on a stuck peer.
• http.NewRequestWithContext(ctx, ...) so a canceled request actually stops; defer resp.Body.Close(); check the status code before decoding; cap how much of the body you read (io.LimitReader) even on success.
• Retry only idempotent requests (GET, PUT, DELETE, or a POST carrying an idempotency key the server honors — api-design-conventions) with exponential backoff and jitter and a hard cap on attempts. Retrying a bare POST "to be safe" is how a flaky network turns into a duplicate charge or duplicate task.
• Java: configure timeouts on RestClient/the MicroProfile REST Client, and use the framework's fault-tolerance annotations (@Retry/@Timeout on Quarkus, Resilience4j on Spring) only where the repo already has the dependency — don't add a new resilience library for one call site.

Background work

• Never fire-and-forget a goroutine from a request handler (Uber guide) — if the handler returns before the goroutine finishes, there's no way to know it ran, and a panic inside it with no recover brings down the process.
• Work that must genuinely outlive the request uses context.WithoutCancel(ctx) (Go ≥1.21, keeps request-scoped values like the logger/trace id without inheriting cancellation) and is tracked so the process can wait for it on shutdown — sync.WaitGroup or wg.Go (Go ≥1.25).
• Graceful shutdown: signal.NotifyContext(ctx, os.Interrupt, syscall.SIGTERM), then srv.Shutdown(ctx) (app.ShutdownWithContext on Fiber) so in-flight requests finish instead of being cut off mid-response.