messaging-streaming

Messaging & Streaming

Move work off the synchronous request path and decouple producers from consumers, so a slow, spiky, or failure-prone operation doesn't block the caller. Getting this wrong is subtle: a queue silently changes the delivery and ordering guarantees, and under load it can absorb a spike gracefully or become the thing that hides a meltdown until the backlog is unrecoverable.

When to reach for this

A step is too slow to do inline (image transcode, fan-out, third-party call); the write path is spiky and needs a buffer to smooth bursts (→ back-of-the-envelope for the spike factor); two services must be decoupled so one can fail or deploy independently; or many consumers need the same event stream. The async hand-off buys responsiveness, isolation, and elasticity (scale producers and consumers separately).

When NOT to

The caller needs the result now to proceed (a synchronous read, a balance check before confirming) — a queue only adds latency and a place for work to get lost. Strong read-after-write within one request. Trivial in-process work that a function call handles. Don't add a broker before a number or a coupling problem justifies it (YAGNI): it's a new stateful system to operate, monitor, and reason about under failure. "We'll need Kafka eventually" is name-dropping, not a requirement.

Clarify first

• Sync or async? Does the caller need the result inline, or is fire-and-react acceptable? This decides whether a queue belongs here at all.
• Delivery guarantee needed — is a dropped message acceptable (at-most-once), or must every message be processed (at-least-once + idempotent consumers)?
• Ordering — must messages be processed in order, globally or per-key (per user, per account)? Global ordering is expensive; per-key usually suffices.
• Throughput and retention — messages/sec at peak, and how long must they be replayable? (→ back-of-the-envelope.) One-shot work vs. a replayable log.
• Consumer count and pattern — one worker pool draining a job, or many independent subscribers each reading every event?
• Failure handling — what happens to a message that keeps failing? Where does it go, and who looks at it?