messaging-queues

Messaging & Queues — Async Delivery That Survives Failure

A practical guide to moving work off the request path and across service boundaries without losing, duplicating, or reordering it into corruption. Stack-agnostic; concrete recipes target RabbitMQ, Kafka, AWS SQS/SNS, NATS, and Redis Streams as the reference brokers.

When to Use This Skill

• Introducing the first broker into a system that has only synchronous HTTP calls.
• Choosing between a work queue, an event log, and pub/sub fan-out for a new flow.
• Designing a consumer that must be safe to retry — idempotency, dedup, exactly-once illusions.
• Sizing retry/backoff, dead-letter queues, and poison-message handling before launch.
• Debugging duplicate processing, lost messages, or out-of-order delivery in production.
• Deciding partition keys / ordering guarantees for an event stream.

Skip when: the caller needs the result synchronously in the same request — that is an HTTP/RPC contract (see api-design / grpc-microservices), not a queue.

The First Decision — Queue vs Log vs Pub/Sub

These three are not interchangeable. Pick by who consumes and whether history matters.

Shape	Semantics	Each message goes to	Use when	Reference broker
Work queue	competing consumers, message removed on ack	exactly one worker in the pool	distribute jobs, scale horizontally	RabbitMQ, SQS
Event log	append-only, retained, replayable by offset	every consumer group, independently	event sourcing, audit, replay, multiple readers	Kafka, Redis Streams
Pub/Sub	fan-out, fire-and-forget, no retention	every live subscriber once	broadcast notifications, cache invalidation	SNS, NATS, Redis pub/sub

The classic mistake is reaching for Kafka when a work queue is wanted (now there is partition/offset/consumer-group machinery for a job dispatcher) or using a pub/sub broadcast when at-least-once durable delivery is required (subscribers offline at publish time silently miss the message).