message-queue

Message Queue (RabbitMQ / AMQP)

A message queue decouples producers from consumers and provides durability for async work. The patterns in this skill apply to RabbitMQ — the most common AMQP broker — and generally to any AMQP-compatible system.

When to Use

• Processing work asynchronously without blocking an HTTP request (document indexing, email sending, report generation)
• Communicating between services without tight coupling (service A publishes an event, service B consumes it)
• Ensuring a task is processed at least once even if the consumer crashes mid-processing
• Rate-limiting work against an external API by controlling consumer concurrency
• Your thesis: RabbitMQ for the document processing pipeline (chunking, embedding, indexing)

When NOT to Use

• Synchronous request-response — use HTTP or gRPC
• Simple background jobs that can run in the same process — use setImmediate or asyncio.create_task
• Real-time bidirectional communication — use WebSocket (see websocket-realtime)
• Pub/sub for WebSocket fan-out — Redis pub/sub is simpler for that use case

Core Concepts

Exchange types:

Type	Routing	Use case
direct	Exact routing key match	Task queue — route jobs to specific worker type
topic	Pattern match (*.error, docs.#)	Event routing by category
fanout	All bound queues	Broadcast to all consumers (notifications)
headers	Message header attributes	Rarely needed; prefer topic

Message acknowledgment modes:

• ack — message successfully processed, remove from queue
• nack (requeue: true) — processing failed, return to queue for retry
• nack (requeue: false) — processing failed permanently, route to dead-letter queue
• reject — equivalent to nack for a single message

Never use autoAck: true in production — if the consumer crashes after receiving but before processing, the message is lost.

Queue Topology Design

Design topology before writing code. The topology is infrastructure — changing it in production requires careful migration.