background-jobs

Background Jobs

Purpose

Move work out of the request path without losing it. A job system is a distributed system with a friendly interface; the failure modes are the same, and they are hidden until they are not.

When to Use

• Work that is slow, retriable, or must survive a request timeout: emails, exports, webhooks, media processing.
• Scheduled and recurring work.
• Diagnosing duplicate side effects, stuck queues, or jobs that silently disappear.

Capabilities

• Queue and worker design, including priority and isolation.
• Idempotency and exactly-once effects.
• Retry policy: backoff, jitter, attempt limits, dead-letter handling.
• Scheduling: cron jobs, delayed jobs, and deduplicating them across instances.
• Observability: queue depth, age, failure rate.

Inputs

• The work to be done and whether it can safely run twice.
• Latency expectations: seconds, minutes, or overnight.
• Volume, and how bursty it is.

Outputs

• Job handlers that are idempotent by construction.
• Explicit retry and DLQ policies per job type.
• Alerts on queue depth and oldest-message age.

Workflow

1. Make it idempotent first — Before anything else. A job will run twice; design so that this is harmless. Key the side effect, not the invocation.
2. Pass identifiers, not objects — Enqueue {"order_id": "..."}, not a serialized order. By the time the job runs, the object may be stale.
3. Set the retry policy per job — A transient network failure deserves exponential backoff and ten attempts. A validation failure deserves zero retries and a DLQ.
4. Isolate the queues — A flood of low-priority image resizes must not delay password-reset emails. Separate queues, separate workers.
5. Deduplicate scheduled work — With multiple instances, every instance's scheduler fires. Use a lock keyed on the job name and window.
6. Alert on age, not just depth — A queue with 10 messages that are 3 hours old is a worse signal than a queue with 10,000 that are 5 seconds old.