loom-background-jobs

Background Jobs

Overview

Reliable async task execution: enqueue work, process it in workers decoupled from the request cycle, and survive crashes/retries without corrupting state. This file covers job queues, retries/backoff, DLQs, scheduling, worker pools, and delivery guarantees.

For pub/sub, event sourcing, CQRS, sagas, and streaming brokers (Kafka/Pulsar), see loom-event-driven — don't reimplement those here.

The two invariants everything hangs off

1. At-least-once is the default. Design every handler to be idempotent. Redis-backed queues (Sidekiq, BullMQ, Celery+Redis), SQS standard, and a queue retrying after a crash can deliver work more than once. FIFO producer deduplication does not make a consumer's external side effect exactly once. The achievable goal is exactly-once effect = durable idempotency at the sink plus retry-safe handling.
2. Ack after success, never before. The job must stay owned by the worker until the side effect is durably committed. Ack-then-process = at-most-once = silent data loss on crash. Process-then-ack = at-least-once = duplicates you dedup away. Always choose the latter.

Idempotency: the non-negotiable pattern

Derive a stable key from the job's business identity (not a random UUID per enqueue), and make the durable sink reject duplicates. A separate “done” flag cannot atomically cover an external side effect and a crash.

def handle(job):
    key = job["idempotency_key"]          # e.g. f"receipt:{order_id}"
    # One transaction: UNIQUE(key) makes duplicate deliveries a successful no-op.
    with db.transaction():
        inserted = db.execute(
            "INSERT INTO receipts (idempotency_key, order_id) VALUES (?, ?) "
            "ON CONFLICT (idempotency_key) DO NOTHING",
            [key, job["order_id"]],
        ).rowcount == 1
        if inserted:
            db.execute(
                "INSERT INTO outbox (idempotency_key, kind, aggregate_id) "
                "VALUES (?, 'send-receipt', ?)",
                [key, job["order_id"]],
            )
    # The outbox makes the intent durable; the email sender must deduplicate by key too.