background-jobs

Add a background job

A background job is a promise the request made and a worker must keep, later, maybe on another machine, maybe twice. Queues deliver at least once: a worker that crashes, times out or pauses past its visibility window gets its job handed to another worker while it may still finish. Design for redelivery, overlap and partial failure first.

Steps

1. Use the repo's job system. A queue library (BullMQ, Celery, Sidekiq, River, Hangfire, Quartz), a database-backed queue or outbox, or the platform queue (SQS, Pub/Sub). Call search_lore for it and its retry defaults; never add a second system for one job. Time-driven work belongs to the scheduled-jobs skill.
2. Keep the payload small and versioned. Identifiers (orderId) and a schema version, never whole objects that will be stale when the job runs. Validate the payload at the worker like any untrusted input; an invalid payload is a permanent failure.
3. Enqueue atomically with the state change. Write the row and the job in one transaction (a jobs/outbox table in the same database, drained by a relay), or enqueue only after commit. Enqueue-before-commit creates jobs for rows that rolled back; enqueue-after-commit without an outbox loses the job if the process dies in between. An outbox relay either runs as one instance or claims rows with FOR UPDATE SKIP LOCKED.
4. Make the handler idempotent. Redelivery is normal. Key the effect on the job id or a natural key: a unique constraint on the result row, a processed-jobs table written in the same transaction as the effect, or a conditional update (… WHERE status = 'pending'). External side effects (email, webhook, charge) pass an idempotency key or go through a deduplicating ledger (the idempotency-and-retries skill).
5. Fence overlapping workers. If the job holds a lease (visibility timeout, locked_until, a lock row), a paused or slow worker can outlive it while a second worker runs the same job. Heartbeat to extend the lease while working, and make the final write conditional on still owning it, in the same transaction as the job's database effects (UPDATE jobs SET status='done' WHERE id = ? AND lease_token = ?; 0 rows → roll back); a worker whose lease was lost must not commit. Ack or delete the message only after the work commits (acks_late-style), never on receipt. See the concurrency-and-async skill for fencing tokens.
6. Classify failures and bound retries. Transient (timeouts, connection resets, 502/503/504, 429 honouring Retry-After, deadlocks and serialization failures): retry with exponential backoff, jitter and a maximum attempt count. Permanent (validation, other 4xx, missing row): fail immediately. After the last attempt, move to a dead-letter queue or a failed status with the error, visible and requeueable (the error-handling skill).
7. Set limits deliberately. Per-job timeout shorter than the visibility window or lease; worker concurrency sized to the downstream's capacity; a poison job isolated so it cannot block the queue; graceful shutdown stops taking jobs and finishes or releases in-flight ones before the deadline.
8. Instrument it. Log job enqueued/started/succeeded/failed/dead-lettered with the job id, attempt number and the correlation id from the originating request (the structured-logging-and-tracing skill); expose queue depth, age of oldest job, and duration metrics.
9. Test the lifecycle, one test per AC exercising the AC's own behaviour, plus:
   • the job is enqueued with the state change and absent when the transaction rolls back;
   • the handler run twice with the same job produces one effect;
   • two workers claiming concurrently process it once;
   • a worker paused past its lease cannot commit after resuming, and a worker killed mid-job has its job retried and completed once (the paused- and crashed-holder tests in the concurrency-and-async skill, step 9);
   • a transient failure retries then succeeds; a permanent failure dead-letters without retrying.