scheduled-jobs

Add a scheduled job

Scheduled work fails quietly: the tick never fires, two instances fire at once, the job runs in the wrong zone, or a lock "expires" while its holder is merely paused and two runs write at once. Gaffer's own builds shipped exactly that last defect: a time-only stale-lock lease lost an acknowledged write when the holder was paused past the threshold. Make the schedule explicit, the run idempotent, mutual exclusion enforced at write time, and absence of a run alertable. Sources: Martin Kleppmann, "How to do distributed locking" (fencing tokens); PostgreSQL docs on advisory locks and SKIP LOCKED; Kubernetes CronJob docs ("jobs should be idempotent"); Google SRE Workbook (alerting on absence).

Steps

1. Use the repo's scheduler. Cron library, platform scheduler (Kubernetes CronJob: set concurrencyPolicy: Forbid and startingDeadlineSeconds), a jobs table with next_run_at, or the queue's delayed jobs. Call search_lore for the convention. Assume any scheduler can fire twice or not at all; the steps below make both harmless.
2. Write the schedule in UTC unless the meaning is local. A "9am local report" uses a named IANA zone, never a fixed offset, and has tests for both DST transitions (spring forward: 02:30 does not exist; fall back: 01:30 happens twice). Avoid :00 when the exact minute does not matter; jitter by a few minutes.
3. Make each run idempotent and catch-up-safe. Process "everything due up to now" from a persisted watermark or per-item state (status, processed_at), not "the last interval" computed from wall-clock arithmetic, so a missed tick is recovered by the next and a double tick does nothing extra. Advance the watermark in the same transaction as the work it covers. Handler shape: the background-jobs skill.
4. Enforce mutual exclusion at write time, not only at start. A lease that expires by time alone is not exclusion: GC pauses, VM suspension or a stalled event loop can outlast any TTL, and the paused holder then writes alongside the new one. Pick one:
   • Fenced lease: acquiring increments a token, UPDATE job_lease SET owner=:me, lease_until=:now+:ttl, fence=fence+1 WHERE name=:job AND lease_until < :now RETURNING fence, with :now from the injected clock (or now() with a tiny TTL in tests); every write the run makes is conditional on that fence still being current, checked with a row-locking statement in the same transaction as the write (UPDATE job_lease … WHERE name=:job AND fence=:fence or SELECT … FOR UPDATE; zero rows → abort and roll back). A plain SELECT of the fence is check-then-act and does not fence. Renewal checks the fence too; if renewal fails, stop work immediately.
   • Connection-bound lock: pg_try_advisory_xact_lock(key) held by the transaction doing the work, or a session advisory lock on a dedicated connection (not through a transaction-mode pooler) that also carries every write the run makes. Postgres releases it when that connection or transaction ends (a long pause can end it via idle timeouts), taking the uncommitted work with it; writes on any other connection or to files are unguarded.
   • Per-item claims: workers claim rows with SELECT … FOR UPDATE SKIP LOCKED and commit each item's result in the claiming transaction.
5. Bound the run. A timeout shorter than the interval, a maximum batch per run, and a rule for backlog larger than one run (loop to a cap, then stop and report). A nightly job that takes 25 hours overlaps itself forever.
6. Record every run. Start, end, items processed, outcome, fence/owner, next run, in structured logs (the structured-logging-and-tracing skill) and a job_runs table or a last_success_timestamp metric.
7. Alert on absence. The failure mode of a cron is silence. Configure (or document for the operator) an alert when now - last_success > interval + slack (a dead man's switch), plus one on consecutive failures.