scheduled-jobs

Scheduled Jobs

A scheduled job is unattended by design — nobody watches it fire, which means nobody notices when it stops firing, fires twice, or fires at the wrong time until the downstream effect shows up days later as a data gap or a stale report. Every property that makes cron convenient (fire-and-forget) is also what makes it dangerous without deliberate guardrails.

Design a scheduled job assuming nobody is watching it run — because nobody is, until it breaks.

For cron syntax, DST pitfalls, locking, and Kubernetes CronJob settings, read references/cron-patterns.md.

1. Make the job idempotent, not just retry-friendly

A scheduler that fires a job that's already running, or re-fires after a timeout that wasn't actually a hang, needs the job itself to tolerate a duplicate execution. This is the same idempotency discipline from scripting-automation, but scheduled jobs need it more: nobody is present to notice a duplicate run happened.

• Key the job's effect on the scheduled time or a run ID, not on "whatever the current time is when it happens to execute" — a job that computes "yesterday" from now() breaks if it runs late.
• Make writes upserts, not appends — a job that inserts a daily summary row should overwrite today's row if it runs twice, not create two.
• Verify against the last successful run's output, not just "did the command exit zero" — exit zero with half the work done is a false success.

Done when: manually re-triggering the job for a time slot that already ran produces the same result as the original run, not a duplicate or a corruption.

2. Prevent overlapping runs explicitly

A job that takes longer than its interval will eventually have two instances running at once — both writing, both reading, both assuming they're the only one. This is one of the most common causes of scheduled-job data corruption, and it's entirely preventable with an explicit lock.

• Use a lock with a TTL — a file lock, a database row, a distributed lock — so a second scheduled fire while one is still running skips or queues instead of racing.
• Set the TTL longer than the worst-case run time, or the lock expires mid-run and a second instance starts anyway.
• Log and alert on a skipped-due-to-overlap event — it's not a failure, but a pattern of skips means the job's interval no longer fits its actual runtime.