running-resumable-sync-jobs

Running Resumable Sync Jobs

A sync job is a loop over N items against a service that will fail partway through. Everything hard about these jobs is what happens on item K of N: what the process reports, what it wrote down, and what the next run does with that. Get the happy path right and you still ship a job that silently under-collects, double-applies, or bills for work it never did.

The exit code is the only thing automation reads

The most common failure: per-item errors are printed as warnings, the loop continues, and the run finishes by saving state, committing, printing "Sync complete!" and returning success. A cron wrapper, a CI step, or a supervisor sees exit 0 and reports a healthy job forever.

# Bad — the warning goes to a log nobody reads; the process says "fine".
for user in users:
    try:
        mirror(user)
    except MirrorError as e:
        print(f"Warning: failed to mirror {user}: {e}")
        continue
save_state(); commit(); print("Sync complete!")
return 0

# Good — failures are counted and surfaced in the status the caller can act on.
failed = []
for user in users:
    try:
        mirror(user)
    except MirrorError as e:
        log.error("failed to mirror %s: %s", user, e)
        failed.append(user)
save_state()           # keep the work that succeeded
if failed:
    log.error("%d of %d targets failed: %s", len(failed), len(users), failed)
    return 1           # or 2, if you want "partial" distinguishable from "total"
return 0

Continuing past a failure is usually right — you want the other 99 items. What is never right is continuing and claiming success. If partial success must be acceptable to the caller, make it an explicit opt-in (--allow-partial), not the default silence.

Checkpoint what actually happened, not what you attempted

Resumable jobs record "item X is done" so the next run skips it. Two ways that record goes wrong, and both are worse than no checkpoint at all:

Attempted-but-unfinished recorded as done. State is saved for everything processed before the failure, so the next run skips those and the post-failure items stay pending forever — and because the run exited 0, nothing ever says so. This is the exit-code bug above compounding into permanent data loss.