concurrency-and-async

Write concurrent code that is correct under interleaving

Concurrency bugs pass every sequential test. Live builds of the same app twice shipped the same class past review: fixed temp-file names colliding under parallel requests (HTTP 500), unsynchronised read-modify-write losing updates, and a time-only stale-lock lease that dropped an acknowledged write when the holder was paused past the threshold. Assume that any code with a shared file, row, counter or lock has one of these until a test that forces the schedule proves otherwise.

Steps

1. Inventory shared state and writers. For every file, row, counter, cache entry or lock file the change touches, list who can write it: concurrent requests in one async process (every await is a yield point), threads, other worker processes or instances, and a restarted process after a crash. Call search_lore for the repo's lock helper, worker model and transaction rules. Each item must end up immutable, single-owner (one queue/actor/goroutine does all writes), or guarded.
2. Pick the guard by scope. In one process: a mutex or per-key async lock, or a single-writer queue. Across processes on one host: an OS lock the kernel releases on death (flock/fcntl, LockFileEx). Across hosts or on database rows: the database (UPDATE … SET n = n + 1, SELECT … FOR UPDATE, a unique constraint, a version column, pg_advisory_xact_lock). An in-memory flag guards nothing once there are two processes.
3. Serialise every read-modify-write. Read → compute → write with no guard loses updates. Use one atomic statement; or compare-and-swap (UPDATE t SET v = v + 1, data = ? WHERE id = ? AND v = ?, check the affected row count, retry a bounded number of times, then 409); or hold one lock across the whole read, modify and write. A lock around only the write does not help. Check-then-act across an await (if (!exists) await create()) is the same bug.
4. Write files atomically. Create the temp file in the same directory with a unique name (mkstemp, os.CreateTemp, O_CREAT|O_EXCL, or pid + random suffix — never a fixed data.json.tmp or a timestamp), write, fsync it when the data must survive power loss, rename over the target (os.replace in Python), then fsync the directory for a durable rename; delete the temp on failure. Use a library such as write-file-atomic (Node) or renameio (Go) only if it is already a dependency; never install one. Rename gives readers old-or-new, never a torn file — it does not serialise writers, so step 3 still applies. Acknowledge (2xx, return success) only after the rename, and after the fsyncs when durability is the contract.
5. Make cross-process locks safe against pauses. A process can stop for seconds without dying: GC, SIGSTOP, VM suspend, swap, a debugger. A lock declared stale because of age alone is then stolen while its holder still believes it owns it; the holder resumes and overwrites the new owner's acknowledged write. In order of preference:
   • a kernel-held lock (flock/fcntl) released when the owner process dies and never because time passed; or a Postgres advisory lock taken on the same connection or transaction that performs every guarded write (pg_advisory_xact_lock). Postgres releases it whenever the session ends (idle timeout, dropped connection, pooler), so it protects only writes made through that session — never a file or another store;
   • an owner-liveness check: the lock records pid, host and process start time or a nonce, and is broken only when that owner is provably gone — same host, and either kill(pid, 0) fails with ESRCH or the process now running under that pid has a different start time (pid reuse). EPERM only means some process holds that pid: compare its start time, and never treat EPERM alone as gone. Age may trigger the check, never decide it;
   • a fencing token when a time lease is unavoidable (distributed lock, job lease): each grant returns a monotonically increasing number, every write carries it, and the resource rejects a token lower than the highest it has accepted. "Re-check I still hold the lock, then write" is not a substitute (unless the check takes a row lock held until the write commits, as in the scheduled-jobs skill); the pause can land between check and write.