What a Race Condition Really Feels Like at 3am

The pager went off at 3:14am for a job scheduler in my homelab that had run flawlessly for eight months, and the only thing that had changed was a second worker process I’d added that afternoon to clear a backlog faster. Two workers picked up the same queued job at almost the same instant, both decided it hadn’t been claimed yet, and both ran it — once wasn’t a bug, twice was data corruption. That’s a race condition in its purest, most common shape: two things checking a condition, both getting the same answer because neither has finished acting on it yet, and both proceeding as if they were the only one in the room. It’s one of the few bug classes that gets meaningfully worse the more successful your system becomes, because concurrency and race conditions are the same problem wearing different clothes.

The Gap Between Check and Act

Almost every race condition traces back to the same structural shape: check a condition, then act based on what you found, with a gap between those two steps where something else can intervene. In the job-queue example, the check was “is this job’s status still pending?” and the act was “mark it running and execute it.” Between the check and the act, there’s a window — sometimes nanoseconds, sometimes long enough to matter under load — during which another process can run the exact same check, see the exact same still-pending status, and proceed down the exact same path, because from that second process’s point of view, nothing has changed yet.

1
2
3
4
5
6

# The race:
job = db.query("SELECT * FROM jobs WHERE id = ? AND status = 'pending'", job_id)
if job:
    # <-- another worker can run the same query RIGHT HERE
    db.execute("UPDATE jobs SET status = 'running' WHERE id = ?", job_id)
    run_job(job)

This is why race conditions are so maddeningly intermittent. Under light load, one worker almost always finishes its check-then-act sequence before another worker starts the same one, so the gap never gets exploited and the bug never shows itself, sometimes for months. Under heavier load — more workers, slower database round trips, a busier CPU introducing more scheduling jitter — the odds of two processes landing in that gap simultaneously go up, and the bug starts appearing, seemingly out of nowhere, exactly when the system is under the most pressure to behave correctly. This is also why race conditions are notoriously hard to reproduce in a debugger: single-stepping through code slows everything down so much that the gap effectively disappears, and the very act of investigating the bug can make it stop happening.

Why It’s Never About the Code Looking Wrong

The code above isn’t obviously wrong on a read-through — it checks a condition, then acts on it, which is what code is supposed to do. The bug isn’t in either line individually; it’s in the fact that the two lines aren’t atomic, meaning nothing guarantees they execute as one indivisible unit with no other process able to interleave between them. This is the conceptual leap that trips people up moving from single-threaded to concurrent thinking: correctness of each line in isolation says nothing about correctness of the sequence when another actor can run its own version of the same sequence at any point in between.

The fix, in essentially every race condition of this shape, is to collapse the check and the act into one atomic operation, so there’s no gap left for another process to land in. For the job queue, that means using the database’s own atomicity rather than doing the check and the update as two separate round trips:

1
2
3
4
5
6
7

-- Atomic: the row is only updated if it's still pending,
-- and the database guarantees no other transaction can
-- see and act on the same row in between.
UPDATE jobs
SET status = 'running'
WHERE id = ? AND status = 'pending'
RETURNING *;

If this returns a row, you won the race and own the job. If it returns nothing, another worker got there first, and that’s fine — no corruption, no duplicate execution, because the database itself serialises the check-and-set into one operation no other transaction can split apart. The lesson generalises far beyond databases: mutexes, atomic compare-and-swap instructions, and message-queue visibility timeouts all exist to solve exactly this same problem in exactly this same way, collapsing a check-then-act sequence into something the underlying system guarantees is indivisible.

Why More Workers Made It Worse, Not Better

Adding a second worker didn’t create the race condition — it had been there in the code the entire time, latent, since the day it was written with a single worker in mind. What the second worker did was create an opportunity for the pre-existing gap to actually get exploited, because a single worker can never race against itself; there was only ever one process running the check-then-act sequence at any moment, so the gap existed on paper but never mattered in practice. This is a specific, common trap: code that appears to work correctly for a long time isn’t proof that it’s correct, only proof that it’s never yet been tested under the specific concurrency conditions that would expose the flaw. Scaling up — more workers, more replicas, more traffic — is exactly the kind of change that turns a theoretical race condition into an actual incident, which is why concurrency bugs disproportionately show up right after a scaling change, a deploy that added parallelism, or a traffic spike, rather than during the quiet period when the code was first written and tested.

The Security Version of the Same Bug

The check-then-act gap isn’t only a data-corruption problem; it’s a well-studied security vulnerability class in its own right, usually called TOCTOU — time-of-check to time-of-use. A classic example: a program checks whether a file exists and is safe to write to, then opens it a moment later to actually write, and in the gap between those two operations an attacker with local access swaps the target for a symlink pointing somewhere sensitive, so the program’s write lands somewhere it never intended. The shape is identical to the job-queue race — check a condition, act on it later, gap in between — except here the “other worker” racing against you is deliberately hostile rather than merely another instance of your own code.

1
2
3
4
5

# TOCTOU symlink attack, classic shape:
if os.path.exists("/tmp/safe-tempfile") and not os.path.islink("/tmp/safe-tempfile"):
    # <-- attacker swaps /tmp/safe-tempfile for a symlink to /etc/passwd RIGHT HERE
    with open("/tmp/safe-tempfile", "w") as f:
        f.write(sensitive_data)

The fix follows the same pattern as the database example: collapse the check and the use into one atomic operation the operating system itself guarantees, rather than trusting that nothing changes in the gap. open() with the O_EXCL and O_CREAT flags together, for instance, atomically fails if the file already exists, rather than checking existence first and creating it as a separate step — the kernel performs the check-and-create as one indivisible syscall, closing the gap an attacker would otherwise race into. Anywhere security-sensitive code checks a filesystem condition and then acts on it moments later, that’s worth auditing specifically for this pattern, because the underlying flaw is exploitable by exactly the same mechanism as the accidental concurrency bugs above, just with an adversary timing the race on purpose instead of two workers colliding by accident.

Tools That Find Races Before Production Does

Because races are so hard to catch by reading code or by manual testing, several languages ship dedicated tooling specifically to detect them automatically, by instrumenting memory access and flagging any case where two goroutines, threads, or processes access the same piece of shared state without a proper synchronisation point between them. Go’s race detector is the most approachable example, and running it is close to free during development:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

$ go test -race ./...
==================
WARNING: DATA RACE
Write at 0x00c000012345 by goroutine 8:
  main.claimJob()
      /app/worker.go:42

Previous read at 0x00c000012345 by goroutine 6:
  main.claimJob()
      /app/worker.go:39
==================

This kind of tool won’t catch every race — it only detects ones that actually get exercised during the test run, so it’s only as good as the concurrency your tests genuinely stress — but running it routinely in CI against a test suite that deliberately spins up multiple concurrent workers catches an enormous share of these bugs before they ever reach a production pager. C and C++ have ThreadSanitizer doing the equivalent job; most mainstream concurrent languages have something comparable, and the honest advice is to actually turn it on rather than assuming code review alone will spot a bug that’s specifically defined by not being visible in a single-threaded read-through.

Troubleshooting: Finding a Race Condition Without Losing More Sleep

Start by looking for any place in the code where a condition is checked and then acted on as two separate steps, particularly across a network or database boundary where the gap between the two is measured in milliseconds rather than nanoseconds — that gap is orders of magnitude more exploitable than an in-process gap, and it’s where I’d look first. Reproducing the bug on demand rather than waiting for it to happen at 3am again usually means deliberately increasing contention: run more concurrent workers than production normally would, add an artificial small delay between the check and the act to widen the window, and hammer the system with concurrent requests aimed at the same resource — this converts an intermittent, unreproducible bug into a reliably reproducible one, which is most of the actual debugging work. Logging alone often won’t catch it, because by the time a log line is written the race has usually already resolved one way or the other; instrumenting the actual check-then-act boundary with a counter or a deliberately widened sleep is far more effective than staring at logs after the fact hoping to spot the gap in hindsight.

Once you’ve found the vulnerable window, fix it by making the operation atomic at the lowest level that can actually guarantee it — a database UPDATE ... WHERE with the condition baked directly into the query, an application-level lock (Redis’s SETNX or a Postgres advisory lock are common, approachable choices) if the resource isn’t naturally backed by something transactional, or a proper mutex if the race is entirely in-process rather than across services. Avoid the tempting but usually inadequate quick fix of “just add a small delay before acting” — it narrows the window without closing it, which means the bug becomes rarer and correspondingly harder to catch in testing, while remaining exactly as possible in production under just the right timing.

Is It Worth Understanding This?

Yes, urgently, if you run anything with more than one worker, more than one request handler, or more than one replica touching shared state — which describes almost everything past a certain scale, including most homelab setups the moment you add a second instance of anything for redundancy. Race conditions are one of the few bug classes that get worse exactly when your infrastructure is doing well enough to need scaling, which makes them a genuinely nasty trap for anyone who assumes success means fewer problems. If concurrency bugs are new territory, what eventual consistency is actually promising you covers the distributed-systems cousin of this same problem, and the message queue you built is a table with anxiety is a good next read on exactly the kind of homegrown queue where this class of bug loves to hide.