← Back to the library
System designDeep dive · 9 min

Events from different machines appear out of order

“A reply is logged before the message it answers, a duration comes out negative, or a last-write-wins update silently loses to an older one written on a machine whose clock runs ahead.”

THE PRINCIPLE

Wall-clock time tells people when; it cannot tell machines which came first. Order with counters that only move forward.

FIRST MOVES

  1. Measure durations and deadlines with a monotonic clock (CLOCK_MONOTONIC, performance.now(), process.hrtime.bigint()), never by subtracting Date.now() values.
  2. Where one writer owns the data, order by a sequence it assigns: a database sequence, a per-entity version column, or a Kafka partition offset.
  3. For concurrent writers, replace last-write-wins on wall time with a version check (compare-and-set) or a hybrid logical clock that never runs backwards.
  4. Carry the sender's HLC or version on every message so receivers order causally, and break ties deterministically (e.g. by node id); reject remote clocks implausibly far ahead, or one bad host drags everyone forward.
  5. Alert on clock offset from your time source (chrony/NTP tracking, cloud time sync); if you rely on bounded skew, read the bound from ClockBound or similar instead of assuming it.

TOOLS: THEN → NOW

Date.now() / System.currentTimeMillis() for ordering and timingMonotonic clocks for timing; sequences, versions or HLCs for ordering
Assume NTP keeps servers in syncBounded-uncertainty clocks (Spanner TrueTime, AWS ClockBound) when ordering by time is unavoidable

PATTERN SNAPSHOT

// Hybrid logical clock: tracks wall time but never goes backwards, and a
// receive always sorts after the send. Compare (wall, logical), then node id.
type Hlc = { wall: number; logical: number };
let last: Hlc = { wall: 0, logical: 0 };

export function tick(remote?: Hlc): Hlc {
  const wall = Math.max(Date.now(), last.wall, remote?.wall ?? 0);
  let logical = 0;
  if (remote && wall === last.wall && wall === remote.wall) {
    logical = Math.max(last.logical, remote.logical) + 1;
  } else if (wall === last.wall) logical = last.logical + 1;
  else if (remote && wall === remote.wall) logical = remote.logical + 1;
  return (last = { wall, logical });
}

CLOSE THE AI. EXPLAIN THIS.

Two servers write the same key 5 ms apart, and the second server's clock is 50 ms behind. Under last-write-wins on wall time, which value survives, and what would you use instead?

SOURCES

Guide reviewed