← Back to the library
System designDeep dive · 9 min

Every deploy causes a spike as all clients reconnect at once

“Each rollout shows a wall of new connections in the same second, auth and the database spike as every client refetches its state, and the newest instances sometimes fall over before the rollout finishes.”

ERROR CODES YOU MAY SEE

A code is a clue. Use its meaning and surrounding evidence to narrow the cause.

1001WebSocket close codes · RFC 6455 · standard

The endpoint is going away, for example a server shutting down or a browser navigating away from the page.

A spike of 1001 that lines up with a rollout means instances are closing sockets during shutdown; check whether clients then reconnect at the same moment.

Official reference for 1001 (opens in new tab)Code definition reviewed
1012WebSocket close codes · IANA registry · standard

Service Restart: the server is restarting and the client may reconnect.

Clients should treat 1012 as "reconnect after a randomized delay", not "reconnect now"; otherwise every deploy creates a synchronized reconnect wave.

Official reference for 1012 (opens in new tab)Code definition reviewed
1006WebSocket close codes · RFC 6455 · standard

The connection ended abnormally without a received Close frame. This is a local indication; endpoints must not send 1006 in a Close frame.

Check network interruptions, proxy timeouts, and process restarts before choosing a reconnect and state-recovery strategy.

Official reference for 1006 (opens in new tab)Code definition reviewed
503HTTP status codes · RFC 9110 · standard

The server is temporarily unable to handle the request due to overload or maintenance. It may send Retry-After.

Treat it as transient: honor Retry-After when present, otherwise back off with jitter and cap total attempts.

Official reference for 503 (opens in new tab)Code definition reviewed

THE PRINCIPLE

A deploy is a synchronized failure for long-lived connections; desynchronize it at both ends and make the cost per reconnect small.

FIRST MOVES

  1. Graph new connections per second and per-connect work (auth, subscriptions, snapshot queries) across a deploy to see which part of the reconnect costs most.
  2. Drain gradually: on SIGTERM, stop accepting, then close existing sockets in batches across the grace period with 1012 (Service Restart) or 1001 (Going Away), not all at once.
  3. Randomize every reconnect, including the first: add a jittered delay before attempt one, then use capped exponential backoff with jitter.
  4. Protect new instances: limit upgrades per second per instance and reject the excess cheaply (503 with Retry-After before the upgrade, or close 1013 Try Again Later).
  5. Resume instead of resyncing: send the last sequence number or a resume token and replay only the gap; fall back to a full snapshot only when the gap is too old.

TOOLS: THEN → NOW

Fixed reconnect delay or immediate retryJittered first attempt plus capped exponential backoff (Socket.IO randomizationFactor defaults to 0.5)
Full state reload on every reconnectResume from an offset: Socket.IO connection state recovery, Redis Streams, or your own sequence numbers

PATTERN SNAPSHOT

ws.addEventListener("close", (event) => {
  // 1008 (policy): refresh the token on "token_expired", sign out otherwise.
  if (event.code === 1008) return handleAuthClose(event.reason);
  // 1001/1012 mean a planned restart: spread clients over the next ~10s.
  const planned = event.code === 1001 || event.code === 1012;
  const base = planned ? 10_000 : Math.min(30_000, 1_000 * 2 ** attempt);
  const delay = Math.random() * base; // full jitter
  attempt++;
  setTimeout(() => connect({ resumeFrom: lastSeq }), delay);
});

CLOSE THE AI. EXPLAIN THIS.

If 50,000 clients use a fixed 1-second reconnect delay, what does the load look like during a rollout, and which two changes flatten it most?

HOW IT WORKS UNDERNEATH

SOURCES

Guide reviewed