← Back to the library
System designApplied · 9 min

Requests fail with 504 while the server keeps working after the client gave up

“Slow requests return 504 at almost exactly 60 seconds, a few return 502, and the database still shows the query running a minute after the user closed the tab. Some actions run twice because the client retried while the first attempt was still going.”

ERROR CODES YOU MAY SEE

A code is a clue. Use its meaning and surrounding evidence to narrow the cause.

504HTTP status codes · RFC 9110 · standard

A gateway or proxy did not receive a timely response from the upstream server it needed to complete the request.

Compare the proxy's timeout with the upstream's real latency. The upstream may still finish the work after the client saw 504, so only retry idempotent requests.

Official reference for 504 (opens in new tab)Code definition reviewed
502HTTP status codes · RFC 9110 · standard

A gateway or proxy received an invalid response from the upstream server it contacted, including a connection that was reset or closed mid-request.

If 502s cluster around deploys, check whether instances exit before the load balancer stops routing to them, and whether the app's keep-alive timeout is shorter than the proxy's idle timeout.

Official reference for 502 (opens in new tab)Code definition reviewed
524Cloudflare 5xx errors · platform

Cloudflare connected to the origin, but the origin did not send an HTTP response before the Proxy Read Timeout (125 seconds by default).

The origin usually keeps working after the 524. Move long work to a background job with a status resource; only Enterprise zones can raise the timeout.

Official reference for 524 (opens in new tab)Code definition reviewed
DEADLINE_EXCEEDEDgRPC status codes · standard

The deadline expired before the operation could complete (status 4). For state-changing operations it may be returned even if the operation completed.

Check whether the deadline was spent waiting in queues or on retries downstream. Propagate the caller's remaining deadline instead of starting a fresh timeout at each hop, and only retry idempotent calls.

Official reference for DEADLINE_EXCEEDED (opens in new tab)Code definition reviewed
57014PostgreSQL SQLSTATE · platform

The query was canceled (query_canceled).

Read the accompanying message to identify the cancellation cause. Check timeouts and query traces; cancellation alone does not establish N+1 access.

Official reference for 57014 (opens in new tab)Code definition reviewed

THE PRINCIPLE

Timeouts must decrease from the outside in, and work should stop when nobody is waiting for its result.

FIRST MOVES

  1. Write down every timeout on the path with its kind: total request, idle (time between bytes), connect, keep-alive. The status code and its source tell you which fired: 504 from nginx or the ALB, 524 from Cloudflare, DEADLINE_EXCEEDED from gRPC.
  2. Order them so each hop gives up after the one inside it: client > CDN/gateway > proxy > service handler > downstream calls > DB statement_timeout. Idle timeouts (ALB defaults to 60 s, nginx proxy_read_timeout to 60 s between reads) cut silent long requests regardless of total budgets.
  3. For keep-alive, invert the rule: the app's keep-alive timeout must exceed the load balancer's idle timeout, or the LB reuses a connection the app just closed and returns 502.
  4. Propagate one deadline instead of fresh per-hop timeouts: gRPC deadlines, or pass the remaining budget downstream with an AbortSignal and a header.
  5. Cancel on disconnect: Go's request context is canceled when the client connection closes; in Node use req.signal (added in v26.1, but before v26.7 it could abort as soon as the request body had been read) or listen for the response 'close' event, and pass the signal to fetch and to drivers that support cancellation.

TOOLS: THEN → NOW

Round-number timeouts set per team (30 s here, 60 s there)A documented timeout ladder per route, checked in config review
Independent timeouts at each hopPropagated deadlines (gRPC deadlines, context.Context, AbortSignal.any)
Work continues after the client leavesCancellation wired to client disconnect, plus statement_timeout as the floor

PATTERN SNAPSHOT

// Ladder: client 30 s > gateway 29 s > this handler 25 s > downstream gets what's left.
http.createServer(async (req, res) => {
  const gone = new AbortController(); // on Node 26.7+ req.signal does this
  res.on("close", () => { if (!res.writableFinished) gone.abort(); });
  const deadline = Date.now() + 25_000;
  const signal = AbortSignal.any([gone.signal, AbortSignal.timeout(25_000)]);
  try {
    const upstream = await fetch(PRICING_URL, {
      signal, headers: { "x-timeout-ms": String(deadline - Date.now() - 100) },
    });
    res.end(await upstream.text());
  } catch {
    if (!gone.signal.aborted && !res.headersSent) { res.statusCode = signal.aborted ? 504 : 502; res.end(); }
  }
});

CLOSE THE AI. EXPLAIN THIS.

Your ALB idle timeout is 60 s, nginx proxy_read_timeout is 60 s, the Node handler has no timeout and the query takes 90 s. What does the user see, what happens to the query, and which numbers do you change?

HOW IT WORKS UNDERNEATH

SOURCES

Guide reviewed