← Back to the library
ArchitectureDeep dive · 8 min

One request waits on a chain of services and times out

“Loading a page calls service A, which calls B, which calls C and D. Latency is the sum of all of them, p99 is terrible, and when any one of them is slow or down the whole request fails.”

ERROR CODES YOU MAY SEE

A code is a clue. Use its meaning and surrounding evidence to narrow the cause.

DEADLINE_EXCEEDEDgRPC status codes · standard

The deadline expired before the operation could complete (status 4). For state-changing operations it may be returned even if the operation completed.

Check whether the deadline was spent waiting in queues or on retries downstream. Propagate the caller's remaining deadline instead of starting a fresh timeout at each hop, and only retry idempotent calls.

Official reference for DEADLINE_EXCEEDED (opens in new tab)Code definition reviewed
503HTTP status codes · RFC 9110 · standard

The server is temporarily unable to handle the request due to overload or maintenance. It may send Retry-After.

Treat it as transient: honor Retry-After when present, otherwise back off with jitter and cap total attempts.

Official reference for 503 (opens in new tab)Code definition reviewed

THE PRINCIPLE

Every synchronous hop is a dependency on another service's latency and availability. Keep the critical path short, and make everything else optional or asynchronous.

FIRST MOVES

  1. Trace one slow request end to end and draw the call tree: depth, fan-out, which calls are serial, which repeat. Two services that are always called together may belong in one.
  2. Do the arithmetic: serial latencies add, and availability multiplies (five 99.9% dependencies in series give roughly 99.5%).
  3. Set one deadline at the edge and propagate the remaining budget downstream (some gRPC implementations, such as Java and Go, propagate deadlines automatically; with fetch, pass an AbortSignal); don't give each hop its own independent timeout.
  4. Aggregate close to the services (gateway aggregation or a BFF): fan out in parallel, return partial results for optional data, fail only on required data.
  5. Move side effects the caller doesn't need to wait for (emails, analytics, projections) to messages, and serve read-heavy data from a local copy fed by events instead of a live call.

TOOLS: THEN → NOW

Client calls every service itselfGateway aggregation or a backend-for-frontend per client
Fixed timeout per hopPropagated deadlines (gRPC deadlines, AbortSignal) plus circuit breakers
Synchronous call for every side effectEvents and queues for work that can finish after the response

PATTERN SNAPSHOT

// Aggregator: one budget for the request; optional parts degrade instead of failing.
async function orderPage(id: string, signal: AbortSignal) {
  const budget = AbortSignal.any([signal, AbortSignal.timeout(800)]);
  const [order, shipment, recs] = await Promise.allSettled([
    orders.get(id, { signal: budget }),          // required
    shipping.get(id, { signal: budget }),        // optional
    recommendations.for(id, { signal: budget }), // optional
  ]);
  if (order.status === "rejected") throw order.reason;
  return {
    order: order.value,
    shipment: shipment.status === "fulfilled" ? shipment.value : null,
    recs: recs.status === "fulfilled" ? recs.value : [],
  };
}

CLOSE THE AI. EXPLAIN THIS.

If a request passes serially through four services each at 99.9% availability and 50 ms p50, what are the rough end-to-end availability and p50, and which two changes help most?

HOW IT WORKS UNDERNEATH

SOURCES

Guide reviewed