← Back to the library
System designDeep dive · 9 min

Scaling out a gRPC service doesn't spread the load

“We went from 3 to 10 pods, but one pod still runs at 95% CPU and throws DEADLINE_EXCEEDED while the new pods sit almost idle. Restarting the clients evens it out for a while.”

ERROR CODES YOU MAY SEE

A code is a clue. Use its meaning and surrounding evidence to narrow the cause.

DEADLINE_EXCEEDEDgRPC status codes · standard

The deadline expired before the operation could complete (status 4). For state-changing operations it may be returned even if the operation completed.

Check whether the deadline was spent waiting in queues or on retries downstream. Propagate the caller's remaining deadline instead of starting a fresh timeout at each hop, and only retry idempotent calls.

Official reference for DEADLINE_EXCEEDED (opens in new tab)Code definition reviewed

THE PRINCIPLE

L4 balancers spread connections, not requests. With HTTP/2, balance where the individual calls are visible.

FIRST MOVES

  1. Compare per-pod RPC rate with per-pod connection count. A skewed RPC rate with few long-lived connections confirms connection-level balancing.
  2. For internal callers, use a headless Service (clusterIP: None) so DNS returns every pod IP, and set round_robin in the client's service config. The default pick_first sends all calls to one address.
  3. Or put an HTTP/2-aware L7 proxy in the path (Envoy, a service mesh such as Linkerd or Istio, a gRPC-aware ingress) so it balances each RPC.
  4. Set MAX_CONNECTION_AGE (with a grace period) on servers so clients reconnect periodically and re-resolve. This lets them find new pods even behind L4.
  5. After a rollout, check that new pods get traffic within minutes and that client keepalive settings match what the server allows.

TOOLS: THEN → NOW

ClusterIP Service in front of gRPCHeadless Service + client-side round_robin, or an L7 proxy/mesh that balances per request
Static DNS resolved once at startupxDS-based control planes (Envoy, proxyless gRPC xDS) that push endpoint updates

PATTERN SNAPSHOT

import * as grpc from "@grpc/grpc-js";

// Client: headless Service DNS returns every pod; spread RPCs across them.
const orders = new OrdersClient(
  "dns:///orders.prod.svc.cluster.local:50051",
  grpc.credentials.createInsecure(), // in-cluster plaintext; use TLS creds otherwise
  { "grpc.service_config": JSON.stringify({ loadBalancingConfig: [{ round_robin: {} }] }) },
);

// Server: recycle connections so clients reconnect and discover new pods.
const server = new grpc.Server({
  "grpc.max_connection_age_ms": 5 * 60_000,
  "grpc.max_connection_age_grace_ms": 30_000,
});

CLOSE THE AI. EXPLAIN THIS.

Why does adding pods behind a ClusterIP Service help an HTTP/1.1 service but not a gRPC one, and which two places can balance per RPC instead?

HOW IT WORKS UNDERNEATH

SOURCES

Guide reviewed