gRPC in depth: protobuf contracts, deadlines, streaming and load balancing

HOW IT FITS TOGETHER

KEY TERMS

Protobuf schema evolution

Only field numbers and wire types travel. Adding fields and removing fields are safe; changing or reusing a number is not, and old binaries silently decode new data into the wrong field. When you delete a field, reserve its number and name (reserved 4; reserved "legacy_id";). Numbers 1-15 encode in one byte, and 19000-19999 are reserved by protobuf. Use optional (explicit presence) for scalars where unset differs from zero; message fields always have presence. The first enum value must be 0 and should be FOO_UNSPECIFIED, because unknown or unset values decode to it. Editions (2023, 2024) replace syntax = "proto3" with per-feature defaults but keep the same wire rules.

Deadlines and cancellation

A deadline is an absolute point; the wire carries the remaining duration in grpc-timeout. Servers see it as the context deadline, and passing that context to outbound calls propagates it. When the deadline passes, the client reports DEADLINE_EXCEEDED and the server's context is cancelled (CANCELLED); in-flight downstream calls are reset. Your handler must still check cancellation (ctx.Done() in Go, context.cancelled() in Python) to stop database queries and goroutines it started.

Status codes and rich errors

Every call ends with grpc-status and grpc-message in trailers. Use codes deliberately: INVALID_ARGUMENT and FAILED_PRECONDITION are not retryable, UNAVAILABLE is the canonical retryable code, RESOURCE_EXHAUSTED signals quota, and UNKNOWN or INTERNAL usually mean a bug. The richer model (google.rpc.Status with typed details such as BadRequest, ErrorInfo, RetryInfo, QuotaFailure) travels as a binary trailer; it is supported in C++, Go, Java, Python and Ruby, but proxies and loggers cannot see it and large details can exceed header limits.

Retries, hedging and keepalive

The service config declares per-method policy: retryPolicy (maxAttempts, initialBackoff, maxBackoff, backoffMultiplier, retryableStatusCodes) or hedgingPolicy (maxAttempts, hedgingDelay, nonFatalStatusCodes), never both on one method; maxAttempts is capped at 5 by default for either, plus retryThrottling (maxTokens, tokenRatio) to stop retry storms. Retries are enabled but no policy is configured by default; only transparent retries of requests that never left the client happen. Hedging sends duplicates, so use it only for idempotent methods. Keepalive pings detect dead connections; client KEEPALIVE_TIME is off by default, servers enforce a minimum interval (PERMIT_KEEPALIVE_TIME, 5 minutes by default) and send GOAWAY too_many_pings to clients that ping faster, so enabling client keepalive means lowering the server's limit too.

Load balancing long-lived connections

One HTTP/2 connection carries thousands of calls, so an L4 (TCP) load balancer balances connections, not requests: new pods get no traffic and one hot pod gets everything. Options: client-side round_robin over a Kubernetes headless Service (DNS returns pod IPs; the default pick_first does not balance), an L7 proxy that balances per stream (Envoy, Linkerd, a gRPC-aware cloud LB), or xDS, where a control plane pushes endpoints and policies to proxyless gRPC clients. Set server MaxConnectionAge so clients reconnect and rediscover new pods after scale-ups.

Browsers, reflection and tooling

Browsers cannot read HTTP/2 trailers or control frames, so they need gRPC-Web (via Envoy or a server-side handler) or the Connect protocol, which speaks gRPC, gRPC-Web and a plain JSON-over-HTTP mode from the same server; both support unary and server streaming only. Server reflection lets grpcurl or buf curl list and call services without .proto files; restrict it to internal networks. Buf lints protos (STANDARD rules) and runs buf breaking --against the main branch in CI, at FILE, PACKAGE, WIRE_JSON or WIRE strictness.

IN YOUR STACK

// proto/orders/v1/orders.proto
syntax = "proto3";
package orders.v1;

service OrderService {
  rpc GetOrder(GetOrderRequest) returns (GetOrderResponse) {
    option idempotency_level = NO_SIDE_EFFECTS; // Connect can serve it over GET
  }
  rpc WatchOrders(WatchOrdersRequest) returns (stream WatchOrdersResponse);
}

enum OrderStatus {
  ORDER_STATUS_UNSPECIFIED = 0; // unknown and unset values land here
  ORDER_STATUS_PENDING = 1;
  ORDER_STATUS_PAID = 2;
}

message Order {
  reserved 4;                  // was legacy_total; never reuse
  reserved "legacy_total";
  string id = 1;
  OrderStatus status = 2;
  optional int64 discount_cents = 3; // presence: unset differs from 0
  int64 total_cents = 5;
}

// buf.yaml:  version: v2 / lint: { use: [STANDARD] } / breaking: { use: [WIRE_JSON] }
// CI:        buf lint && buf breaking --against '.git#branch=main'

func (s *checkoutServer) Checkout(ctx context.Context, req *pb.CheckoutRequest) (*pb.CheckoutReply, error) {
	if dl, ok := ctx.Deadline(); !ok || time.Until(dl) < 100*time.Millisecond {
		return nil, status.Error(codes.DeadlineExceeded, "not enough budget to start")
	}
	if req.GetCartId() == "" {
		st, _ := status.New(codes.InvalidArgument, "cart_id is required").WithDetails(
			&errdetails.BadRequest{FieldViolations: []*errdetails.BadRequest_FieldViolation{
				{Field: "cart_id", Description: "must not be empty"}}})
		return nil, st.Err()
	}
	// Passing ctx forwards the remaining deadline and cancellation downstream.
	order, err := s.orders.CreateOrder(ctx, &pb.CreateOrderRequest{CartId: req.GetCartId()})
	if err != nil {
		return nil, err // already a gRPC status from the downstream call
	}
	select { // long local work must watch ctx too
	case res := <-s.payments.Charge(ctx, order):
		return &pb.CheckoutReply{OrderId: order.GetId(), PaymentId: res.ID}, nil
	case <-ctx.Done():
		return nil, status.FromContextError(ctx.Err()).Err()
	}
}

const serviceConfig = `{
  "loadBalancingConfig": [{ "round_robin": {} }],
  "methodConfig": [{
    "name": [{ "service": "orders.v1.OrderService", "method": "GetOrder" }],
    "timeout": "1s",
    "retryPolicy": {
      "maxAttempts": 3, "initialBackoff": "0.1s", "maxBackoff": "1s",
      "backoffMultiplier": 2, "retryableStatusCodes": ["UNAVAILABLE"]
    }
  }],
  "retryThrottling": { "maxTokens": 10, "tokenRatio": 0.1 }
}`

// Headless Service: DNS returns every pod IP, round_robin spreads calls across them.
conn, err := grpc.NewClient("dns:///orders.prod.svc.cluster.local:50051",
	grpc.WithTransportCredentials(credentials.NewTLS(tlsConfig)),
	grpc.WithDefaultServiceConfig(serviceConfig),
	grpc.WithKeepaliveParams(keepalive.ClientParameters{Time: time.Minute, Timeout: 20 * time.Second}),
)
if err != nil {
	log.Fatal(err)
}
client := pb.NewOrderServiceClient(conn)

// Server side: force periodic reconnects, and permit the client's 1-minute pings (default floor: 5 min).
// grpc.NewServer(grpc.KeepaliveParams(keepalive.ServerParameters{MaxConnectionAge: 5 * time.Minute}),
//   grpc.KeepaliveEnforcementPolicy(keepalive.EnforcementPolicy{MinTime: 30 * time.Second}))

import * as grpc from "@grpc/grpc-js";
import * as protoLoader from "@grpc/proto-loader";

const def = protoLoader.loadSync("proto/orders/v1/orders.proto", { keepCase: false, enums: String });
const { orders } = grpc.loadPackageDefinition(def) as any;

const client = new orders.v1.OrderService("dns:///orders.internal:50051", grpc.credentials.createSsl(), {
  "grpc.service_config": JSON.stringify({ loadBalancingConfig: [{ round_robin: {} }] }),
  "grpc.keepalive_time_ms": 60_000,
});

client.getOrder({ id: "42" }, { deadline: Date.now() + 1_000 }, (err: grpc.ServiceError | null, order: unknown) => {
  if (err?.code === grpc.status.DEADLINE_EXCEEDED) return console.warn("orders slow");
  if (err) return console.error(err.code, err.details);
  console.log(order);
});

const stream = client.watchOrders({ customerId: "c1" }, { deadline: Infinity });
stream.on("data", (event: unknown) => console.log(event));
stream.on("error", (err: grpc.ServiceError) => {
  if (err.code !== grpc.status.CANCELLED) scheduleResubscribe(); // resume from last seen id
});
// later: stream.cancel();

import asyncio
import grpc
from google.protobuf import any_pb2
from google.rpc import code_pb2, error_details_pb2, status_pb2
from grpc_status import rpc_status
import inventory_pb2, inventory_pb2_grpc

class Inventory(inventory_pb2_grpc.InventoryServicer):
    async def Reserve(self, request, context: grpc.aio.ServicerContext):
        if request.quantity <= 0:
            detail = any_pb2.Any()
            detail.Pack(error_details_pb2.BadRequest(field_violations=[
                error_details_pb2.BadRequest.FieldViolation(field="quantity", description="must be > 0")]))
            await context.abort_with_status(rpc_status.to_status(status_pb2.Status(
                code=code_pb2.INVALID_ARGUMENT, message="invalid quantity", details=[detail])))
        remaining = context.time_remaining() or 5.0
        try:  # bound our own work by the caller's budget
            await asyncio.wait_for(reserve_rows(request), timeout=max(remaining - 0.05, 0))
        except asyncio.TimeoutError:
            await context.abort(grpc.StatusCode.DEADLINE_EXCEEDED, "reservation timed out")
        return inventory_pb2.ReserveReply(ok=True)

async def serve():
    server = grpc.aio.server(options=[("grpc.max_connection_age_ms", 300_000)])
    inventory_pb2_grpc.add_InventoryServicer_to_server(Inventory(), server)
    server.add_insecure_port("[::]:50051")
    await server.start()
    await server.wait_for_termination()

import { createClient, ConnectError, Code } from "@connectrpc/connect";
import { createConnectTransport } from "@connectrpc/connect-web";
import { OrderService } from "./gen/orders/v1/orders_pb";

// Connect protocol over fetch: JSON or binary, HTTP/1.1 or HTTP/2, no proxy needed.
const transport = createConnectTransport({ baseUrl: "https://api.example.com", useHttpGet: true });
const client = createClient(OrderService, transport);

export async function loadOrder(id: string, signal: AbortSignal) {
  try {
    return await client.getOrder({ id }, { timeoutMs: 2_000, signal });
  } catch (err) {
    if (err instanceof ConnectError && err.code === Code.Canceled) return null; // user navigated away
    if (err instanceof ConnectError && err.code === Code.DeadlineExceeded) throw new Error("Order service is slow");
    throw err;
  }
}

export async function watch(customerId: string, onEvent: (e: unknown) => void, signal: AbortSignal) {
  for await (const event of client.watchOrders({ customerId }, { signal })) onEvent(event); // server streaming works
}

# Server must register reflection (Go: reflection.Register(s)); keep it internal-only.
grpcurl orders.internal:443 list
grpcurl orders.internal:443 describe orders.v1.Order

# Call with metadata and a deadline; output is the JSON form of the response.
grpcurl -H "authorization: Bearer $TOKEN" -max-time 2 \
  -d '{"id": "42"}' orders.internal:443 orders.v1.OrderService/GetOrder

# No reflection? Point at the protos instead.
grpcurl -import-path proto -proto orders/v1/orders.proto \
  -d '{"customer_id": "c1"}' orders.internal:443 orders.v1.OrderService/WatchOrders

# Same with Buf (defaults to the Connect protocol; also grpcweb).
buf curl --protocol grpc --schema proto --data '{"id":"42"}' \
  https://orders.internal/orders.v1.OrderService/GetOrder

# Health checks use the standard grpc.health.v1 service.
grpcurl orders.internal:443 grpc.health.v1.Health/Check

WHEN TO REACH FOR WHAT

WHERE IT BITES

• No deadline on client calls. gRPC defaults to none, so a stuck dependency holds threads, connections and memory until something else fails. Set a deadline on every call and propagate the incoming context.
• Reusing or renumbering protobuf fields, or changing a field's type. Old binaries decode garbage without errors. Reserve deleted numbers and names, and run buf breaking in CI.
• Putting gRPC behind an L4 load balancer or a ClusterIP Service and wondering why one pod is at 100% CPU. Balance per call with round_robin over a headless Service, an L7 proxy, or xDS, and cap connection age.
• Retrying or hedging non-idempotent methods, or adding retries at the client, the mesh and the gateway at once. Retry only UNAVAILABLE-like codes, enable retryThrottling, and keep retries in one layer.
• Mapping every error to UNKNOWN or INTERNAL. Clients cannot decide whether to retry, fix input or give up; choose codes deliberately and attach google.rpc details for field errors and retry hints.
• Keepalive settings that disagree. Clients pinging more often than the server permits (5 minutes by default) get GOAWAY too_many_pings and reconnect loops; when you enable client keepalive, lower the server's enforcement minimum to match (Go EnforcementPolicy.MinTime), and avoid intervals much below one minute.

WHEN IT BREAKS IN PRODUCTION

• Scaling out a gRPC service doesn't spread the load
• After a proto change, old clients read the wrong values
• Requests fail with 504 while the server keeps working after the client gave up
• One request waits on a chain of services and times out
• A brief blip turns into a long outage
• A backend deploy breaks old app versions
• You can't follow one request across services
• Every deploy causes a burst of 502s

SOURCES

• gRPC: Deadlines (opens in new tab)Checked 2026-11-06
• gRPC: Retry (opens in new tab)Checked 2026-11-06
• gRPC: Error handling (opens in new tab)Checked 2026-11-06
• gRPC: Keepalive (opens in new tab)Checked 2026-11-06
• Protocol Buffers: Language Guide (proto3) (opens in new tab)Checked 2026-11-06
• Buf: Breaking change detection (opens in new tab)Checked 2026-11-06