Load Balancing & Health

The problem: one connection, many calls

Everything that makes gRPC fast also makes load balancing tricky. A gRPC client opens one long-lived HTTP/2 connection and multiplexes every call over it. That’s great for latency — but a traditional L4 (TCP) load balancer balances connections, not requests. It picks a backend once, when the connection opens, and then every call rides that single connection to that single backend.

flowchart TB
  subgraph l4["L4 (TCP) balancer — pins the connection"]
    c1["client"] -->|one connection| p["L4 LB"]
    p --> b1["backend A
(gets everything)"]
    b2["backend B
(idle)"]
    b3["backend C
(idle)"]
  end
  subgraph csl["Client-side LB — spreads calls"]
    c2["client"] --> r["resolver + policy"]
    r --> ba["backend A"]
    r --> bb["backend B"]
    r --> bc["backend C"]
  end

An L4 balancer pins all calls to one backend; client-side LB spreads them

The result: one backend gets hammered while the others sit idle, and autoscaling misfires. This is the single most common gRPC scaling surprise.

Two ways to balance gRPC

There are two correct approaches:

L7 (application-aware) proxy — a proxy that speaks HTTP/2 and gRPC (Envoy, Linkerd, or a gRPC-aware cloud LB) balances individual calls, not connections. The client just talks to the proxy.
Client-side load balancing — the client itself knows about all the backends and spreads calls across them. It uses a name resolver (to discover the set of backend addresses, e.g. via DNS or xDS) and a load-balancing policy (like round_robin) to choose a subchannel — one connection per backend — for each call.

Client-side LB is idiomatic in gRPC because it avoids an extra network hop. The classic setup: resolve dns:///my-service:50051 to several IPs and apply round_robin.

conn, _ := grpc.NewClient(
    "dns:///my-service:50051",
    grpc.WithDefaultServiceConfig(`{"loadBalancingConfig":[{"round_robin":{}}]}`),
    grpc.WithTransportCredentials(insecure.NewCredentials()),
)

channel = grpc.insecure_channel(
    "dns:///my-service:50051",
    options=[("grpc.lb_policy_name", "round_robin")],
)

const client = new MyServiceClient(
  'dns:///my-service:50051',
  credentials.createInsecure(),
  { 'grpc.service_config': JSON.stringify({
      loadBalancingConfig: [{ round_robin: {} }],
    }) },
);

Health checking

Balancing across backends is only useful if you route away from unhealthy ones. gRPC defines a standard Health Checking Protocol — a tiny service every server can expose:

syntax = "proto3";
package grpc.health.v1;

message HealthCheckRequest { string service = 1; }
message HealthCheckResponse {
  enum ServingStatus { UNKNOWN = 0; SERVING = 1; NOT_SERVING = 2; }
  ServingStatus status = 1;
}

service Health {
  rpc Check(HealthCheckRequest) returns (HealthCheckResponse);
  rpc Watch(HealthCheckRequest) returns (stream HealthCheckResponse);
}

Load balancers and orchestrators (Kubernetes, Envoy) call Check — or subscribe with Watch — and stop routing to a backend that reports NOT_SERVING. Because it’s a standard proto, tooling like grpc_health_probe works with any language’s server.

Keepalive

Long-lived connections can silently die (an idle NAT drops them, a backend disappears). Keepalive pings — small HTTP/2 PINGs on an interval — detect a dead connection quickly so the client can reconnect and re-resolve, rather than sending calls into a black hole. Tune them carefully: too aggressive and servers may reject you for pinging too often.