gRPC and protobuf: the compiled contract, HTTP/2 transport, and the balancer that ignored your new pods

The contract compiles in both directions

By the end of this section you will know exactly why renaming a field is free and changing its tag is not — and why that distinction determines what your load balancer sees.

REST contracts live in documentation and drift; a gRPC contract is a file that compiles into both sides of the wire:

syntax = "proto3";
package inventory.v1;

service Inventory {
  rpc GetStock(GetStockRequest) returns (StockReply);
  rpc WatchStock(WatchRequest) returns (stream StockReply); // server streaming
}

message GetStockRequest {
  string sku = 1;       // tag 1 — the number IS the field's identity
  string warehouse = 2;
}

Run buf generate (or raw protoc with the Go plugins) and you get two artifacts: a typed client stub whose methods serialize, dispatch, and decode for you, and a server interface — InventoryServer — that your code must implement. The compiler, not a wiki page, enforces that both teams agree on the shape.

The detail that changes how you think about protobuf: field names never reach the wire. Each encoded field is a key — the tag number shifted left three bits, OR-ed with a wire type — followed by the value. Rename sku to stock_unit and the bytes are identical; change the tag from 1 to 9 and you have silently created a different field. The key is a varint, which is why tag values matter: tags 1 through 15 fit the key in a single byte, 16 through 2047 take two. On a message logged millions of times per minute, giving a hot repeated field tag 16 instead of tag 3 is a measurable storage tax. Plan tag space like you plan an API: small tags for hot fields, and leave gaps for the future.

Honest numbers: protobuf payloads typically come out 2–5× smaller than the equivalent JSON and parse roughly 5–10× faster, because decoding is arithmetic on tagged binary instead of text scanning. But gzip eats much of the size advantage on large payloads — compressed JSON often lands within 1.5–2× of compressed protobuf. The wins that survive compression are parse cost, allocation behavior, and the generated types: nobody writes json.RawMessage archaeology against a gRPC API.

One connection, multiplexed — and what your balancer sees

gRPC runs on HTTP/2: the client opens one TCP connection and every RPC becomes a stream on it, frames from concurrent calls interleaving freely. No new handshake per call, no connection-pool tuning, and request 2 does not wait for request 1 at the HTTP layer (packet loss still stalls all streams at the TCP layer — HTTP/2 fixed HTTP head-of-line blocking, not TCP’s).

That efficiency has a production consequence the Hook paid for. A layer-4 balancer — Kubernetes ClusterIP, AWS NLB — balances at connection-accept time. A gRPC client presents exactly one long-lived connection, so all of its RPCs pin to whichever backend accepted it. Scaling out adds pods that no existing client will ever talk to. You have three real options:

• Client-side balancing: a headless Service (DNS returns all pod IPs) plus "loadBalancingConfig": [{"round_robin":{}}] — the channel maintains a sub-connection per backend and spreads calls. Caveat: DNS re-resolution is lazy; pair it with server-side MaxConnectionAge so connections recycle and discover new pods.
• An L7 proxy (Envoy, Linkerd): the proxy speaks HTTP/2 on both sides and balances per stream, not per connection. This is the mesh answer and costs you a hop.
• Doing nothing is also a decision — fine for two pods and fatal for autoscaling.

Together these options span the spectrum from zero infrastructure change to full service mesh. When you scale a gRPC service and traffic distribution does not follow, the cause is almost always that you chose the third option without knowing it.

Four call types, deadlines that travel, and codes that lie to dashboards

The stream keyword generalizes RPC into four shapes: unary (one request, one response — most calls), server streaming (one request, many responses: watches, feeds, paging a huge result set without buffering it), client streaming (many requests, one response: uploads, telemetry batching), and bidirectional (both at once: chat, sync protocols, live transcription). Generated Go code gives streams Send and Recv methods over the same context machinery you already know.

Deadlines are where gRPC quietly beats hand-rolled HTTP conventions. Set one on the client context and it is transmitted as the grpc-timeout header on the call — and on every downstream call made from that context, with the remaining budget. A 500ms edge deadline arrives at the third hop as whatever is left, and every service fails fast together with DeadlineExceeded. With plain HTTP you reinvent this with custom headers and team discipline; gRPC ships it in the runtime.

Two more things senior engineers get burned by. First, gRPC status codes are not HTTP status codes: the HTTP layer says 200 for nearly everything, and the real verdict — OK, Unavailable, DeadlineExceeded, NotFound — rides in the grpc-status trailer. A dashboard counting HTTP 200s on a gRPC service is measuring nothing. Second, cross-cutting concerns go in interceptors — the middleware of gRPC, client and server, unary and stream variants:

func authInterceptor(ctx context.Context, req any, info *grpc.UnaryServerInfo,
	next grpc.UnaryHandler) (any, error) {
	if err := verifyToken(ctx); err != nil {
		return nil, status.Error(codes.Unauthenticated, "bad token")
	}
	return next(ctx, req) // chain continues — same shape as http middleware
}