Batching in Kafka and Postgres

A team migrates a logging pipeline to Kafka. Throughput tops out at 40 MB/s and the brokers are bored — disk and network both under 20%. They add producers, add partitions, nothing moves. The cause was one default no one looked at: linger.ms=0. Every message was flushed as its own request the instant send() returned, so “batching” never happened. One line — linger.ms=20, batch.size=131072 — and the same cluster pushed 300 MB/s. The bottleneck was never the brokers. It was a producer that refused to wait.

Every write has a fixed cost (network round-trip, syscall, ACK, an fsync) and a variable cost (the bytes themselves). When the fixed cost dominates — and for small messages it always does — batching amortizes it across N records. The two-dimensional window from the previous lesson (max-size OR max-wait) is exactly how both Kafka and Postgres decide when the batch goes out. The systems differ only in the mechanism, not the idea.

Kafka producer batching

The Kafka producer does not send one request per send(). It accumulates records in an in-memory buffer, one batch per (topic, partition), and ships a batch when either trigger fires first:

• batch.size is reached — the per-partition byte cap, default 16 KB (16384 bytes).
• linger.ms expires — the max time the producer will wait for more records to fill a batch.

Here is the trap that bit the team above. Through Kafka 3.x, linger.ms defaulted to 0, meaning “send as soon as you can.” That does not disable batching outright — records that pile up while a request is in flight still coalesce — but under steady, well-paced load it effectively sends tiny batches, capping throughput far below the hardware. Kafka 4.0 changed the default to linger.ms=5 precisely because so many deployments left throughput on the table. The fix for a throughput-bound producer is to deliberately wait: linger.ms in the 5–100 ms range, batch.size raised to 64–512 KB. You trade a few ms of producer-side latency for fewer, fatter requests.

The senior tradeoff is sharp: linger.ms adds latency only when the batch is not already full. Under high load, batch.size fills before the timer expires, so the timer never costs you anything — you get big batches for free. Under low load, linger.ms is a small, bounded tax that buys efficiency. That is why a non-zero linger almost always wins: it can only help when you have spare time anyway.

Postgres bulk writes: INSERT vs COPY

The same shape appears on the database side, three rungs of amortization:

• Per-row INSERT — every statement is parsed, planned, executed, and (with autocommit) committed on its own. The commit forces a WAL flush. Throughput lands around 1–5k rows/s. In a hot path this is where ingestion goes to die.
• Multi-row INSERT ... VALUES (...), (...), ... — one parse, one plan, one commit for the whole statement. Now the fixed cost is amortized across the rows in the statement: roughly 5–50k rows/s. The catch is a practical cap on rows per statement (parameter limits, statement size), so you chunk into batches of a few hundred to a few thousand.
• COPY — Postgres’s bulk path. It streams rows over a dedicated wire protocol, bypassing the per-row SQL parser and planner entirely, and commits once. The docs are explicit that COPY is “significantly more optimized” than a series of INSERTs and that disabling autocommit isn’t even needed because it is already a single command. Throughput reaches 50–500k rows/s.

ORMs expose the middle and top rungs so you rarely hand-write them: Django’s bulk_create, SQLAlchemy’s executemany and COPY helpers, Rails’ insert_all. The senior move is to recognize which rung a code path is silently on — an ORM loop that calls save() per object is per-row INSERT wearing a nicer API, and it will quietly cap a bulk import at a few thousand rows a second.

Why compression needs batching first

Compression and batching are not two separate wins — compression depends on batching. Dictionary-based codecs (lz4, snappy, zstd) find repetition within their input window. A single 200-byte log line has almost no internal repetition, so compressing it alone saves little and can even add overhead. Batch a few hundred of those similar lines together and the shared structure — same field names, same hostnames, same JSON keys — becomes highly compressible. In Kafka the codec runs per batch, on the whole record set, which is why compression.type does so little with linger.ms=0 and so much with fat batches.

The numbers favor it overwhelmingly. A 128 KB batch of repetitive event data routinely compresses 2–4x with lz4 or zstd (zstd typically 20–30% better than lz4 at higher CPU cost). Compression CPU runs around 50–100 MB/s per core — cheap next to the network bytes and the broker fsync you avoid, since the compressed batch is what hits the disk and what replicas copy. Batch first, then compress: the order is not optional.