Streams, pipes and backpressure

Why streams exist: bytes you never hold all at once

What if your 4GB export job OOMed not because of a bug in your logic, but simply because you loaded the file before touching it? That’s the problem streams solve — and once you see it, you’ll reach for them every time data size is unbounded.

A stream is an abstraction for data that arrives or departs over time, processed in small chunks instead of materialised whole in memory. The win is bounded memory: piping a 4GB file through gzip to disk holds only a few chunks at a time — kilobytes, not gigabytes — so a 4GB job and a 4TB job have nearly the same memory footprint. The second win is latency to first byte: a streaming HTTP response can start sending the head of a result while its tail is still being computed, instead of buffering the whole body before the first byte leaves.

Node models four stream types. A Readable is a source you pull from (a file read, an HTTP request body, a socket). A Writable is a sink you push to (a file write, an HTTP response, a socket). A Duplex is both, with independent read and write sides (a TCP socket). A Transform is a Duplex whose output is a function of its input — gzip, a cipher, a CSV parser, a JSON.parse per line. Transforms are how you do a streaming map: data flows in one side, modified data flows out the other, and you never hold the full dataset.

Readables have two reading modes. In paused mode you explicitly call read() to pull the next chunk. In flowing mode chunks are pushed at you via 'data' events as fast as they arrive — convenient, but if you can’t keep up there’s no built-in brake, which is exactly how you lose data or blow memory. Attaching a 'data' handler or calling pipe() switches a stream to flowing. There’s also object mode: instead of Buffer/string chunks, the stream carries arbitrary JS objects (one parsed record per chunk), which is what makes Transform-based record pipelines ergonomic.

The problem: a fast producer, a slow consumer

The OOM in the hook is the canonical stream failure. A producer (the file read, fast — local disk) feeds a consumer (the S3 upload, slow — network). When you call writable.write(chunk), Node copies the chunk into the writable’s internal buffer and tries to flush it to the underlying resource. If the resource can’t keep up, the buffer grows. Nothing stops it growing — write() always accepts the chunk and returns. If your producer ignores the return value and keeps writing, the buffer grows without bound until the process runs out of heap.

// BUG: ignores backpressure. The buffer grows unbounded under a slow sink.
import { createReadStream, createWriteStream } from "node:fs";

async function copyNaive(src, dst) {
  const read = createReadStream(src);
  const write = createWriteStream(dst);
  for await (const chunk of read) {
    write.write(chunk); // return value discarded — no brake
  }
  write.end();
}

The for await pulls chunks from the fast source as fast as it can and shovels them into a writable that may be far slower. Memory is now whatever the difference in speed times the duration is — i.e. unbounded.

Backpressure: the brake built into write()

Node gives the producer a signal. writable.write(chunk) returns false once the amount buffered exceeds the stream’s highWaterMark (the default is 16KB for byte streams, 16 objects in object mode). false does not mean the write failed — the chunk is still accepted — it means stop sending; the buffer is full enough. A correct producer, on seeing false, pauses and waits for the writable to emit the 'drain' event, which fires once the buffer has been flushed below the high-water mark. Then it resumes. That loop — write until false, wait for 'drain', repeat — is backpressure, and it caps memory at roughly the highWaterMark.

You almost never write that loop by hand. readable.pipe(writable) wires it for you: it reads, writes, and when write() returns false it calls readable.pause(), then resumes on 'drain'. The producer and consumer stay coupled, and memory stays bounded — no matter the size of the source.

// CORRECT: pipeline() wires backpressure AND error propagation AND cleanup.
import { pipeline } from "node:stream/promises";
import { createReadStream, createWriteStream } from "node:fs";
import { createGzip } from "node:zlib";

async function gzipFile(src, dst) {
  await pipeline(
    createReadStream(src),   // Readable  (fast)
    createGzip(),            // Transform (streaming map)
    createWriteStream(dst),  // Writable  (slow)
  );
  // resolves only when the last byte is flushed; rejects if ANY stage errors
}

Where it bites in production

Anywhere a fast side meets a slow side. A reverse proxy piping an upstream response to a client on a slow mobile link: without backpressure the proxy buffers the whole upstream body in RAM per connection, and a few thousand slow clients OOM the box. Large file uploads straight to object storage, like the hook. CSV/NDJSON pipelines that parse millions of rows: a Transform in object mode keeps memory flat regardless of row count, where a JSON.parse(fs.readFileSync(...)) would load the entire file. Server-sent events and log tailing: the client is the slow consumer and backpressure stops the server from queueing megabytes per subscriber.

Modern Node also speaks Web Streams (ReadableStream/WritableStream, the WHATWG API used by fetch and the browser). They have the same backpressure semantics expressed differently — a ReadableStream won’t pull from its source until the consumer asks — and stream.Readable.fromWeb() / .toWeb() bridge the two worlds when you’re gluing fetch() bodies to Node sinks.

When you do write a Transform yourself, override _transform(chunk, encoding, callback) to push transformed output and call callback() when done with that chunk — the framework handles buffering and backpressure on both sides for you, including honouring your output’s highWaterMark. Tuning highWaterMark upward trades memory for fewer drain round-trips (useful for high-throughput byte streams); tuning it down caps memory harder. Most code should leave it at the default.