Streaming responses: ReadableStream, SSE for LLM UIs, and the proxies that buffer them

Returning a stream from a route handler

A route handler returns a web Response, and a Response body can be a ReadableStream — the moment you return it, Next sends headers immediately and flushes chunks as your code enqueues them. Nothing waits for the function to “finish”:

// app/api/chat/route.ts — proxy an LLM, re-emitting tokens as SSE
export async function POST(request: Request) {
  const { prompt } = await request.json();
  const upstream = await llm.stream(prompt);          // async iterable of tokens
  const encoder = new TextEncoder();

  const stream = new ReadableStream({
    async pull(controller) {
      const { value, done } = await upstream.next();
      if (done) {
        controller.enqueue(encoder.encode("data: [DONE]\n\n"));
        controller.close();
        return;
      }
      controller.enqueue(encoder.encode(`data: ${JSON.stringify(value)}\n\n`));
    },
    cancel() {
      upstream.return?.();   // client disconnected — stop paying the LLM
    },
  });

  return new Response(stream, {
    headers: {
      "content-type": "text/event-stream",
      "cache-control": "no-cache, no-transform",
      "x-accel-buffering": "no",                     // nginx: do not buffer
    },
  });
}

Three details in this code carry the production weight. The pull callback is the backpressure hook — more on that below. The cancel callback runs when the client disconnects; without it, a user closing the tab mid-answer leaves your handler happily consuming (and paying for) the rest of the LLM generation. And the headers are the armor against the Hook’s nine-second spinner.

SSE: the boring, right protocol for token streams

For LLM-style UIs you need a framing convention — raw chunks have no message boundaries; a token can arrive split across two network reads. Server-Sent Events is the established answer: text/event-stream content type, messages framed as data: <payload>\n\n, consumed in the browser by EventSource (auto-reconnecting, with Last-Event-ID resume support) or, as every AI SDK does for POST bodies, by reading response.body with a parser. SSE wins over WebSockets here because token streaming is one-directional: no upgrade handshake, plain HTTP semantics (works through standard proxies and HTTP/2 multiplexing), trivially resumable, and a fetch away. WebSockets earn their complexity only when the client must push frequent messages over the same connection.

The failure mode unique to SSE is framing corruption: if you emit a payload containing a raw newline without splitting it across data: lines, the client parser silently truncates messages — JSON-encode every payload (as above) and the problem disappears.

Backpressure: produce at the consumer’s pace

A stream has a producer (your LLM proxy, a database cursor, a file reader) and a consumer (the user’s connection, often a slow phone on weak Wi-Fi). When the producer is faster, the difference accumulates in your server’s memory. Web streams handle this with a pull model: the runtime calls your pull(controller) when the internal queue has room (tracked by controller.desiredSize, fed by the consumer draining bytes), so a correctly written source generates only when asked. The anti-pattern is bypassing the model — a start() callback with a loop that enqueues an entire table scan, or piping a fast upstream into a slow client without awaiting the writer:

// WRONG: ignores backpressure — the whole cursor lands in memory
start(controller) {
  for await (const row of hugeCursor) controller.enqueue(encode(row));
}
// RIGHT: pull() reads one item per request from the consumer side
async pull(controller) {
  const { value, done } = await hugeCursor.next();
  done ? controller.close() : controller.enqueue(encode(value));
}

For token streams the stakes are low — tokens are tiny and humans read slowly. For data export endpoints (CSV dumps, log tails) backpressure is the difference between a 30MB-flat memory profile and an OOM-killed pod when one customer with a slow connection requests a 2GB export. If you compose transforms, pipeThrough/pipeTo propagate backpressure for you; hand-rolled loops are where it leaks.

Two streamings, one word — and the buffering gauntlet

Next.js streams in two distinct senses, and conflating them muddles debugging. Suspense streaming of pages is the App Router rendering HTML progressively: the shell flushes immediately, loading.tsx/<Suspense> fallbacks render, and late server components arrive as out-of-order chunks that inline scripts slot into place. Its unit is UI; the framework owns the protocol. Handler streaming of data — this lesson — is you returning bytes the client code consumes: SSE events, NDJSON lines, CSV rows. Its unit is data; you own the framing, the headers, and the cancellation. They share the transport (chunked HTTP) and therefore share the enemy: anything that buffers.

The gauntlet, hop by hop. nginx: proxy_buffering on by default — defeat it per-response with X-Accel-Buffering: no. Compression: gzip/brotli middleware accumulates input to build compression windows; Cache-Control: no-transform tells well-behaved intermediaries not to re-encode, and your own compression layer must exempt text/event-stream. CDNs: some buffer or even cache streamed responses unless marked no-cache/private. Serverless platforms: classic AWS Lambda behind API Gateway buffered the entire response by design — streaming needs explicitly streaming-capable infrastructure (Lambda response streaming, or edge/Node servers that flush). The diagnostic that cuts through all of it: curl -N directly against the app (streams? your code is fine), then against each layer outward until the stream stops streaming — that hop is your culprit.