TokenPad

Recipe · Streaming

Stream responses in Node

Stream tokens as they generate in Node, handle abort correctly, and check the stop reason so a truncated answer never reaches a user unnoticed.

The problem

A long response takes tens of seconds to generate. Without streaming the user sees nothing until it finishes, and your HTTP client may time out before it does.

Streaming solves both, but the naive implementation drops two things that matter: cancellation and the stop reason.

Stream to a client, with aborttypescript
import OpenAI from 'openai';

const client = new OpenAI();

export async function streamAnswer(
  prompt: string,
  onToken: (text: string) => void,
  signal?: AbortSignal,
): Promise<{ text: string; stopReason: string | null }> {
  const stream = await client.chat.completions.create(
    {
      model: 'gpt-5.6-terra',
      messages: [{ role: 'user', content: prompt }],
      max_tokens: 1024,
      stream: true,
    },
    { signal },
  );

  let text = '';
  let stopReason: string | null = null;

  for await (const chunk of stream) {
    const delta = chunk.choices[0]?.delta?.content;
    if (delta) {
      text += delta;
      onToken(delta);
    }
    // The final chunk carries the reason generation ended.
    stopReason = chunk.choices[0]?.finish_reason ?? stopReason;
  }

  return { text, stopReason };
}
Treat truncation as a failure, not a resulttypescript
const { text, stopReason } = await streamAnswer(prompt, onToken);

// "length" means the model hit max_tokens, not that it finished.
// There is no error — the request succeeded and the answer is incomplete.
if (stopReason === 'length') {
  throw new TruncatedResponseError(
    'Generation hit the token ceiling. Raise max_tokens or ask for a shorter answer.',
  );
}

return text;
Server-sent events from an HTTP handlertypescript
export async function GET(request: Request) {
  const encoder = new TextEncoder();

  const stream = new ReadableStream({
    async start(controller) {
      try {
        await streamAnswer(
          'Explain tokenization in two sentences.',
          (delta) => controller.enqueue(encoder.encode(`data: ${JSON.stringify({ delta })}\n\n`)),
          request.signal,
        );
        controller.enqueue(encoder.encode('data: [DONE]\n\n'));
      } catch (error) {
        // Surface the failure to the client instead of closing silently.
        const message = error instanceof Error ? error.message : 'stream failed';
        controller.enqueue(encoder.encode(`data: ${JSON.stringify({ error: message })}\n\n`));
      } finally {
        controller.close();
      }
    },
  });

  return new Response(stream, {
    headers: {
      'content-type': 'text/event-stream',
      'cache-control': 'no-cache, no-transform',
      connection: 'keep-alive',
    },
  });
}

Why it is written this way

Pass the abort signal through

When a user closes the tab, the request should stop generating. Without the signal you keep paying for tokens nobody will read — on a chat product with any abandonment rate, that is a real line on the bill.

Streaming does not make generation faster

It changes what the user waits for: the first token rather than the last. That is commonly a five to ten times improvement in perceived latency for no cost beyond handling the stream.

Disable proxy buffering

The no-transform cache header matters. Some proxies and CDNs buffer responses by default, which collects the whole stream and delivers it at once — removing the entire benefit while looking like it works locally.

What breaks the naive version

  • Ignoring finish_reason. A truncated answer arrives as a successful request and will be rendered to a user mid-sentence with nothing in your logs.
  • Not handling errors mid-stream. Once headers are sent you cannot return an error status, so the failure has to travel in the stream body.
  • Accumulating the full text but forgetting the abort path, so cancelled requests keep billing.

Check your numbers