What is streaming in an LLM API?

Streaming is delivering a model's response piece by piece as it is generated instead of waiting for the full answer. The API keeps the connection open and sends small chunks, typically as server-sent events. Users see text appear immediately, which makes long answers feel fast even when total generation time is unchanged.

Also called: Server-sent events, SSE streaming · Last updated

How to use it

Set stream to true in a chat completions request and read the events as they arrive. Each event carries a fragment of the answer; the final events carry the finish reason and token usage.

Errors in a stream

Once a stream has started, the HTTP status is already 200. An error that happens later arrives as an event in the stream, so clients must read errors from the event payload.

Learn more

Related terms