- Glossary
- Streaming
What is streaming in an LLM API?
Streaming is delivering a model's response piece by piece as it is generated instead of waiting for the full answer. The API keeps the connection open and sends small chunks, typically as server-sent events. Users see text appear immediately, which makes long answers feel fast even when total generation time is unchanged.
Also called: Server-sent events, SSE streaming · Last updated
How to use it
Set stream to true in a chat completions request and read the events as they arrive. Each event carries a fragment of the answer; the final events carry the finish reason and token usage.
Errors in a stream
Once a stream has started, the HTTP status is already 200. An error that happens later arrives as an event in the stream, so clients must read errors from the event payload.