--- title: Streaming description: Stream tokens as they are generated in all three API formats. Learn how the gateway hides early failures, how stalls and mid-stream errors appear, and what a cancelled stream costs. --- Streams use server-sent events in the vendor's own format. The bytes you receive are the bytes the model produced; the gateway only watches them to measure usage and to protect you from failures. ## Turn on streaming | API | How to stream | Stream ends with | | --- | --- | --- | | Anthropic Messages | `"stream": true` in the body | `event: message_stop` | | OpenAI Chat Completions | `"stream": true` in the body | `data: [DONE]` | | Gemini | `POST /v1beta/models/{model}:streamGenerateContent?alt=sse` | An event whose candidate has a `finishReason` | The SDK guides show the code: [Anthropic](/docs/models/anthropic#stream-responses), [OpenAI](/docs/models/openai#stream-with-usage), [Gemini](/docs/models/gemini#stream-responses). ## What a stream looks like :::tabs :::tab Anthropic ```sse event: message_start data: {"type":"message_start","message":{"id":"msg_01HCz7","type":"message","role":"assistant","model":"claude-sonnet-5-5","content":[],"stop_reason":null,"usage":{"input_tokens":14,"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"output_tokens":1}}} event: content_block_start data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}} event: content_block_delta data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"One, two, three."}} event: content_block_stop data: {"type":"content_block_stop","index":0} event: message_delta data: {"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":null},"usage":{"output_tokens":8}} event: message_stop data: {"type":"message_stop"} ``` ::: :::tab OpenAI ```sse data: {"id":"chatcmpl-9x2","object":"chat.completion.chunk","model":"gpt-6.1-sol","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]} data: {"id":"chatcmpl-9x2","object":"chat.completion.chunk","model":"gpt-6.1-sol","choices":[{"index":0,"delta":{"content":"One, two, three."},"finish_reason":null}]} data: {"id":"chatcmpl-9x2","object":"chat.completion.chunk","model":"gpt-6.1-sol","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]} data: {"id":"chatcmpl-9x2","object":"chat.completion.chunk","model":"gpt-6.1-sol","choices":[],"usage":{"prompt_tokens":14,"completion_tokens":8,"total_tokens":22,"prompt_tokens_details":{"cached_tokens":0},"completion_tokens_details":{"reasoning_tokens":0}}} data: [DONE] ``` The chunk with `"choices": []` and `usage` is always present: the gateway sets `stream_options.include_usage` on every streaming request. ::: :::tab Gemini ```sse data: {"candidates":[{"content":{"role":"model","parts":[{"text":"One, two,"}]}}],"usageMetadata":{"promptTokenCount":6,"candidatesTokenCount":4,"totalTokenCount":10}} data: {"candidates":[{"content":{"role":"model","parts":[{"text":" three."}]},"finishReason":"STOP"}],"usageMetadata":{"promptTokenCount":6,"candidatesTokenCount":8,"totalTokenCount":14}} ``` Without `?alt=sse`, Gemini streams a single JSON array instead. Prefer SSE: it is easier to parse incrementally, and it is the only Gemini mode in which a stall is reported with an error event. ::: ::: ## Early failures are retried for you A channel sometimes accepts a request and then fails before it produces any output, for example by sending `message_start` followed by an overload error. The gateway holds back the start of every stream until the first token of content arrives (text, thinking or a tool call). If the channel fails before that point, nothing has reached you yet, so the gateway retries the request on another channel and you receive a clean stream from the start. A channel that is merely slow to start counts too: for Claude models without extended thinking, a channel that hasn't produced its first token within 10 seconds (1 second more per 10K prompt tokens) is left for another one, unless it is the last channel available. Requests with thinking can take as long as the model thinks. Non-streaming requests follow the same rule. What this means for your client: - The HTTP status and headers arrive together with the first token, so time to first byte equals time to first token. - If every channel fails before the first token, you get an ordinary HTTP error response in your API's error format, not a broken stream. - The hold lasts at most 15 seconds. A model that thinks silently for longer starts streaming at that point, and from then on the request is committed to its channel. - Allow at least 30 seconds before the response starts. A request can also wait a few seconds for capacity first; see [Rate limits & failover](/docs/models/rate-limits). ## Stalls and errors mid-stream Once tokens are flowing, the request is committed to one channel and can no longer move. Two things can still end it early. **A stall.** If the channel sends nothing for 3 minutes, the gateway ends the stream with an error event in your API's format and closes the connection: :::tabs :::tab Anthropic ```sse event: error data: {"type":"error","error":{"type":"api_error","message":"Upstream stopped responding mid-stream"}} ``` ::: :::tab OpenAI ```sse data: {"error":{"message":"Upstream stopped responding mid-stream","type":"api_error","code":"upstream_stall"}} ``` ::: :::tab Gemini ```sse data: {"error":{"code":504,"message":"Upstream stopped responding mid-stream","status":"DEADLINE_EXCEEDED"}} ``` ::: ::: **An error from the channel.** A channel can report an error inside a running stream, such as an overloaded model. You receive a generic error event in your API's format instead of the channel's own text: `overloaded_error` (OpenAI `rate_limit_exceeded`, Gemini `RESOURCE_EXHAUSTED`) when the channel is busy, otherwise `api_error` (Gemini `UNAVAILABLE`) with the message `The model provider returned an error. Please retry.` In both cases the stream ends without its normal last event (`message_stop`, `data: [DONE]` or a `finishReason`). Treat a stream that ends without it as incomplete: keep or discard the partial output, then retry the whole request if you need a full answer. The Anthropic and OpenAI SDKs raise an exception when they receive an error event. ## Cancel a stream Close the connection to stop generation, for example by breaking out of the loop or calling `stream.close()` / `controller.abort()`. The gateway stops the model and charges for what was produced up to that point: - Input uses the count the stream already reported (Anthropic reports it at the start), or an estimate of about four characters per token of your request if none was reported. - Output is the larger of what the stream reported so far and an estimate of about four characters per token of the text already sent to you. A stream cut short rarely reports its final output count, so the estimate is what usually applies. A stream that ends with a stall error is billed the same way, for what you received. Every cancelled or stalled stream appears in [Logs](/app/logs) with its charge. See [Models & pricing](/docs/models/pricing#cancelled-and-interrupted-streams).