Streaming

Streams use server-sent events in the vendor's own format. The bytes you receive are the bytes the model produced; the gateway only watches them to measure usage and to protect you from failures.

Turn on streaming#

The SDK guides show the code: Anthropic, OpenAI, Gemini.

What a stream looks like#

event: message_start
data: {"type":"message_start","message":{"id":"msg_01HCz7","type":"message","role":"assistant","model":"claude-sonnet-5-5","content":[],"stop_reason":null,"usage":{"input_tokens":14,"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"output_tokens":1}}}

event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"One, two, three."}}

event: content_block_stop
data: {"type":"content_block_stop","index":0}

event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":null},"usage":{"output_tokens":8}}

event: message_stop
data: {"type":"message_stop"}

Early failures are retried for you#

A channel sometimes accepts a request and then fails before it produces any output, for example by sending message_start followed by an overload error. The gateway holds back the start of every stream until the first token of content arrives (text, thinking or a tool call). If the channel fails before that point, nothing has reached you yet, so the gateway retries the request on another channel and you receive a clean stream from the start.

A channel that is merely slow to start counts too: for Claude models without extended thinking, a channel that hasn't produced its first token within 10 seconds (1 second more per 10K prompt tokens) is left for another one, unless it is the last channel available. Requests with thinking can take as long as the model thinks. Non-streaming requests follow the same rule.

What this means for your client:

• The HTTP status and headers arrive together with the first token, so time to first byte equals time to first token.
• If every channel fails before the first token, you get an ordinary HTTP error response in your API's error format, not a broken stream.
• The hold lasts at most 15 seconds. A model that thinks silently for longer starts streaming at that point, and from then on the request is committed to its channel.
• Allow at least 30 seconds before the response starts. A request can also wait a few seconds for capacity first; see Rate limits & failover.

Stalls and errors mid-stream#

Once tokens are flowing, the request is committed to one channel and can no longer move. Two things can still end it early.

A stall. If the channel sends nothing for 3 minutes, the gateway ends the stream with an error event in your API's format and closes the connection:

event: error
data: {"type":"error","error":{"type":"api_error","message":"Upstream stopped responding mid-stream"}}

An error from the channel. A channel can report an error inside a running stream, such as an overloaded model. You receive a generic error event in your API's format instead of the channel's own text: overloaded_error (OpenAI rate_limit_exceeded, Gemini RESOURCE_EXHAUSTED) when the channel is busy, otherwise api_error (Gemini UNAVAILABLE) with the message The model provider returned an error. Please retry.

In both cases the stream ends without its normal last event (message_stop, data: [DONE] or a finishReason). Treat a stream that ends without it as incomplete: keep or discard the partial output, then retry the whole request if you need a full answer. The Anthropic and OpenAI SDKs raise an exception when they receive an error event.

Cancel a stream#

Close the connection to stop generation, for example by breaking out of the loop or calling stream.close() / controller.abort(). The gateway stops the model and charges for what was produced up to that point:

• Input uses the count the stream already reported (Anthropic reports it at the start), or an estimate of about four characters per token of your request if none was reported.
• Output is the larger of what the stream reported so far and an estimate of about four characters per token of the text already sent to you. A stream cut short rarely reports its final output count, so the estimate is what usually applies.

A stream that ends with a stall error is billed the same way, for what you received. Every cancelled or stalled stream appears in Logs with its charge. See Models & pricing.