Streaming: who pays when nobody listens

The frames, in order

A chat completion stream is a sequence of newline-delimited JSON payloads, each prefixed data: and terminated by a blank line. Every payload is an object with type chat.completion.chunk. The shape is OpenAI's exactly, because the whole premise of this API is that an existing SDK keeps working after a base-URL change.

What varies between implementations is the frame order, and that is what breaks a hand-rolled parser. Ours is fixed. One role frame with no content, then one frame per content chunk, then a frame carrying the finish reason and an empty delta, then optionally a usage frame, then the sentinel.

curl -N https://router.xark.io/api/v1/chat/completions \
  -H "Authorization: Bearer sk-..." \
  -H "Content-Type: application/json" \
  -d '{"model":"z-ai/glm-5.2","stream":true,
       "stream_options":{"include_usage":true},
       "messages":[{"role":"user","content":"Hi"}]}'

# data: {"id":"...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}],"usage":null}
# data: {"id":"...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}],"usage":null}
# data: {"id":"...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}],"usage":null}
# data: {"id":"...","object":"chat.completion.chunk","choices":[],"usage":{"prompt_tokens":2,"completion_tokens":77,"total_tokens":79,"cost_usd":0.00019799}}
# data: [DONE]

Usage arrives on exactly one frame, and only if you ask

The usage block does not exist on a stream by default. A streaming caller who does not set stream_options.include_usage never learns what the call cost, from us or from anyone else implementing this contract, because the field only lives on the blocking response.

Set it and one extra frame appears after the last content frame and before the sentinel: empty choices array, populated usage, including our cost_usd. While include_usage is on, every earlier frame carries usage: null rather than omitting the key. That is deliberate. A client can read chunk.usage on every frame without a presence check and will get its answer on exactly one of them, rather than having to branch on whether the key exists.

Setting stream_options without stream: true is a 400, not a silent no-op. The message is 'stream_options' can only be set when 'stream' is true, with param stream_options. Accepting it quietly would mean a caller who forgot stream: true gets a blocking response, sees a usage block, and concludes their configuration works.

The responses endpoint speaks a different protocol

This is the single most common thing that breaks when a migration moves from chat completions to the newer responses endpoint, and it breaks at the parser rather than at the request. Responses does not emit untyped chunks and has no [DONE] sentinel at all.

It emits named SSE events: an event: line followed by a data: line whose payload repeats the type and adds a sequence_number, so a client can detect a dropped frame rather than silently truncating an answer. The stream terminates on response.completed, or on response.incomplete when generation stopped at the output-token ceiling. A client waiting for [DONE] on this endpoint waits forever.

The frame count is also fixed rather than proportional. Eight structural events wrap the content: created, in_progress, output_item.added, content_part.added, then the deltas, then output_text.done, content_part.done, output_item.done and the terminator. Every one of them carries the full response envelope or an index into it, which is why walking output[] and switching on the item type is the only parse that survives a reasoning item appearing ahead of the message.

// Chat completions: untyped frames, [DONE] terminates.
for (const frame of frames) {
  if (frame === "[DONE]") break;
  const chunk = JSON.parse(frame);
  if (chunk.usage) cost = chunk.usage.cost_usd;   // exactly one frame
  out += chunk.choices[0]?.delta?.content ?? "";
}

// Responses: named events, no sentinel. Terminate on the event name.
if (event.type === "response.output_text.delta") out += event.delta;
if (event.type === "response.completed") done = true;
if (event.type === "response.incomplete") done = true;   // hit the ceiling
// event.sequence_number increments by 1; a gap means a dropped frame.

Batched prompts stream one at a time, not interleaved

The legacy completions endpoint accepts an array of prompts, and a client that assumes the results come back interleaved will assemble nonsense. They do not. Each choice is streamed to completion, every frame tagged with that choice's index, before the next choice starts. The object type is text_completion rather than chat.completion.chunk, and the same optional usage frame and the same [DONE] sentinel apply.

Two other restrictions on that endpoint are worth knowing before you build on it. n must be 1, because generation here is deterministic and additional choices would be identical copies you were billed for. And echo output is not counted as completion tokens: charging a caller twice for their own prompt is precisely what this platform's pricing argument is against.

What the relay does to a stream, and what it refuses to do

The proxy in front of the gateway passes the response body through as a stream rather than buffering it. That sounds like an implementation detail and is not: a buffered stream and a real one are indistinguishable in a status check and instantly obvious to a human watching a cursor sit still for eight seconds and then dump a paragraph.

Three headers are managed on the way back. content-encoding and content-length are stripped from every relayed response, because both describe a body that has already been re-framed. And on any response whose content-type is text/event-stream, cache-control: no-cache, no-transform and x-accel-buffering: no are set explicitly, because a single buffering intermediary anywhere in the path destroys the entire property.

Errors are the exception to streaming: a response with a status of 400 or above and a JSON content-type is buffered and normalised. Buffering a failed request costs nothing, because the body is a few hundred bytes. Buffering a successful one would cost everything.

Who pays when the client disconnects

Until 7 September 2026 the honest answer was that you did, and the reason was one line of the relay: the outbound fetch carried a single abort signal, a 300-second timeout, never chained to the inbound request's signal. A client that closed the connection propagated nothing upstream, the model finished the completion it had started, and the full token count was metered. That line now reads `AbortSignal.any([request.signal, AbortSignal.timeout(TIMEOUT_MS)])`, so a disconnect does reach the upstream and generation stops. One caveat stated plainly: at the time of writing no streamable model is callable on this account's credit type, so cancellation propagation has been verified by inspection and by a non-streaming request completing normally, not by watching a stream stop.

What has not changed is that cancellation is not instantaneous, and everything generated before the abort lands is metered. Treating a disconnect as a refund is still wrong. Here is what abandoning a 2,000-token completion after the first 500 tokens costs if none of it is cut short -- the worst case, and the one worth budgeting against -- computed from the published output rates.

Why that matters more than the per-request number suggests

Individually these are fractions of a cent. The DeepSeek V4 Flash figure, $0.00027, is 135 quota units on the gateway's integer grid, where one unit is $0.000002. Nobody notices one.

The shape that costs money is a user-facing chat where people interrupt. Ten thousand abandoned GLM-5.2 streams a day, each cut halfway through a 2,000-token answer, is about $38 a day in tokens that were generated after the reader left. That is the number to check before deciding whether an interruptible UI needs a shorter max_completion_tokens rather than a longer one.

Three things reduce it far more reliably than cancellation does. Cap max_completion_tokens at what your interface can actually display. Ask for shorter answers in the prompt, which is free. Or move the interruptible surface to a cheaper model: the same abandonment pattern on DeepSeek V4 Flash costs $2.70 a day rather than $38.

And it is worth being clear about what a five-minute timeout is and is not. It bounds a stream that never terminates. It does not bound your bill on a stream that terminates normally three seconds after you stopped listening.

An error after the first frame is not rewritten

This is the one asymmetry in the error contract, and it exists because of what the relay can see. The rewrite that turns the gateway's 403 with a Chinese-language insufficient_user_quota into the documented 429 with insufficient_credits fires on any upstream response with a status of 400 or above. It used to additionally require a JSON content-type, which meant a stream: true request that failed before its first frame -- the gateway answers those with a 403 whose content-type is still text/event-stream -- skipped the rewrite entirely. That condition was removed on 7 September 2026, and the streaming out-of-credit path now returns the documented 429, verified against production.

A stream that has already begun has neither. It is a 200 with content-type text/event-stream. So a failure that occurs after the first frame -- a balance exhausted mid-generation, an upstream fault -- arrives inside the stream in whatever shape and whatever language the gateway used, and passes through untouched.

The practical consequence is that a robust client cannot treat the SSE body as guaranteed-clean JSON of a known shape. Parse defensively, and treat a stream that ends without its terminator -- no [DONE] on chat or completions, no response.completed or response.incomplete on responses -- as a failed request rather than a short answer. That distinction is the one your retry logic needs, and the terminator is the only reliable signal for it.

A short checklist for consuming a stream

Set stream_options.include_usage, always. It costs nothing and it is the only way a streaming caller ever learns the real charge rather than an estimate.

Terminate on the terminator, not on the socket closing. [DONE] for chat and completions; response.completed or response.incomplete for responses. A socket that closes without one is an error, not an end.

Watch sequence_number on the responses endpoint. It exists so a dropped frame is detectable, and nothing else will tell you.

Do not budget on cancellation saving money. It now propagates, but it lands late and is not a refund; if interruption is normal in your interface, the reliable levers are max_completion_tokens and model choice.

Log usage.cost_usd from the usage frame per request. A cost regression shows up in the distribution of that field long before it shows up on a balance.