Streaming

Chat completions as server-sent events, token by token.

"stream": true returns the reply as server-sent events in the OpenAI chat.completion.chunk format.

Stream a reply

Stream every request that can take longer than 100 s: a non-streamed request that sends no response byte for 100 s ends with 524, a gateway timeout.

curl

curl -N https://api.albastov.com/v1/chat/completions \
  -H "Authorization: Bearer $ALBASTAR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.8-27b",
    "stream": true,
    "stream_options": {"include_usage": true},
    "max_tokens": 20,
    "messages": [
      {"role": "user", "content": "Count from 1 to 5, separated by spaces."}
    ],
    "chat_template_kwargs": {"enable_thinking": false}
  }'

Python

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.albastov.com/v1",
    api_key=os.environ["ALBASTAR_API_KEY"],
)
stream = client.chat.completions.create(
    model="qwen3.8-27b",
    stream=True,
    stream_options={"include_usage": True},
    max_tokens=20,
    messages=[
        {"role": "user", "content": "Count from 1 to 5, separated by spaces."},
    ],
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
    if chunk.usage:
        print("\n", chunk.usage)

JavaScript

const res = await fetch("https://api.albastov.com/v1/chat/completions", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.ALBASTAR_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "qwen3.8-27b",
    stream: true,
    stream_options: { include_usage: true },
    max_tokens: 20,
    messages: [
      { role: "user", content: "Count from 1 to 5, separated by spaces." },
    ],
    chat_template_kwargs: { enable_thinking: false },
  }),
});
if (!res.ok) throw new Error(`${res.status} ${await res.text()}`);

const decoder = new TextDecoder();
let buffer = "";
for await (const bytes of res.body) {
  buffer += decoder.decode(bytes, { stream: true });
  let end;
  while ((end = buffer.indexOf("\n\n")) >= 0) {
    const event = buffer.slice(0, end);
    buffer = buffer.slice(end + 2);
    for (const line of event.split("\n")) {
      if (!line.startsWith("data:")) continue; // ": loading ..." comments
      const data = line.slice(5).trim();
      if (data === "[DONE]") continue;
      const chunk = JSON.parse(data);
      if (chunk.error) {
        throw new Error(`${chunk.error.type}: ${chunk.error.message}`);
      }
      const text = chunk.choices?.[0]?.delta?.content;
      if (text) process.stdout.write(text);
      if (chunk.usage) console.log("\n", chunk.usage);
    }
  }
}

Read the stream

Each event is one data: line and a blank line; each data: line carries one chat.completion.chunk object on one line. Answer text arrives in choices[0].delta.content, reasoning in choices[0].delta.reasoning, tool calls in choices[0].delta.tool_calls. The last choice chunk carries finish_reason; data: [DONE] ends the stream. Lines that start with : are comments: skip them.

The stream of the request above, abridged: each chunk's id, object, created and model are left out, the content chunks between the first and the last are left out, and so are the usage values.

Stream (abridged)

data: {"choices":[{"index":0,"delta":{"role":"assistant","content":""}}]}

data: {"choices":[{"index":0,"delta":{"content":"1"}}]}

data: {"choices":[{"index":0,"delta":{"content":"5"}}]}

data: {"choices":[{"index":0,"delta":{"content":""},"finish_reason":"stop"}]}

data: {"choices":[],"usage":{"prompt_tokens":…,"completion_tokens":…}}

data: [DONE]

Every chunk of one stream carries the same id, object chat.completion.chunk, the same created and the model of the request.

Token usage

"stream_options": {"include_usage": true} adds a final chunk with usage and an empty choices list, before data: [DONE]. Without include_usage, the stream has no usage chunk.

Model loading

A streamed request for a model that is not loaded, without X-LLM-Wait, waits for the load, up to 1800 s:

After 10 s without a reply, the response starts: 200, text/event-stream, and the comment : loading <model>, repeated every 15 s.
After the load, the model's events follow on the same connection.
If the load fails, the stream sends one data: {"error": {...}} event, then data: [DONE].

With X-LLM-Wait: N, the stream sends no : loading comments: the response starts after the load, or after N seconds with 503 model_loading or 503 queued. Keep N below 90: a response with no bytes for 100 s ends with 524, a gateway timeout. With X-LLM-Wait: 0, a model that is not loaded returns 503 model_loading or 503 queued at once. Load times and the queue: Model loading.

Stream failures

A failure before the first event is an HTTP error with a JSON body. A failure after the stream has started arrives as an event; check every data: event for an error key.

Event	Next
`data: {"error": {"code": "model_loading", ...}}`	`data: [DONE]`
`data: {"error": {"code": "queued", ...}}`	`data: [DONE]`
`data: {"error": {"code": "gpu-busy", ...}}`	`data: [DONE]`
`data: {"error": {"type": "engine_restarted", "code": 502, ...}}`	The connection closes

engine_restarted: the model produced no token for 90 s and restarts. In this event code is the number 502, so read type. Send the request again; the retry waits while the model loads.

Chunk fields, headers and error codes: Chat completions reference.