Streaming

Stream answers as they’re written,or follow long jobs live.

Set stream to true and tokens arrive as the model writes them, in the same format as OpenAI’s streaming API. For longer work, start an async request and follow its progress over a WebSocket from your frontend.

Start building freeRead the streaming docs

How streaming works

Two ways to get answers to people quickly. Use the one that fits the job.

  1. 01

    Ask for a stream

    Add stream: true to a request on /v1/ai or /v1/chat/completions.

  2. 02

    Read the chunks

    Tokens arrive as server-sent events in OpenAI’s chunk format, ending with data: [DONE].

  3. 03

    Or go async

    POST /v1/ai/async answers right away with a channel ID and a one-time WebSocket token.

  4. 04

    Follow it live

    Your frontend joins the channel and receives progress, streamed tokens and the final answer.

The numbers

What it looks like

A stream is a normal request with one extra field. Async gives you a channel to follow.

streamed response
POST /v1/ai
{ "workflow": "support_reply", "messages": [ … ], "stream": true }

// Server-sent events, in OpenAI’s chunk format
data: {"object":"chat.completion.chunk","choices":[{"delta":{"content":"Sure"}}]}
data: {"object":"chat.completion.chunk","choices":[{"delta":{"content":", here’s how"}}]}
: heartbeat
data: [DONE]
async response
POST /v1/ai/async  →  202 Accepted
{
  "channel_id": "550e8400-…",
  "ws_token": "…",
  "websocket_url": "wss://…",
  "websocket_channel": "ai_response:<project>:<channel>"
}

  • Works with OpenAI stream readers

    Code that already reads OpenAI streams can read ModelRiver streams.

  • Live updates without polling

    Async requests push progress and the final answer to the browser, so your frontend never has to poll.

  • Survives a page refresh

    The client SDK remembers the channel, so after a refresh it can pick up where it left off.

Good to know

Frequently asked questions

Which endpoints can stream?

/v1/ai and /v1/chat/completions, with stream set to true. Async requests with stream set to true send tokens over the WebSocket instead.

Can I stream structured outputs?

No. Structured outputs are checked against your schema once the answer is complete, so they come back in one piece.

How long can a stream stay open?

A heartbeat is sent every 15 seconds, and a stream on /v1/ai can stay open for up to 5 minutes.

When should I use async instead?

For long jobs, for backend pipelines, or when the browser should get updates without holding a request open.

Do I need the client SDK?

No. The WebSocket uses Phoenix Channels, so any Phoenix client works. The SDK wraps it for React, Vue, Angular and Svelte and handles reconnecting.

Stream your first answer

Start on the free plan and add stream: true to a request.

Start building freeRead the streaming docs