Streaming

Stream output as it is produced instead of waiting for the full response — cutting perceived latency for chat, and enabling real-time playback and transcription for voice. This guide covers chat streaming; for audio output see streaming speech below, and for live transcription see realtime STT further down.

How it works

Set stream: true in your request. The API responds with a series of server-sent events (SSE). Each event contains a chat.completion.chunk with a delta object — read choices[0].delta.content and concatenate to build the full message. The stream ends with data: [DONE].

Python

from inferexai import InferexAI

client = InferexAI(api_key="sk-live-your-key")

stream = client.chat.completions.create(
    model="default",
    messages=[{"role": "user", "content": "Write a haiku about APIs."}],
    stream=True,
)

for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)
print()  # newline at end

Node.js

import InferexAI from "inferexai";

const client = new InferexAI({ apiKey: "sk-live-your-key" });

const stream = await client.chat.completions.create({
  model: "default",
  messages: [{ role: "user", content: "Write a haiku about APIs." }],
  stream: true,
});

for await (const chunk of stream) {
  const delta = chunk.choices[0]?.delta?.content ?? "";
  process.stdout.write(delta);
}

Raw fetch (browser / edge)

Parse SSE manually (browser / edge runtimes without the SDK):

const response = await fetch(
  "https://api.inferexai.in/v1/chat/completions",
  {
    method: "POST",
    headers: {
      "Authorization": "Bearer sk-live-your-key",
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      model: "default",
      messages: [{ role: "user", content: "Hello!" }],
      stream: true,
    }),
  }
);

const reader = response.body.getReader();
const decoder = new TextDecoder();

while (true) {
  const { done, value } = await reader.read();
  if (done) break;

  const lines = decoder.decode(value).split("\n");
  for (const line of lines) {
    if (!line.startsWith("data: ")) continue;
    const data = line.slice(6).trim();
    if (data === "[DONE]") break;

    const chunk = JSON.parse(data);
    const delta = chunk.choices[0]?.delta?.content ?? "";
    process.stdout.write(delta);
  }
}

Streaming speech (TTS)

Text-to-speech streams too, so playback can start before synthesis finishes — essential for voice agents and any real-time audio. Two ways to consume it:

• Server-sent events — set stream_format: "sse" to receive OpenAI speech events (speech.audio.delta / speech.audio.done).
• Raw PCM — request a raw response_format (pcm, or mulaw for telephony) and read the body incrementally. Voice-agent frameworks like pipecat and LiveKit do this out of the box — no client changes.

Full request/response details and examples are in the Text-to-Speech reference → Streaming.

Streaming transcription (STT)

Speech-to-text streams in the other direction: open a WebSocket, send audio up as it is captured, and receive transcript events back with sub-second latency — for live calls rather than pre-recorded files. Not every STT model supports this; check the streaming field on GET /v1/models before opening the socket rather than discovering it from a rejected handshake.

Connection, message shapes, and error handling are in the Speech-to-Text reference → Realtime transcription.