OpenAI Realtime transcription protocol and its dialects — STT (WebSocket)

Code: brisvox_providers::openai_realtime_stt (feature provider-openai-realtime-stt, server feature of the same name, provider = "openai_realtime_stt" with [stt.openai_realtime_stt] dialect = "openai" | "nim" | "speaches" | "vllm"), on the shared session core brisvox_providers::wsstt. Survey and alternatives: docs/providers/local-speech.md; the speech-to-speech use of the same API: docs/providers/openai-realtime.md.

Read 2026-10-14 through a summarising fetch tool (documentation pages: field lists, not wire captures) and, for Speaches and vLLM, from the server source itself (labelled "source"). Status: not yet verified against a real server. This file has no live results; nothing was run against any of the four servers (the stack that was run locally uses sherpa-onnx instead: docs/providers/sherpa-onnx.md). The mocks in crates/brisvox-providers/tests/openai_realtime_stt.rs implement what is written here; a live run (with the existing OpenAI key) must confirm it.

Sources:

• OpenAI: https://developers.openai.com/api/docs/guides/realtime-transcription (session shape, models, events) and https://developers.openai.com/api/docs/guides/voice-websockets (URL and auth of a Realtime WebSocket)
• NVIDIA Speech NIM: https://docs.nvidia.com/nim/speech/latest/reference/api-references/asr/realtime-asr.html
• Speaches (source): https://raw.githubusercontent.com/speaches-ai/speaches/master/src/speaches/ routers/realtime_ws.py, realtime/session.py, realtime/session_event_router.py, realtime/input_audio_buffer_event_router.py, types/realtime.py
• vLLM (source): https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/entrypoints/speech_to_text/realtime/ connection.py, protocol.py; docs https://docs.vllm.ai/en/latest/serving/online_serving/speech_to_text/ and https://docs.vllm.ai/en/latest/examples/speech_to_text/realtime/

Dialect table

What the adapter does

• One protocol, a small table. Dialect fixes the URL, rate, session message(s), whether interims exist and the minimum commit. Event names are matched for all dialects at once (...transcription.delta/completed, transcription.delta/done, error, ...failed, input_audio_buffer.committed); everything else (session.created, rate_limits.updated, ...) is ignored. The mock servers, not the table, decide what a server sends.
• Turn control is ours. The engine's VAD ends the turn; Finalize is input_audio_buffer.commit (vLLM: the final: true commit, with the audio gate shut until transcription.done and a new generation started by a plain commit; audio that arrives meanwhile is held in the ring and sent after it). Server-side turn ends (Speaches' VAD, NIM's endpointing) arrive as additional speech_final transcripts that are not from_finalize; the engine closes them with the next UtteranceFinal (crates/brisvox-engine/src/stt.rs).
• Which transcript answers a commit. The item_id of input_audio_buffer.committed pairs the commit with its completed; a completed of another item is a server turn. Servers that send no committed (vLLM has no items): the first transcript after the commit answers it.
• Nothing to flush. A Finalize with nothing appended since the last commit, or less than the minimum (min_commit: 100 ms OpenAI and Speaches, 0 NIM and vLLM), is answered empty at once and sends nothing; an error event while a commit is open (an empty buffer, a failed transcription) answers it empty too. The engine promotes the latest interim when an answer is empty.
• Timeout. No answer within finalize_timeout (default 2 s, the engine gives up at 3 s): the latest hypothesis is delivered as the answer, the item is remembered and its late transcript is dropped (no second final). vLLM: the connection is replaced instead (a generation may still run).
• Errors. invalid_api_key / authentication_error -> Rejected 401, model_not_found -> Rejected 404, insufficient_quota -> Rejected 429; any other error event is logged and the session goes on. Handshake status -> Rejected. Not JSON -> Protocol.
• Liveness. These servers do not answer every chunk, so the session core proves liveness with WebSocket pings (any frame counts) instead of expecting messages: see wsstt.
• Auth transport. A key is only sent over wss:// or to a loopback host; a keyless local server may be reached over plain ws:// anywhere (a GPU box on the LAN).

Configuration

docs/CONFIG.md ([stt.openai_realtime_stt]). Minimal local stacks:

# NVIDIA Speech NIM (Nemotron ASR Streaming, Turkish and English), 16 kHz
[stt]
provider = "openai_realtime_stt"
[stt.openai_realtime_stt]
dialect = "nim"
base_url = "ws://gpu-box.lan:9000"

# Speaches (Whisper; no interim text), 24 kHz
[stt]
provider = "openai_realtime_stt"
[stt.openai_realtime_stt]
dialect = "speaches"
model = "Systran/faster-whisper-small"

The session language (agent.language / stt.language) is mapped per dialect: OpenAI and Speaches get the primary subtag (tr), NIM a locale (tr -> tr-TR, en -> en-US, others as written), vLLM none (the model decides).

Tests

tests/openai_realtime_stt.rs: OpenAI (URL, bearer, session message with turn_detection: null, 16 -> 24 kHz resampling, deltas, commit/completed, empty finalizes, error while a commit is open, unanswered commit and its late transcript, reconnect with session message and replay, disconnect, garbage, invalid_api_key, refusals, live-but-silent server, dead server, hang-up, config validation), NIM (URL, nested session message, audio byte for byte), Speaches (URL query, no session message, VAD turn as endpoint, no interims), vLLM (model, start commit, final commit, held audio, restart, timeout replaces the connection), and the provider contract suite for the OpenAI, NIM and vLLM dialects (Speaches sends no interims, which the suite requires). Live: BRISVOX_LIVE_TESTS=1 LOCAL_REALTIME_STT_URL=ws://127.0.0.1:9000 LOCAL_REALTIME_STT_DIALECT=nim [LOCAL_REALTIME_STT_MODEL=..] [LOCAL_REALTIME_STT_LANGUAGE=tr] cargo test -p brisvox-providers --features provider-openai-realtime-stt --test openai_realtime_stt live_ -- --nocapture prints partial counts, first-partial time and finalize latency for the two-utterance fixture.

Open points (all for a live run)

• OpenAI: the transcription URL (?model= vs ?intent=transcription), the committed event and the error codes on a too-short commit.
• NIM: whether a delta is an increment (delta_replaces = true if not), what commit does exactly (one completed per commit?), whether server endpointing interferes, WSL2 is unsupported by the Nemotron container (docs/providers/local-speech.md).
• Speaches: Blackwell compatibility of its CUDA 12.6 image, latency of a commit on CPU/GPU.
• vLLM: the first commit before session.update's model check, behaviour at realtime_max_tokens (the adapter restarts a generation that ends by itself).