Realtime Voice¶
orqest.realtime is a small, provider-neutral seam for speech-to-speech sessions, plus one adapter, OpenAIRealtime. It holds no audio and runs no loop. It describes the traffic around a voice call so that an application can keep the provider behind one interface.
The shape it assumes:
- The browser holds the call. It sends microphone audio to the provider over WebRTC and plays the reply. Provider events arrive on a data channel.
- The server holds the key. It mints a short-lived credential (
create_client_session), reads the events the browser relays (parse_event), writes the client events the browser sends on (scripted_line,tool_result,context_update), hangs up a call that outstays its welcome (hangup) and prices usage (cost_usd).
The long-lived API key never reaches the browser. The browser posts its SDP offer to ClientSession.connect_url with Authorization: Bearer <client_secret>, and the secret expires after ttl_s seconds (60 by default). It only has to outlive connecting.
Quickstart¶
This runs without a network: the adapter takes any httpx.AsyncClient, here one with a fake transport.
import asyncio
import httpx
from orqest.realtime import (
OpenAIRealtime,
RealtimeUsage,
SessionConfig,
ToolDef,
TurnDetection,
verbatim_match,
)
def fake_provider(request: httpx.Request) -> httpx.Response:
# Stands in for POST /v1/realtime/client_secrets.
return httpx.Response(200, json={"value": "ek_example", "expires_at": 1_794_220_000})
async def main() -> None:
http = httpx.AsyncClient(transport=httpx.MockTransport(fake_provider))
provider = OpenAIRealtime("sk-your-key", http=http)
config = SessionConfig(
instructions="You are a concise voice assistant. Answer in the language you hear.",
tools=[
ToolDef(
name="get_time",
description="The current local time.",
parameters={"type": "object", "properties": {}},
)
],
voice="marin",
model="gpt-realtime-2.1",
transcription_model="gpt-4o-mini-transcribe",
turn_detection=TurnDetection(), # server VAD, 300 ms silence, threshold 0.6
)
session = await provider.create_client_session(config)
print(session.client_secret, session.connect_url) # hand these to the browser
# The browser relays provider events; the seam turns them into typed events.
event = provider.parse_event(
{
"type": "conversation.item.input_audio_transcription.completed",
"item_id": "item_1",
"transcript": "What time is it?",
}
)
print(event) # type='user_transcript' text='What time is it?' final=True item_id='item_1'
# Code-written words, spoken exactly; the browser sends this on the data channel.
line = provider.scripted_line("It's 07:00.", line_id="time-1")
print(line["response"]["metadata"]) # {'line_id': 'time-1'}
# Check the model's own transcript of that response against the words asked for.
print(verbatim_match("It's 07:00.", "it's seven o'clock")) # True
usage = RealtimeUsage(audio_in=600, audio_out=1200) # about a minute of talk both ways
print(round(provider.cost_usd(usage, "gpt-realtime-2.1"), 4)) # 0.096
await http.aclose()
asyncio.run(main())
The seam¶
| Name | What it is |
|---|---|
TurnDetection |
How the provider decides a turn started and ended: mode (server or semantic), silence_ms 300, threshold 0.6, prefix_ms 300, eagerness auto, interrupt_on_speech_only true. Frozen. |
ToolDef |
A function the model may call: name, description, JSON Schema parameters. |
SessionConfig |
instructions, tools, voice, model, transcription_model, turn_detection, and language (None leaves transcription free to recognise any language). Optional tuning, sent only when set (the provider's defaults apply otherwise): reasoning_effort (minimal, low, medium or high; less is faster), max_output_tokens (a cap per response) and noise_reduction (near_field for a phone or headset, far_field for a laptop or room). |
ClientSession |
provider, client_secret, expires_at, connect_url, model. |
RealtimeUsage |
Tokens for one response: audio_in, audio_out, text_in, text_out, cached_in. audio_in and text_in exclude cached input, which cached_in counts. Usages add with +. |
RealtimeEvent |
A union discriminated on type, listed below. |
RealtimeProvider |
The protocol an adapter implements. It is runtime-checkable. |
RealtimeError |
A server-side provider call failed or was refused. Messages never carry the key. |
verbatim_match |
Did the spoken transcript say the scripted words? |
Events:
| Event | Fields | Meaning |
|---|---|---|
UserTranscript |
text, final, item_id |
What the user said. Deltas have final=False. |
AssistantTranscript |
text, final, response_id |
What the model said, from its own output transcript. |
ToolCall |
call_id, name, arguments |
A function call, with arguments already parsed into a dict. |
SpeechStarted / SpeechStopped |
item_id |
The provider heard the user start or stop. item_id is the user item the speech becomes, the same id its UserTranscript carries, or None when the provider didn't say. Transcripts arrive late, often after later speech began, so use it to tie a transcript to when its speech started (for example, to accept only a reply begun after a line finished playing). |
PlaybackStopped |
response_id |
All of the response's audio has been played out, so the user has heard it all. It comes after ResponseDone, which marks the end of generation, not of playback. Use it before acting on what the user says in reply to a line they must hear in full. Not every transport reports playback: OpenAI does over WebRTC and SIP only. |
PlaybackCleared |
response_id |
The rest of the response's audio was dropped, because the user talked over it or the client cut it off. The user didn't hear all of it. |
ResponseDone |
response_id, usage, status, cancelled, line_id |
A response ended. status is completed, cancelled (cut off, for example by the user talking over it), failed or incomplete. cancelled is shorthand for status == "cancelled": give either one when constructing, and they must agree. line_id is set for a scripted line. |
ProviderError |
message |
The provider reported an error on the event channel. |
parse_event never raises. An unknown event type, or a known one with missing or mistyped fields, gives None and a DEBUG log. That includes a function call whose arguments aren't a JSON object. A response.done keeps its usage whatever else is odd about it, since a spend cap depends on it: a missing, null or non-numeric count reads as 0, cached tokens never push a count below 0, and odd metadata is ignored. Only the provider's own completed makes a response completed; a missing or unrecognised status reads as incomplete.
ResponseDone.line_id is an addition to the original sketch of the seam. Without it, a server can't tell which response was the scripted line it asked for, so it can't check that line's transcript.
Scripted lines¶
Some words have to be exact: amounts, dates, read-backs before an action. scripted_line(text, line_id=...) builds an out-of-band response.create. It sets conversation: "none" and an empty input, so the model sees only the instruction ("Say exactly the following, word for word, and nothing else: …"). It also sets tool_choice: "none" (no tool calls), reasoning: {effort: "minimal"} (there is nothing to decide, only words to say) and metadata: {"line_id": ...}. The client events reference lists tool_choice and reasoning on response.create (checked 2026-11-09).
The provider has no guaranteed verbatim mode: compliance comes from the instruction. So check every line. Collect the AssistantTranscript events for the response whose ResponseDone.line_id matches, and compare them with verbatim_match.
The model's transcript is its own record of what it generated, not proof of what the speaker played. An out-of-band response is also not added to the conversation. If the model needs to remember the line (for example, to understand a "yes" to a read-back), the application has to tell it, for instance through context_update.
verbatim_match(expected, spoken) compares the two word sequences after normalising:
- case, whitespace, punctuation, curly quotes, dashes and accents;
- apostrophes (
o'clock,I'll,'s avonds); - numbers 0–100 as digits or English or Dutch words (
twenty-one,eenentwintig,tweeëntwintig,a hundred); - clock times (
07:00is7,07:30is7 30,19.00 uuris19), witho'clockanduurdropped after any number; - currency symbols, which are ignored (
€12is12); - a minus sign before a digit, which is the word "minus" (
-5is "minus five", and isn't5).
Decimals and thousands stay as written: 0.05 is neither 0.5 nor 0,05, and 1,000 isn't 1000. Leading zeros are dropped only from a standalone whole number (07 is 7).
It doesn't guess between 12- and 24-hour times (19:00 against "seven"), and it doesn't read "half past" or "half acht". Those come back as mismatches. That's the safe answer for a check whose failures go to a person. It is symmetric and reflexive, and property tests hold it to both.
OpenAI adapter¶
OpenAIRealtime(api_key, *, http=None, ttl_s=60). With http, that client is used and never closed. Without it, each call opens and closes its own client, with a 30 s timeout.
| Seam | OpenAI |
|---|---|
create_client_session |
POST /v1/realtime/client_secrets with expires_after: {anchor: "created_at", seconds: ttl_s} and a realtime session: model, instructions, function tools, tracing: null (off), audio.input.transcription (model, plus language only when set), audio.input.turn_detection, audio.output.voice, and when set reasoning.effort, max_output_tokens and audio.input.noise_reduction.type. The response's value is the secret, and expires_at is in Unix seconds. connect_url is https://api.openai.com/v1/realtime/calls. |
hangup(call_id) |
POST /v1/realtime/calls/{call_id}/hangup. A 404 counts as already closed. The id must be 1–100 letters, digits, _ or -, since it comes from the browser (the Location header of its connect call). |
parse_event |
conversation.item.input_audio_transcription.delta / .completed, response.output_audio_transcript.delta / .done, response.function_call_arguments.done, input_audio_buffer.speech_started / .speech_stopped (item_id), output_audio_buffer.stopped / .cleared (response_id), response.done (status, usage, metadata.line_id) and error. |
tool_result |
conversation.item.create with a function_call_output item, then response.create. With respond=False the response.create is left out: the result is recorded and the model says nothing more about it until the next turn, for a result the model has already acknowledged. |
context_update |
session.update with {type: "realtime", instructions}. |
cost_usd |
The price table below. Any other model raises ValueError, so a cost cap never prices a call at zero. |
TurnDetection maps like this:
serverbecomesserver_vadwiththreshold,prefix_padding_ms,silence_duration_ms,create_response: trueandinterrupt_response: true.semanticbecomessemantic_vadwitheagerness,create_response: trueandinterrupt_response: true.interrupt_on_speech_only: OpenAI has no setting that tells a short non-speech sound from speech. The VAD guide's only lever for noise is the threshold ("a higher threshold will require louder audio to activate the model").interrupt_response: falsestops all interruption, speech included. So forservermode the adapter raises the threshold by 0.1, capped at 1.0 (0.6 becomes 0.7). Semantic VAD has no threshold, so this flag changes nothing there. Whether a cough still cuts off a reply at 0.7 hasn't been measured; that needs a trial by ear.
Prices, in USD per 1M tokens:
| Model | Audio in | Audio out | Text in | Text out | Cached in |
|---|---|---|---|---|---|
gpt-realtime-2.1 |
32 | 64 | 4 | 24 | 0.40 |
gpt-realtime-2.1-mini |
10 | 20 | 0.60 | 2.40 | 0.30 |
The provider prices cached audio and cached text separately. gpt-realtime-2.1-mini's cached text is $0.06. RealtimeUsage has one cached figure, so the table uses the higher price, and the estimate errs high. Audio runs at about 10 tokens a second in and 20 out, so a minute of both sides talking costs about $0.096 on gpt-realtime-2.1. Input transcription (about $0.003 a minute for gpt-4o-mini-transcribe) is billed separately and isn't in RealtimeUsage.
Checked against the provider's docs on 2026-11-09¶
The OpenAI docs live at developers.openai.com/api/docs. Checked: the client secrets reference (request body, a TTL of 10–7200 s, the response's value and expires_at, and the server_vad and semantic_vad fields with their defaults), the server events reference (event names and fields, response.done usage details and status values; on 2026-11-10, item_id on input_audio_buffer.speech_started and .speech_stopped, "the ID of the user message item that will be created"), the client events reference (response.create takes tool_choice and reasoning.effort), the conversations guide (out-of-band responses, function call output, session.update, and the 60-minute session limit), the VAD guide (no setting for non-speech sounds) and pricing.
output_audio_buffer.stopped and output_audio_buffer.cleared (WebRTC/SIP only, each with event_id and response_id) were checked on 2026-11-10 against the server events reference and the openai Python SDK 3.46.0 types (OutputAudioBufferStopped, OutputAudioBufferCleared). stopped comes once the output buffer is fully drained, after response.done. cleared comes when the user interrupts in VAD mode, or when the client sends output_audio_buffer.clear.
The hang-up endpoint and the WebRTC connect URL come from a working spike on 2026-11-08 (POST /v1/realtime/calls with the client secret, and the call id from the Location header).
What it doesn't do¶
- No audio, no loop. The seam doesn't connect, stream or reconnect. The browser does that, and the application decides what each event means.
- A call can outlive its credential. The secret only gates starting a call. Once connected, a call runs until someone hangs up, or until the provider's 60-minute limit. An application that caps session length must call
hangupitself. - Retention is the provider's.
tracingis off, but OpenAI keeps abuse-monitoring logs for/v1/realtimefor up to 30 days unless the account has zero retention. - Transcripts aren't proof. Both transcripts are model output.
verbatim_matchchecks words, not sound.