Skip to content

Realtime API

realtime

Realtime voice: a provider-neutral seam plus an OpenAI adapter.

The browser holds the audio connection to the provider; the server mints short-lived credentials, reads relayed events and writes client events through a :class:RealtimeProvider. See docs/concepts/realtime.md.

OpenAIRealtime

OpenAIRealtime(
    api_key: str,
    *,
    http: AsyncClient | None = None,
    ttl_s: int = 60,
)

:class:~orqest.realtime.RealtimeProvider for OpenAI Realtime over WebRTC.

http is used as given and never closed here; without one, each call opens and closes its own client. ttl_s is the client secret's lifetime (the provider allows 10–7200 s); it only has to outlive connecting.

Source code in orqest/realtime/openai.py
def __init__(
    self,
    api_key: str,
    *,
    http: httpx.AsyncClient | None = None,
    ttl_s: int = 60,
) -> None:
    if not 10 <= ttl_s <= 7200:
        raise ValueError(f"ttl_s must be between 10 and 7200 seconds, got {ttl_s}")
    self._api_key = api_key
    self._http = http
    self._ttl_s = ttl_s
session_payload
session_payload(config: SessionConfig) -> dict[str, Any]

The session object sent when minting a client secret.

Source code in orqest/realtime/openai.py
def session_payload(self, config: SessionConfig) -> dict[str, Any]:
    """The ``session`` object sent when minting a client secret."""
    transcription: dict[str, Any] = {"model": config.transcription_model}
    if config.language is not None:
        transcription["language"] = config.language
    audio_input: dict[str, Any] = {
        "transcription": transcription,
        "turn_detection": _turn_detection(config.turn_detection),
    }
    if config.noise_reduction is not None:
        audio_input["noise_reduction"] = {"type": config.noise_reduction}
    payload: dict[str, Any] = {
        "type": "realtime",
        "model": config.model,
        "instructions": config.instructions,
        "tools": [
            {
                "type": "function",
                "name": tool.name,
                "description": tool.description,
                "parameters": tool.parameters,
            }
            for tool in config.tools
        ],
        "tracing": None,
        "audio": {"input": audio_input, "output": {"voice": config.voice}},
    }
    if config.reasoning_effort is not None:
        payload["reasoning"] = {"effort": config.reasoning_effort}
    if config.max_output_tokens is not None:
        payload["max_output_tokens"] = config.max_output_tokens
    return payload

AssistantTranscript

What the model said, from its own output transcript.

ClientSession

A short-lived credential for the browser.

The browser posts its SDP offer to connect_url with Authorization: Bearer <client_secret> before expires_at. The long-lived key never leaves the server.

PlaybackCleared

The response's audio still to be played was dropped (the user talked over it, or the client cut it off): the user didn't hear all of it.

PlaybackStopped

The response's audio has all been played out: the user has heard all of it.

Comes after the response's ResponseDone, which marks the end of generation, not of playback. Not every transport reports playback.

ProviderError

The provider reported an error on the event channel.

RealtimeError

A provider call made by the server failed or was refused.

RealtimeProvider

One realtime voice provider, seen from the server.

The client-event builders return provider wire events for the browser to send on its event channel; the server never holds the audio.

create_client_session async
create_client_session(
    config: SessionConfig,
) -> ClientSession

Mint a short-lived credential carrying config.

Source code in orqest/realtime/types.py
async def create_client_session(self, config: SessionConfig) -> ClientSession:
    """Mint a short-lived credential carrying ``config``."""
    ...
hangup async
hangup(call_id: str) -> None

End a call the browser opened, whether or not it is still there.

Source code in orqest/realtime/types.py
async def hangup(self, call_id: str) -> None:
    """End a call the browser opened, whether or not it is still there."""
    ...
parse_event
parse_event(raw: dict[str, Any]) -> RealtimeEvent | None

Translate one provider event. Unknown or malformed events give None; this never raises.

Source code in orqest/realtime/types.py
def parse_event(self, raw: dict[str, Any]) -> RealtimeEvent | None:
    """Translate one provider event. Unknown or malformed events give
    ``None``; this never raises."""
    ...
scripted_line
scripted_line(text: str, *, line_id: str) -> dict[str, Any]

The client event that makes the model say exactly text.

Source code in orqest/realtime/types.py
def scripted_line(self, text: str, *, line_id: str) -> dict[str, Any]:
    """The client event that makes the model say exactly ``text``."""
    ...
tool_result
tool_result(
    call_id: str, output: str, *, respond: bool = True
) -> list[dict[str, Any]]

The client events that hand a function result back and, with respond, let the model continue. respond=False only records the result: the model says nothing more about it until the next turn (for a result it has already acknowledged).

Source code in orqest/realtime/types.py
def tool_result(
    self, call_id: str, output: str, *, respond: bool = True
) -> list[dict[str, Any]]:
    """The client events that hand a function result back and, with
    ``respond``, let the model continue. ``respond=False`` only records
    the result: the model says nothing more about it until the next turn
    (for a result it has already acknowledged)."""
    ...
context_update
context_update(instructions: str) -> dict[str, Any]

The client event that replaces the session's instructions.

Source code in orqest/realtime/types.py
def context_update(self, instructions: str) -> dict[str, Any]:
    """The client event that replaces the session's instructions."""
    ...
cost_usd
cost_usd(usage: RealtimeUsage, model: str) -> float

Price usage for model. Raises ValueError for a model without known prices.

Source code in orqest/realtime/types.py
def cost_usd(self, usage: RealtimeUsage, model: str) -> float:
    """Price ``usage`` for ``model``. Raises ``ValueError`` for a model
    without known prices."""
    ...

RealtimeUsage

Tokens used by one response.

audio_in and text_in exclude cached input tokens; cached_in counts those, since they are priced separately.

ResponseDone

A response ended.

status says how: completed, cancelled (for example the user talked over it), failed or incomplete (cut short, or a status the adapter didn't recognise). cancelled is kept as a shorthand for status == "cancelled"; give either one, and both must agree. line_id is set when the response was a scripted line, so its transcript can be checked against the words asked for.

SessionConfig

Everything a session starts with. language=None leaves transcription free to recognise any language.

The last three fields are optional tuning, sent only when set (otherwise the provider's defaults apply): how much the model reasons before answering (less is faster), a cap on output tokens per response, and microphone noise reduction for a close (near_field, a phone or headset) or distant (far_field, a laptop or room) microphone.

SpeechStarted

The provider heard the user start speaking.

item_id is the user item this speech becomes, the same id its UserTranscript carries, so a transcript can be tied to when its speech began (transcripts arrive late, often after later speech started). None when the provider didn't say.

SpeechStopped

The provider heard the user stop speaking; item_id as for :class:SpeechStarted.

ToolCall

The model asked for a function call; arguments is already parsed.

ToolDef

A function the model may call. parameters is a JSON Schema object.

TurnDetection

How the provider decides that the user started and stopped talking.

server is a voice-activity detector tuned by threshold (0–1, higher needs louder audio), prefix_ms (audio kept from before speech began) and silence_ms (quiet that ends a turn). semantic lets the model judge the end of a turn, tuned by eagerness. interrupt_on_speech_only asks the adapter to keep a short non-speech sound, such as a cough, from cancelling a response in progress, using whatever the provider offers for that.

UserTranscript

What the user said, as transcribed. Deltas have final=False.

verbatim_match

verbatim_match(expected: str, spoken: str) -> bool

True when spoken says the same words as expected.

Source code in orqest/realtime/verbatim.py
def verbatim_match(expected: str, spoken: str) -> bool:
    """True when ``spoken`` says the same words as ``expected``."""
    return _words(expected) == _words(spoken)