openai-realtime-voice

OpenAI Realtime voice

Use this skill when the deliverable is a live spoken interaction: a browser voice assistant, phone-like agent, low-latency speech-to-speech support flow, guided interview, tutoring conversation, live kiosk, or realtime voice UI that listens, reasons, speaks, interrupts, and may call tools.

Do not use this skill for batch speech-to-text, offline TTS, voiceover generation, audio editing, dubbing, music, or file-based audio tasks. Those belong to the separate OpenAI audio/request APIs or other speech/audio skills. The boundary is simple: if a persistent realtime session and turn-taking behavior are part of the product, use this skill; if the job is "turn this file/text into text/audio," do not.

All OpenAI API facts below were verified against first-party OpenAI documentation on 2026-07-10. Re-check model identifiers, prices, session limits, voices, API object fields, data retention, and policy requirements before production release because Realtime surfaces change quickly.

Source-grounded facts to carry into every design

Documented facts:

• OpenAI positions Realtime sessions as best for live audio that needs low latency; request-based audio APIs are best for files, bounded requests, or generated speech that does not need a live session. Source: Realtime and audio overview.
• A voice-agent session is for applications where the model should respond to the user, call tools, and manage conversation state on /v1/realtime. Translation and transcription sessions are separate Realtime session types. Source: Realtime and audio overview.
• OpenAI's voice-agent guide distinguishes speech-to-speech live audio sessions from chained STT -> text reasoning -> TTS pipelines. Live audio is the starting point for barge-in, low first-audio latency, natural turn taking, and realtime tool use. Source: Voice agents.
• The documented browser path is WebRTC, preferably via Agents SDK helpers such as RealtimeAgent and RealtimeSession; server-to-server integrations use WebSocket. Source: Voice agents, WebRTC guide, WebSocket guide.
• A Realtime speech-to-speech session contains a Session object, a Conversation, and Responses. The Session controls model, voice, and configuration; the Conversation contains user and model Items; Responses are model-generated audio or text Items. Source: Realtime conversations.
• After the server sends session.created, the client can use session.update. Most session properties can be updated at any time, but the voice cannot be changed after the model has responded with audio once in the session. The documented maximum Realtime session duration is 60 minutes. Source: Realtime conversations.
• For WebSocket audio, the client manually sends base64-encoded audio through input-audio-buffer events; WebRTC handles much of the media transport. Source: Realtime conversations.
• VAD is enabled by default in speech-to-speech Realtime sessions and can be configured at session.audio.input.turn_detection. Documented VAD modes are server_vad and semantic_vad; server VAD can tune threshold, prefix padding, and silence duration. Source: Voice activity detection.
• Realtime sessions can use function tools executed by the application, MCP tools exposed by a remote MCP server, and built-in connectors. Tools may be attached at the session level or for one response. Source: Realtime with tools.
• Realtime voice-agent sessions accrue input and output tokens across text, audio, and image modalities; translation and transcription streaming sessions are duration-billed. Later conversational turns cost more because the whole server-side conversation is input to each Response. OpenAI documents audio-token accounting as 1 token per 100 ms for user-message audio and 1 token per 50 ms for assistant-message audio. Source: Managing costs.
• Input-audio transcription is a separate ASR process from the Realtime model's native audio understanding, and OpenAI warns the transcript may diverge from the model's interpretation. Treat transcripts as observability and accessibility artifacts, not as the exact reasoning input. Source: Realtime API reference.
• OpenAI states that data sent to the API is not used to train or improve models unless the organization explicitly opts in. Source: Data controls in the OpenAI platform.
• OpenAI's usage policies and safety best practices still apply to Realtime voice experiences, including harmful-content, impersonation, privacy, high-impact decision, and user-protection constraints. Sources: Usage policies, Safety best practices.