How to Use the OpenAI Realtime API in Unreal Engine 5
For years, putting "Voice AI" in a game meant chaining three different systems together: Speech-to-Text (STT) to understand the player, an LLM to generate the text response, and Text-to-Speech (TTS) to speak it back. This pipeline inherently creates latency — often 2 to 4 seconds of awkward silence between the player speaking and the NPC replying.
Voice-to-voice models solve this. OpenAI's Realtime API (and Google's Gemini Live) use a persistent WebSocket connection. The AI listens to the audio stream directly and generates audio back directly. The latency drops to milliseconds, allowing for natural, fluid, and fully interruptible conversations.
The Challenge of Realtime Voice in UE5
Implementing a persistent WebSocket connection that streams raw PCM audio chunks in and out of Unreal Engine 5 is notoriously difficult. You have to manage thread safety, buffer underruns, microphone permissions, and "barge-in" logic (handling what happens when a player interrupts an NPC mid-sentence).
The Solution: Realtime Conversational AI Components
The GenAI for Unreal plugin handles the entire WebSocket audio lifecycle natively. It includes a custom Realtime Audio Capture Component that game designers can drop onto any Blueprint Actor to handle microphone input and feed it directly to the AI.
1. Handling Interruptions (Barge-in) and Phantom Audio
When a player interrupts an NPC, two things must happen instantly:
- The NPC's current audio playback must stop.
- The remaining queued audio data from the AI must be cleared, so the NPC doesn't suddenly spit out the end of a canceled sentence later.
The plugin manages this automatically via a Clear Input Buffer function on the Realtime Service, preventing "phantom audio" bugs that plague custom implementations.
2. Push-to-Talk vs. Voice Activity Detection (VAD)
The plugin gives you two ways to capture player speech:
Push-to-Talk (PTT): The traditional gaming approach. The player holds a key to talk. When released, the audio is sent. In Blueprints, you simply call Start Recording on key press, and Stop Recording on release using the Realtime Audio Capture Component.
Server-Side VAD (Continuous Mic): For a truly hands-free experience, you can leave the mic open. The plugin supports OpenAI's Server-Side VAD and Semantic VAD. It constantly streams 100ms chunks of audio to the server. The AI intelligently detects when the player has finished speaking and formulates a response on its own.
Blueprint Implementation Example
Here is how a game designer can wire up a Push-to-Talk interaction in UE5 without writing any C++:
// On Key Press (Push-to-Talk)
AI_AudioPlayerComponent -> Stop
RealtimeService -> Clear Input Buffer
RealtimeAudioRecorder -> Start Recording
// On Key Release
RealtimeAudioRecorder -> Stop Recording
// Handling the AI's Response Event
Event OnAudioReceived(AudioChunk)
QueueAudioToProceduralWave(AudioChunk, NPC_ProceduralSoundWave)
Crucial Shipping Tip for Mac Developers
If you are packaging your UE5 game for Mac, you might encounter issues where the microphone isn't picked up or the WebSocket connection instantly fails (Connection Status 0). This is a known UE5 SSL certificate quirk.
The Fix: Copy cacert.pem from your Engine's Engine/Content/Certificates/ThirdParty directory into your project's Content/Certificates directory. Then, add that Certificates folder to Additional Non-Asset Directories To Package in your Project Settings.