Documentation: GenAI for Unreal
Welcome to the complete guide for the GenAI for Unreal plugin. This document provides all the necessary information to integrate, use, and master the plugin in your Unreal Engine projects.
Table of Contents
- 1. Quick Start: Your First AI Chat in 5 Minutes
- 2. Initial Project Setup
- 3. Getting API Keys
- 4. Authentication & Security
- 5. Core Concepts
- 6. Usage Guides & Examples
- 7. Building Long Conversations & Chat Context
- 8. Streaming
- 9. Structured Output
- 10. Text-to-Speech and Transcription
- 11. Image Generation
- 12. How to Run Tests
- 13. Quick Links & API Reference
- 14. Additional Notes
1. Quick Start: Your First AI Chat in 5 Minutes
Follow these minimal steps to get a basic chat completion working instantly.
- Install Plugin: Add "GenAI for Unreal" from the Fab.com marketplace and enable it in
Edit > Plugins
. - Add API Key: Go to
Project Settings > Plugins > GenAI Plugin
and enter your OpenAI API key. - Create Blueprint: Make a new Actor Blueprint. In the Event Graph, add the following nodes:
A simple setup to test chat completion. - Node Setup:
- Connect
Event Begin Play
to aRequest OpenAI Chat Completion
node. - For the
Settings
pin, add aMake Gen AI Chat Settings
node. - In the
Messages
array, add aMake Gen Chat Message
with the roleuser
and content "Tell me a joke." - Drag off the
OnComplete
event and add aPrint String
node, connecting theResponse
output to the string input.
- Connect
- Press Play: You will see an AI-generated joke printed to the screen.
2. Initial Project Setup
Follow these steps to get the plugin running in your project.
Adding the Plugin
- Acquire the GenAI for Unreal plugin from Fab.com. Once downloaded click on
Install to engine
from the epic's launcher. - Open your project and enable the "GenAI For Unreal" plugin from the
Edit > Plugins
menu. Restart the editor if prompted.
C++ Project Configuration (Optional)
To use the plugin's classes in your C++ code, you must add its modules to your project's build file.
- Open
YourProject.Build.cs
located in your project'sSource/YourProject
directory. - Add
"GenAI"
to your list of public dependencies. This gives you access to all the core functionalities.
using UnrealBuildTool;
public class YourProject : ModuleRules
{
public YourProject(ReadOnlyTargetRules Target) : base(Target)
{
PCHUsage = PCHUsageMode.UseExplicitOrSharedPCHs;
PublicDependencyModuleNames.AddRange(new string[] {
"Core",
"CoreUObject",
"Engine",
"InputCore",
"GenAI" // <-- ADD THIS MODULE
});
}
}
3. Getting API Keys
To use this plugin, you need an API key from at least one of the following services. An API key is a unique code that authenticates your requests with the AI provider's servers. You will be billed by the respective provider based on your usage.
- OpenAI (for GPT models, DALL-E, Whisper, TTS):
- Go to the OpenAI Platform and create an account.
- Navigate to the "API Keys" section in your user dashboard.
- Create a new secret key and copy it immediately. For security, OpenAI will not show you the key again.
- You may also need to set up a payment method to access certain models.
- Anthropic (for Claude models):
- Visit the Anthropic Console and sign up.
- Go to "Account Settings" and find the "API Keys" section.
- Generate and copy your new API key.
- DeepSeek / XAI / Google Gemini:
- Follow the specific sign-up and API key generation process on each provider's respective developer platform.
- DeepSeek Platform
- XAI (Grok)
- Google AI for Developers (Gemini)
Once you have your key(s), proceed to the Authentication & Security section below to learn how to add them to the plugin securely.
4. Authentication & Security
Setting API Keys:
Never hard-code API keys. The plugin provides a secure, centralized settings panel.
- In the Unreal Editor, navigate to Edit > Project Settings.
- Scroll down to the Plugins section on the left and click on GenAI Plugin.
Settings panel for API keys - Enter your API keys from each provider (OpenAI, Anthropic, etc.) into the corresponding fields.
- These keys are automatically saved to an encrypted binary file at
YourProject/Saved/Config/GenAI/secureconfig.bin
. - This file must not be committed to source control. Generally "Saved" folder is already part of
.gitignore
, if it is not add the following line to it to prevent leaking your keys:/Saved/Config/GenAI/secureconfig.bin
Using a Proxy Server (Optional, For Production):
Why use a proxy?
- Security: Your API keys never leave your server. The client application makes requests to your server, and your server securely attaches the API key before forwarding the request to the AI provider.
- Centralized Control: Manage keys, monitor usage, and control access for all users from a single backend.
- Custom Logic: Add logging, caching, or pre/post-processing logic on your server before returning data to the game client.
Retargetting the plugin to proxy:
Once your own backend server is running:
- Go to the GenAI Plugin settings in the Unreal Editor.
Retargetting the API requests to proxy server - Under the API Endpoint Management section for each provider, check the "Override" box.
- Enter the URL of your proxy endpoint. For example, for OpenAI, you might enter
https://api.yourgame.com/openai
.
Now, all API calls from the plugin to that provider will be routed through your server instead of directly to the AI provider's endpoint.
5. Core Concepts
Asynchronous by Design
All API calls in this plugin are asynchronous (non-blocking). This is essential to prevent your game from freezing while waiting for a response from the AI server. The workflow is always:
- You call a function to send a request (e.g.,
UGenOAIChat::SendChatRequest
). - You provide a callback delegate (in C++) or connect to an event pin (in Blueprints).
- Your game continues running without interruption.
- When the response arrives, your callback function or event is executed with the result.
Lifetime Management for Asynchronous Operations
When working with asynchronous API calls, a critical challenge in Unreal Engine is Object Lifetime. The core problem occurs when an Actor is destroyed while an API request is in-flight—the callback can attempt to access a "dangling pointer," causing crashes.
Imagine you start an image generation request from an Actor in your level. The request is sent, and the server begins processing it. Before the server can respond, you stop Play-In-Editor (PIE). The engine's Garbage Collector destroys all objects from that session, including your Actor. When the server response arrives, the plugin tries to deliver the result to your Actor, which no longer exists, causing a crash.
Blueprint Solution: Automatic Safety
Blueprint nodes are built on UCancellableAsyncAction
and handle lifetime automatically using TWeakObjectPtr
. The nodes check if the calling object is still valid before executing callbacks, preventing crashes. Key features:
- Automatic Cleanup: When objects are destroyed, weak pointers become invalid and callbacks are safely skipped.
- Explicit Cancellation: Use the Cancel node to terminate requests early and clean up resources.
Blueprint nodes automatically handle lifetime safety - No Extra Work Required: Blueprint developers can trust the async nodes to handle all safety automatically.
C++ Solution: Developer-Managed
When using static C++ functions, you must handle lifetime yourself. The static functions provide maximum flexibility but require proper safety patterns:
- Callback Safety: Always capture a
TWeakObjectPtr
to your calling object and checkWeakThis.IsValid()
at the start of every callback. - Request Cancellation: Store the returned
TSharedPtr<IHttpRequest>
and callCancelRequest()
in your object'sEndPlay
method to prevent wasting resources. - check() Macro: All static functions include
check(OnComplete.IsBound())
to catch unbound delegates in development builds.
// Essential C++ safety pattern
void AMyActor::RequestImageGeneration()
{
FGenOAIImageSettings ImageSettings;
ImageSettings.Prompt = TEXT("A beautiful sunset over mountains");
// Create weak pointer for safe callback capture
TWeakObjectPtr<AMyActor> WeakThis(this);
// Store request handle for potential cancellation
ActiveRequest = UGenOAIImageGeneration::SendImageGenerationRequest(ImageSettings,
FOnImageGenerationCompletionResponse::CreateLambda([WeakThis](const TArray<uint8>& ImageBytes, const FString& Error, bool bSuccess)
{
// CRITICAL: Always check validity first
if (!WeakThis.IsValid()) return;
AMyActor* StrongThis = WeakThis.Get();
StrongThis->OnImageReceived(ImageBytes, Error, bSuccess);
})
);
}
void AMyActor::EndPlay(const EEndPlayReason::Type EndPlayReason)
{
// Cancel any in-flight requests to prevent waste
if (ActiveRequest.IsValid() && ActiveRequest->GetStatus() == EHttpRequestStatus::Processing)
{
ActiveRequest->CancelRequest();
}
ActiveRequest.Reset();
Super::EndPlay(EndPlayReason);
}
Using Custom Models (Not in Enums)
AI providers release new models frequently. You don't have to wait for a plugin update to use them. All settings objects (e.g., FGenOAIChatSettings
) allow you to specify a model name as a simple string, which will always override the enum selection.
Blueprint Example:
In the "Make Gen OpenAI Chat Settings" node, simply type the new model's name (e.g., "gpt-5-exclusive-preview") into the Model (String)
input field. This will be used instead of the enum dropdown.

C++ Example:
#include "Models/OpenAI/GenOAIChat.h"
void AMyActor::SendCustomModelRequest()
{
FGenOAIChatSettings ChatSettings;
ChatSettings.Model = EOpenAIChatModel::Custom;
ChatSettings.CustomModelName = TEXT("gpt-5-exclusive-preview");
// ... complete the rest of the request
}
Additional Points to Note
- HTTP Timeouts: When streaming is not used, complex reasoning models (like DeepSeek Reasoner) can take longer to generate a response. If you experience frequent timeout errors, consider increasing the default HTTP timeout in your project's DefaultEngine.ini file:
[/Script/Engine.NetworkSettings] NetConnectionTimeout=180.0 [HTTP] HttpConnectionTimeout=180.0 HttpReceiveTimeout=180.0
- Audio Formats (TTS): The Text-to-Speech feature currently outputs audio in the PCM format. The plugin includes Blueprint utilities to convert this raw data into a playable USoundWave asset at runtime.
- API Key Security: The
secureconfig.bin
file is encrypted and non-portable by design. Do not attempt to copy it between projects or machines. Always add API keys through the Project Settings panel. Never commit this file to source control. - Platform Compatibility: The plugin uses standard engine modules (HTTP, Json) for maximum cross-platform compatibility. However, always test features on your specific target hardware, especially file-system-dependent operations like loading images for vision requests.
- Plugin Dependencies: This plugin relies only on core Unreal Engine modules. It does not introduce any external third-party libraries, ensuring a clean and stable integration.
6. Usage Guides & Examples
Finding the Example Assets
- Blueprint Examples: Example blueprints can be found either inside the plugin's content folder at
Engine/Plugins/GenAIForUnrealContent/ExampleBlueprints/
, or by downloading the example project hereBlueprint Examples - C++ Examples: The plugin has a seperate module for C++ examples called
GenAIExamples
.
Blueprint Usage Walkthrough
All Blueprint nodes are asynchronous (latent) and have a similar structure. Here is a walkthrough for a OpenAI multimodal chat request.
- Request OpenAI Chat Completion: This is the main node that initiates the API call.
- Settings: Use a "Make" node (e.g.,
Make Gen OpenAI Chat Settings
) to configure the request. Here you set the model, temperature, max tokens, and importantly, the messages. - OnComplete (Event): This event pin fires as soon as a response is received from the server. It provides the final Response string, an Error Message if something went wrong, and a `Success` boolean.
C++ Usage Walkthrough (OpenAI Chat)
The C++ workflow mirrors the Blueprint logic, using settings structs and response delegates.
#include "Data/OpenAI/GenOAIChatStructs.h"
#include "Models/OpenAI/GenOAIChat.h"
void ATOpenAIChatExample::TestTextChat()
{
UE_LOG(LogGenAI, Log, TEXT("--- Testing OpenAI Text-Only Chat ---"));
FGenOAIChatSettings ChatSettings;
ChatSettings.Model = EOpenAIChatModel::GPT_4_1_nano; // Use the latest text model
ChatSettings.Messages.Add(FGenChatMessage(TEXT("user"), TEXT("What is the distance between Earth and the Moon in kilometers?")));
ChatSettings.MaxTokens = 100;
UGenOAIChat::SendChatRequest(
ChatSettings,
FOnChatCompletionResponse::CreateLambda(
[this](const FString& Response, const FString& ErrorMessage, bool bSuccess)
{
if (!UTHelper::IsContextStillValid(this)) return;
if (bSuccess)
{
UE_LOG(LogGenAI, Log, TEXT("OpenAI Text Chat Response: %s"), *Response);
}
else
{
UE_LOG(LogGenAI, Error, TEXT("OpenAI Text Chat Error: %s"), *ErrorMessage);
}
})
);
}
7. Building Long Conversations & Chat Context
When building conversational AI systems, you often need to maintain context across multiple exchanges. Instead of sending isolated messages, you build up a conversation history that includes system prompts, user messages, and AI responses. This allows the AI to remember previous parts of the conversation and respond appropriately.
How Conversation Context Works
AI models are stateless - they don't remember previous conversations. To simulate memory, you must send the entire conversation history with each new request. The typical flow is:
- Initial Setup: Start with a system message that defines the AI's role and behavior
- User Input: Add the user's message to the conversation
- AI Response: Send the full context to get a response, then add that response to the history
- Continue: Repeat steps 2-3, always including the full conversation history
Conversation Structure Example
A typical conversation context grows like this:
Turn 1: [System], [User] → AI responds → [System], [User], [Assistant]
Turn 2: [System], [User], [Assistant], [User] → AI responds → [System], [User], [Assistant], [User], [Assistant]
Turn 3: [System], [User], [Assistant], [User], [Assistant], [User] → AI responds → ...and so on
C++ Implementation
Here's how to implement conversation context in C++, similar to the pattern shown in your example:
#include "Data/OpenAI/GenOAIChatStructs.h"
#include "Models/OpenAI/GenOAIChat.h"
class YOURGAME_API AConversationalNPC : public AActor
{
GENERATED_BODY()
private:
// Store the ongoing conversation context
TArray<FGenChatMessage> ConversationContext;
public:
void InitializeConversation();
void HandlePlayerMessage(const FString& PlayerInput);
void SendChatRequest();
};
void AConversationalNPC::InitializeConversation()
{
// Start with a system message that defines the AI's role
FGenChatMessage SystemMessage;
SystemMessage.Role = TEXT("system");
SystemMessage.Content = TEXT("You are a helpful NPC in a medieval village. You know about local events and can provide information to travelers. Respond in character.");
ConversationContext.Add(SystemMessage);
}
void AConversationalNPC::HandlePlayerMessage(const FString& PlayerInput)
{
// Add the player's message to our conversation history
FGenChatMessage UserMessage;
UserMessage.Role = TEXT("user");
UserMessage.Content = PlayerInput;
ConversationContext.Add(UserMessage);
// Send the request with full conversation context
SendChatRequest();
}
void AConversationalNPC::SendChatRequest()
{
FGenOAIChatSettings ChatSettings;
ChatSettings.Model = EOpenAIChatModel::GPT_4_1_mini;
ChatSettings.Messages = ConversationContext; // Send the entire conversation history
ChatSettings.MaxTokens = 150;
TWeakObjectPtr<AConversationalNPC> WeakThis(this);
UGenOAIChat::SendChatRequest(
ChatSettings,
FOnChatCompletionResponse::CreateLambda([WeakThis](const FString& Response, const FString& Error, bool bSuccess)
{
if (!WeakThis.IsValid()) return;
AConversationalNPC* StrongThis = WeakThis.Get();
if (bSuccess)
{
// Add the AI's response to our conversation context
FGenChatMessage AssistantMessage;
AssistantMessage.Role = TEXT("assistant");
AssistantMessage.Content = Response;
StrongThis->ConversationContext.Add(AssistantMessage);
// Display or process the response
UE_LOG(LogTemp, Log, TEXT("NPC Response: %s"), *Response);
}
else
{
UE_LOG(LogTemp, Error, TEXT("Chat Error: %s"), *Error);
}
})
);
}
Advanced Context Management
Context Length Management
As conversations grow longer, they can exceed the model's context window (token limit). Consider these strategies:
- Sliding Window: Keep only the most recent N messages while preserving the system message
- Summarization: Periodically summarize older parts of the conversation and replace them with a summary
- Important Message Retention: Keep key messages that contain important context while removing less important ones
void AConversationalNPC::TrimConversationContext(int32 MaxMessages)
{
if (ConversationContext.Num() <= MaxMessages) return;
// Always keep the system message (assumed to be first)
FGenChatMessage SystemMessage = ConversationContext[0];
// Keep only the most recent messages
TArray<FGenChatMessage> TrimmedContext;
TrimmedContext.Add(SystemMessage);
int32 StartIndex = FMath::Max(1, ConversationContext.Num() - MaxMessages + 1);
for (int32 i = StartIndex; i < ConversationContext.Num(); i++)
{
TrimmedContext.Add(ConversationContext[i]);
}
ConversationContext = TrimmedContext;
}
Multi-Character Conversations
For group conversations or multi-NPC scenarios, you can use custom roles or include character names in the content:
// Option 1: Use content formatting
FGenChatMessage NPCMessage;
NPCMessage.Role = TEXT("assistant");
NPCMessage.Content = FString::Printf(TEXT("[%s]: %s"), *NPCName, *Response);
// Option 2: Use custom role names (some models support this)
FGenChatMessage CustomMessage;
CustomMessage.Role = NPCName; // "Shopkeeper", "Guard", etc.
CustomMessage.Content = Response;
Best Practices for Conversation Context
- Clear System Messages: Define the AI's role, personality, and knowledge clearly in the system message
- Consistent Formatting: Maintain consistent message formatting throughout the conversation
- Context Awareness: Include relevant world state or character information in system messages
- Error Handling: Always check for API errors and have fallback responses ready
- Performance: Monitor token usage and implement context trimming for long conversations
- Memory Management: Consider persistent storage for conversations that should survive game sessions
8. Streaming
The plugin provides robust support for real-time streaming of AI responses, a crucial feature for creating dynamic and interactive experiences. Streaming allows you to receive the AI's response in small chunks (deltas) as they are generated, rather than waiting for the entire message to be completed. This is ideal for applications like live-narrated dialogue, real-time-generated character responses, or any feature where immediate feedback is essential.
How It Works
Streaming is implemented using Server-Sent Events (SSE), a standard protocol for pushing real-time updates from a server to a client. When a streaming request is made, the connection remains open, and the server sends a series of data packets. Each packet contains a small part of the ongoing response, such as a few words or a single sentence.
The plugin provides dedicated classes for handling streaming from different AI providers, such as UGenOAIChatStream
, UGenDSeekChatStream
, and UGenXAIChatStream
. These classes manage the HTTP connection and parse the incoming SSE events, broadcasting them through delegates that your application can subscribe to.
Blueprint Implementation
In Blueprints, you can use the dedicated "Request...Stream" nodes for each provider. For example, Request OpenAI Chat Stream
initiates a streaming request and provides an OnEvent
delegate that fires for each received chunk.
Here's an example of how to handle streaming events for OpenAI in Blueprints:

C++ Implementation
The C++ implementation offers more fine-grained control and follows a similar pattern. You use the static SendStreamChatRequest
function from the relevant stream class and provide a callback delegate to handle the events.
Here's an example of how to implement OpenAI streaming in C++:
#include "Models/OpenAI/GenOAIChatStream.h"
void AMyActor::TestStreamingChat()
{
FGenOAIChatSettings ChatSettings;
ChatSettings.Model = EOpenAIChatModel::GPT_4_1_nano;
ChatSettings.Messages.Add(FGenChatMessage(TEXT("user"), TEXT("Tell me a story about a brave knight.")));
ChatSettings.bStream = true; // Ensure streaming is enabled
// Send the request with a lambda to handle the stream events
ActiveRequestStreamingChat = UGenOAIChatStream::SendStreamChatRequest(
ChatSettings,
FOnOpenAIChatStreamResponse::CreateUObject(this, &AMyActor::OnStreamingChatEvent)
);
}
void AMyActor::OnStreamingChatEvent(const FGenOpenAIStreamEvent& StreamEvent)
{
if (StreamEvent.bSuccess)
{
if (StreamEvent.EventType == EOpenAIStreamEventType::ResponseOutputTextDelta)
{
// Log each piece of the story as it arrives
UE_LOG(LogGenAI, Log, TEXT("Story chunk: %s"), *StreamEvent.DeltaContent);
}
else if (StreamEvent.EventType == EOpenAIStreamEventType::ResponseCompleted)
{
UE_LOG(LogGenAI, Log, TEXT("Story finished!"));
ActiveRequestStreamingChat.Reset(); // Clean up the request handle
}
}
else
{
UE_LOG(LogGenAI, Error, TEXT("Streaming Error: %s"), *StreamEvent.ErrorMessage);
ActiveRequestStreamingChat.Reset(); // Clean up on error
}
}
Best Practices
- UI Updates: When updating UI elements like text boxes with streamed content, be mindful of performance. Appending text frequently can be expensive, so consider batching updates or using optimized text components.
- Cancellation: Always store the
TSharedPtr<IHttpRequest>
returned bySendStreamChatRequest
and callCancelRequest()
in your object'sEndPlay
orDestroy
method. This prevents orphaned requests and resource leaks if the actor is destroyed while a stream is active. - Error Handling: Implement logic to handle potential errors during the stream, such as network issues or API errors. The
OnEvent
delegate will provide detailed error messages. - Provider Differences: While the plugin standardizes the streaming interface, be aware of minor differences between AI providers. For example, DeepSeek offers a
ReasoningUpdate
event that provides "chain-of-thought" insights, which is unique to its models.
9. Structured Output
Structured Outputs is a feature that ensures the model will always generate responses that adhere to your supplied JSON Schema, so you don't need to worry about the model omitting a required key, or hallucinating an invalid enum value.

Some benefits of Structured Outputs include:
- Reliable type-safety: No need to validate or retry incorrectly formatted responses
- Explicit refusals: Safety-based model refusals are now programmatically detectable
- Simpler prompting: No need for strongly worded prompts to achieve consistent formatting
Schema JSON example
{
"type": "object",
"properties": {
"count": {
"type": "integer",
"description": "The total number of users."
},
"users": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": { "type": "string", "description": "The user's name." },
"heading_to": { "type": "string", "description": "The user's destination." }
},
"required": ["name", "role", "age", "heading_to"]
}
}
},
"required": ["count", "users"]
}
Using structured output in the C++:
FString MySchemaJson = R"(<insert your schema JSON here>)";
UGenAISchemaService::RequestStructuredOutput(
TEXT("Generate a list of users and their details"),
MySchemaJson,
[](const FString& Response, const FString& Error, bool Success) {
if (Success)
{
UE_LOG(LogTemp, Log, TEXT("Structured Output: %s"), *Response);
}
else
{
UE_LOG(LogTemp, Error, TEXT("Error: %s"), *Error);
}
}
);
In Blueprints:

10. Text-to-Speech and Transcription
Convert text to natural-sounding speech and speech back to text using OpenAI's advanced models. The plugin supports TTS with various voices and transcription with the Whisper model.
Text-to-Speech (TTS)
Easily bring your game to life by converting text into natural-sounding speech with OpenAI's advanced TTS models. The plugin supports a variety of voices and delivers high-quality 24kHz audio output ready for immediate use in Unreal Engine.
Use Cases
- Dynamic Dialogue: Convert NPC dialogue text into natural speech in real-time
- Narration: Create voiceovers for tutorials, story elements, or gameplay instructions
- Accessibility: Provide audio versions of in-game text for visually impaired players
- Localization: Generate voices for different languages without recording multiple voice actors
- Procedural Content: Add voice to dynamically generated content like quest descriptions or item details
Blueprint Implementation
Using TTS in Blueprints is straightforward. You can configure the voice, model, and input text, then convert the resulting audio data into a playable USoundWave
.

The key nodes are:
- Request OpenAI Text To Speech: Kicks off the TTS request. Use the
Make Gen OpenAI Text To Speech Settings
node to configure it. - Convert PCM Audio To Sound Wave: A helper node that takes the raw audio data from the API and turns it into a
USoundWave
. - Create Sound 2D: A standard Unreal node to play the generated sound in the world. Set it to "Auto Destroy" to avoid memory leaks.
C++ Implementation
Here's a concise C++ example for converting text to speech and playing the audio.
#include "Models/OpenAI/GenOAITextToSpeech.h"
#include "Data/OpenAI/GenOAIAudioStructs.h"
#include "Utilities/GenAIAudioUtils.h"
#include "Components/AudioComponent.h"
#include "Kismet/GameplayStatics.h"
void AMyActor::SpeakText(const FString& Text)
{
UE_LOG(LogTemp, Log, TEXT("Requesting speech for: '%s'"), *Text);
// 1. Configure the TTS request
FGenOAITextToSpeechSettings TTSSettings;
TTSSettings.InputText = Text;
TTSSettings.Model = EOpenAITTSModel::TTS_1;
TTSSettings.Voice = EGenAIVoice::Nova;
// 2. Send the request with a Lambda callback
UGenOAITextToSpeech::SendTextToSpeechRequest(TTSSettings,
FOnTTSCompletionResponse::CreateLambda([this](const TArray<uint8>& AudioData, const FString& ErrorMessage, bool bSuccess)
{
if (bSuccess && AudioData.Num() > 0)
{
UE_LOG(LogTemp, Log, TEXT("TTS successful! Received %d bytes of audio."), AudioData.Num());
// 3. Convert data to a playable sound wave
if (USoundWave* PlayableSound = UGenAIAudioUtils::ConvertPCMAudioToSoundWave(AudioData))
{
// 4. Play the sound using an auto-destroying audio component
UAudioComponent* AudioComponent = UGameplayStatics::CreateSound2D(this, PlayableSound);
if(AudioComponent)
{
AudioComponent->bAutoDestroy = true;
AudioComponent->Play();
}
}
}
else
{
UE_LOG(LogTemp, Error, TEXT("TTS request failed: %s"), *ErrorMessage);
}
})
);
}
Speech-to-Text (Transcription)
Convert any audio source into text using OpenAI's powerful Whisper model. This feature is perfect for implementing voice commands, transcribing dialogue, or enabling any voice-driven interaction in your game. The plugin can process audio directly from a file or from raw audio data in memory.
Blueprint Implementation
The Blueprint node for transcription is just as simple as the TTS node. You provide the audio data and it returns the transcribed text.

The key node is Request OpenAI Transcription From Data. As seen in the image in the TTS section, you can feed the audio data directly from the TTS output into the transcription node to perform a full round-trip conversion.
C++ Implementation
Here's how you can transcribe audio data from memory. This example assumes you already have the audio data in a TArray<uint8>
.
#include "Models/OpenAI/GenOAITranscription.h"
#include "Data/OpenAI/GenOAIAudioStructs.h"
void AMyActor::TranscribeAudio(const TArray<uint8>& AudioData)
{
if (AudioData.Num() == 0)
{
UE_LOG(LogTemp, Warning, TEXT("TranscribeAudio: Input audio data is empty."));
return;
}
UE_LOG(LogTemp, Log, TEXT("Starting transcription for %d bytes of audio..."), AudioData.Num());
// 1. Configure the transcription request
FGenOAITranscriptionSettings TranscriptionSettings;
TranscriptionSettings.Model = EOpenAITranscriptionModel::Whisper_1;
TranscriptionSettings.Language = TEXT("en"); // Optional: Specify language for better accuracy
// 2. Send the request with a Lambda callback
UGenOAITranscription::SendTranscriptionRequestFromData(AudioData, TranscriptionSettings,
FOnTranscriptionCompletionResponse::CreateLambda([](const FString& Transcript, const FString& ErrorMessage, bool bSuccess)
{
if (bSuccess)
{
UE_LOG(LogTemp, Log, TEXT("Transcription successful! Transcript: '%s'"), *Transcript);
}
else
{
UE_LOG(LogTemp, Error, TEXT("Transcription failed: %s"), *ErrorMessage);
}
})
);
}
11. Image Generation
The GenAI for Unreal plugin integrates OpenAI's powerful image generation models, allowing you to create stunning, high-quality images from simple text prompts. This opens up incredible possibilities for procedural art, dynamic content, and rapid prototyping directly inside Unreal Engine.
Use Cases
- Concept Art: Rapidly generate visual concepts for characters, environments, and props
- Placeholder Assets: Create temporary artwork while waiting for final art assets
- Procedural Content: Generate unique textures, landscapes, or decorative elements
- Marketing Materials: Create promotional images and screenshots for your game
- Prototyping: Quickly visualize game ideas and mechanics
- User-Generated Content: Allow players to create custom imagery within your game
Models and Capabilities
The plugin provides access to OpenAI's state-of-the-art image models:
- ✨ gpt-image-1: The latest and most advanced model, responsible for the popular "Ghibli trend." It excels at creating artistic, coherent, and high-fidelity images with remarkable prompt adherence.
- 🎨 DALL-E 3: A powerful model offering excellent image quality and detail. A great all-rounder for a variety of styles.
- ⚙️ DALL-E 2: A legacy model that is more cost-effective and supports generating multiple images in a single request, making it suitable for bulk generation or rapid iteration.
Blueprint Implementation
Blueprints make image generation incredibly accessible. You can easily configure a prompt, select your model, and save the resulting image to disk.

The workflow is simple:
- Request OpenAI Image Generation: This is the main node that sends your request to the API. Use
Make Gen OpenAI Image Generation Settings
to configure it. - Save Array to File: A utility node to save the raw image data (returned as a byte array) to a
.png
file. - Load Texture 2D from File: (Optional) After saving, you can immediately load the image back into the engine as a
UTexture2D
to display it on a material.
C++ Implementation
For more control, you can call the image generation functions directly from C++. Here is a clean, focused example.
#include "Models/OpenAI/GenOAIImageGeneration.h"
#include "Data/OpenAI/GenOAIImageStructs.h"
#include "Misc/FileHelper.h"
#include "Misc/Paths.h"
void AMyActor::GenerateImage(const FString& Prompt)
{
UE_LOG(LogTemp, Log, TEXT("Requesting image for prompt: '%s'"), *Prompt);
// 1. Configure the image generation settings
FGenOAIImageSettings ImageSettings;
ImageSettings.Prompt = Prompt;
ImageSettings.Model = EGenOAIImageModel::GPT_Image_1; // Use the latest model
ImageSettings.Size = EGenAIImageSize::Size1024x1024; // Standard high-res square
ImageSettings.Quality = EGenAIImageQuality::HD;
// 2. Send the request and handle the response in a Lambda
UGenOAIImageGeneration::SendImageGenerationRequest(ImageSettings,
FOnImageGenerationCompletionResponse::CreateLambda([this](const TArray<uint8>& ImageBytes, const FString& ErrorMessage, bool bSuccess)
{
if (bSuccess && ImageBytes.Num() > 0)
{
UE_LOG(LogTemp, Log, TEXT("Image generated successfully! Saving to disk..."));
// 3. Save the image data to a file
const FString FilePath = FPaths::ProjectSavedDir() / TEXT("AI_Generated_Image.png");
if (FFileHelper::SaveArrayToFile(ImageBytes, *FilePath))
{
UE_LOG(LogTemp, Log, TEXT("Image saved successfully to %s"), *FilePath);
}
else
{
UE_LOG(LogTemp, Error, TEXT("Failed to save image to disk."));
}
}
else
{
UE_LOG(LogTemp, Error, TEXT("Image generation failed: %s"), *ErrorMessage);
}
})
);
}
// Don't forget to cancel any active requests when the actor is destroyed!
void AMyActor::EndPlay(const EEndPlayReason::Type EndPlayReason)
{
if (ActiveRequest.IsValid() && ActiveRequest->GetStatus() == EHttpRequestStatus::Processing)
{
ActiveRequest->CancelRequest();
}
ActiveRequest.Reset();
Super::EndPlay(EndPlayReason);
}
12. How to Run Tests
Disclaimer
Running this test will incur costs, as it makes API requests to all available model combinations across different organisations.
The plugin includes a suite of automation tests to verify that all API integrations are working correctly. To run them:

- Ensure you have set valid API keys in the plugin settings for the providers you wish to test.
- In the Unreal Editor, go to Window > Session Frontend.
- In the "Automation" tab, you will see a list of available tests.
- Find the tests prefixed with
GenAI.
(e.g.,GenAI.OpenAI.Chat
). - Check the boxes for the tests you want to run, and press the "Start Tests" button.
The tests will run and report their status. This is a great way to confirm your setup is correct or to debug integration issues.
13. Quick Links & API Reference
- OpenAI API Documentation
- Anthropic API Documentation
- XAI API Documentation
- Google Gemini API Documentation
- DeepSeek API Documentation
14. Additional Notes
- Unity Build Compatible: The codebase is fully compatible with Unreal's Unity Build and non-unity build workflows, ensuring it can be integrated into any project's compilation pipeline.
This documentation is based on the GenAI for Unreal plugin. Specific function names, struct members, and Blueprint node names may vary slightly based on the final plugin version. Always refer to the plugin's source code or example content for the most accurate implementation details.