Muddy Terrain Games

Products Careers Support

Documentation: GenAI for Unreal

Welcome to the complete guide for the GenAI for Unreal plugin. This document provides all the necessary information to integrate, use, and master the plugin in your Unreal Engine projects.

Table of Contents

1. Quick Start: Your First AI Chat in 5 Minutes

Follow these minimal steps to get a basic chat completion working instantly.

  1. Install Plugin: Add "GenAI for Unreal" from the Fab.com marketplace and enable it in Edit > Plugins.
  2. Add API Key: Go to Project Settings > Plugins > GenAI Plugin and enter your OpenAI API key.
  3. Create Blueprint: Make a new Actor Blueprint. In the Event Graph, add the following nodes:
    Quick Start Blueprint Example
    A simple setup to test chat completion.
  4. Node Setup:
    • Connect Event Begin Play to a Request OpenAI Chat Completion node.
    • For the Settings pin, add a Make Gen AI Chat Settings node.
    • In the Messages array, add a Make Gen Chat Message with the role user and content "Tell me a joke."
    • Drag off the OnComplete event and add a Print String node, connecting the Response output to the string input.
  5. Press Play: You will see an AI-generated joke printed to the screen.

2. Initial Project Setup

Follow these steps to get the plugin running in your project.

Adding the Plugin

  1. Acquire the GenAI for Unreal plugin from Fab.com. Once downloaded click on Install to engine from the epic's launcher.
  2. Open your project and enable the "GenAI For Unreal" plugin from the Edit > Plugins menu. Restart the editor if prompted.

C++ Project Configuration (Optional)

To use the plugin's classes in your C++ code, you must add its modules to your project's build file.

  1. Open YourProject.Build.cs located in your project's Source/YourProject directory.
  2. Add "GenAI" to your list of public dependencies. This gives you access to all the core functionalities.
using UnrealBuildTool;

public class YourProject : ModuleRules
{
    public YourProject(ReadOnlyTargetRules Target) : base(Target)
    {
        PCHUsage = PCHUsageMode.UseExplicitOrSharedPCHs;
    
        PublicDependencyModuleNames.AddRange(new string[] { 
            "Core", 
            "CoreUObject", 
            "Engine", 
            "InputCore", 
            "GenAI"  // <-- ADD THIS MODULE
        });
    }
}

3. Getting API Keys

To use this plugin, you need an API key from at least one of the following services. An API key is a unique code that authenticates your requests with the AI provider's servers. You will be billed by the respective provider based on your usage.

  • OpenAI (for GPT models, DALL-E, Whisper, TTS):
    1. Go to the OpenAI Platform and create an account.
    2. Navigate to the "API Keys" section in your user dashboard.
    3. Create a new secret key and copy it immediately. For security, OpenAI will not show you the key again.
    4. You may also need to set up a payment method to access certain models.
  • Anthropic (for Claude models):
    1. Visit the Anthropic Console and sign up.
    2. Go to "Account Settings" and find the "API Keys" section.
    3. Generate and copy your new API key.
  • DeepSeek / XAI / Google Gemini:
    1. Follow the specific sign-up and API key generation process on each provider's respective developer platform.
    2. DeepSeek Platform
    3. XAI (Grok)
    4. Google AI for Developers (Gemini)

Once you have your key(s), proceed to the Authentication & Security section below to learn how to add them to the plugin securely.

4. Authentication & Security

Setting API Keys:

Never hard-code API keys. The plugin provides a secure, centralized settings panel.

  1. In the Unreal Editor, navigate to Edit > Project Settings.
  2. Scroll down to the Plugins section on the left and click on GenAI Plugin.
    Settings panel for API keys
    Settings panel for API keys
  3. Enter your API keys from each provider (OpenAI, Anthropic, etc.) into the corresponding fields.
  4. These keys are automatically saved to an encrypted binary file at YourProject/Saved/Config/GenAI/secureconfig.bin.
  5. This file must not be committed to source control. Generally "Saved" folder is already part of .gitignore, if it is not add the following line to it to prevent leaking your keys: 
    /Saved/Config/GenAI/secureconfig.bin

Using a Proxy Server (Optional, For Production):

Why use a proxy?
  • Security: Your API keys never leave your server. The client application makes requests to your server, and your server securely attaches the API key before forwarding the request to the AI provider.
  • Centralized Control: Manage keys, monitor usage, and control access for all users from a single backend.
  • Custom Logic: Add logging, caching, or pre/post-processing logic on your server before returning data to the game client.
Retargetting the plugin to proxy:

Once your own backend server is running:

  1. Go to the GenAI Plugin settings in the Unreal Editor.
    Settings panel for API keys
    Retargetting the API requests to proxy server
  2. Under the API Endpoint Management section for each provider, check the "Override" box.
  3. Enter the URL of your proxy endpoint. For example, for OpenAI, you might enter https://api.yourgame.com/openai.

Now, all API calls from the plugin to that provider will be routed through your server instead of directly to the AI provider's endpoint.

5. Core Concepts

Asynchronous by Design

All API calls in this plugin are asynchronous (non-blocking). This is essential to prevent your game from freezing while waiting for a response from the AI server. The workflow is always:

  1. You call a function to send a request (e.g., UGenOAIChat::SendChatRequest).
  2. You provide a callback delegate (in C++) or connect to an event pin (in Blueprints).
  3. Your game continues running without interruption.
  4. When the response arrives, your callback function or event is executed with the result.

Lifetime Management for Asynchronous Operations

When working with asynchronous API calls, a critical challenge in Unreal Engine is Object Lifetime. The core problem occurs when an Actor is destroyed while an API request is in-flight—the callback can attempt to access a "dangling pointer," causing crashes.

Imagine you start an image generation request from an Actor in your level. The request is sent, and the server begins processing it. Before the server can respond, you stop Play-In-Editor (PIE). The engine's Garbage Collector destroys all objects from that session, including your Actor. When the server response arrives, the plugin tries to deliver the result to your Actor, which no longer exists, causing a crash.

Blueprint Solution: Automatic Safety

Blueprint nodes are built on UCancellableAsyncAction and handle lifetime automatically using TWeakObjectPtr. The nodes check if the calling object is still valid before executing callbacks, preventing crashes. Key features:

  • Automatic Cleanup: When objects are destroyed, weak pointers become invalid and callbacks are safely skipped.
  • Explicit Cancellation: Use the Cancel node to terminate requests early and clean up resources.
    Settings panel for API keys
    Blueprint nodes automatically handle lifetime safety
  • No Extra Work Required: Blueprint developers can trust the async nodes to handle all safety automatically.
C++ Solution: Developer-Managed

When using static C++ functions, you must handle lifetime yourself. The static functions provide maximum flexibility but require proper safety patterns:

  1. Callback Safety: Always capture a TWeakObjectPtr to your calling object and check WeakThis.IsValid() at the start of every callback.
  2. Request Cancellation: Store the returned TSharedPtr<IHttpRequest> and call CancelRequest() in your object's EndPlay method to prevent wasting resources.
  3. check() Macro: All static functions include check(OnComplete.IsBound()) to catch unbound delegates in development builds.
// Essential C++ safety pattern
void AMyActor::RequestImageGeneration()
{
    FGenOAIImageSettings ImageSettings;
    ImageSettings.Prompt = TEXT("A beautiful sunset over mountains");
    
    // Create weak pointer for safe callback capture
    TWeakObjectPtr&lt;AMyActor&gt; WeakThis(this);
    
    // Store request handle for potential cancellation
    ActiveRequest = UGenOAIImageGeneration::SendImageGenerationRequest(ImageSettings,
        FOnImageGenerationCompletionResponse::CreateLambda([WeakThis](const TArray&lt;uint8&gt;& ImageBytes, const FString& Error, bool bSuccess)
        {
            // CRITICAL: Always check validity first
            if (!WeakThis.IsValid()) return;
            
            AMyActor* StrongThis = WeakThis.Get();
            StrongThis->OnImageReceived(ImageBytes, Error, bSuccess);
        })
    );
}

void AMyActor::EndPlay(const EEndPlayReason::Type EndPlayReason)
{
    // Cancel any in-flight requests to prevent waste
    if (ActiveRequest.IsValid() && ActiveRequest->GetStatus() == EHttpRequestStatus::Processing)
    {
        ActiveRequest->CancelRequest();
    }
    ActiveRequest.Reset();
    Super::EndPlay(EndPlayReason);
}

Using Custom Models (Not in Enums)

AI providers release new models frequently. You don't have to wait for a plugin update to use them. All settings objects (e.g., FGenOAIChatSettings) allow you to specify a model name as a simple string, which will always override the enum selection.

Blueprint Example:

In the "Make Gen OpenAI Chat Settings" node, simply type the new model's name (e.g., "gpt-5-exclusive-preview") into the Model (String) input field. This will be used instead of the enum dropdown.

Settings panel for API keys
Using a custom model in Blueprint
C++ Example:
#include "Models/OpenAI/GenOAIChat.h"

void AMyActor::SendCustomModelRequest()
{
FGenOAIChatSettings ChatSettings;
ChatSettings.Model = EOpenAIChatModel::Custom;
ChatSettings.CustomModelName = TEXT("gpt-5-exclusive-preview");

    // ... complete the rest of the request

}

Additional Points to Note

  1. HTTP Timeouts: When streaming is not used, complex reasoning models (like DeepSeek Reasoner) can take longer to generate a response. If you experience frequent timeout errors, consider increasing the default HTTP timeout in your project's DefaultEngine.ini file: 
        [/Script/Engine.NetworkSettings]
    NetConnectionTimeout=180.0 
    [HTTP]
    HttpConnectionTimeout=180.0 
    HttpReceiveTimeout=180.0
  2. Audio Formats (TTS): The Text-to-Speech feature currently outputs audio in the PCM format. The plugin includes Blueprint utilities to convert this raw data into a playable USoundWave asset at runtime.
    Blueprint Example
  3. API Key Security: The secureconfig.bin file is encrypted and non-portable by design. Do not attempt to copy it between projects or machines. Always add API keys through the Project Settings panel. Never commit this file to source control.
  4. Platform Compatibility: The plugin uses standard engine modules (HTTP, Json) for maximum cross-platform compatibility. However, always test features on your specific target hardware, especially file-system-dependent operations like loading images for vision requests.
  5. Plugin Dependencies: This plugin relies only on core Unreal Engine modules. It does not introduce any external third-party libraries, ensuring a clean and stable integration.

6. Usage Guides & Examples

Finding the Example Assets

  • Blueprint Examples: Example blueprints can be found either inside the plugin's content folder at Engine/Plugins/GenAIForUnrealContent/ExampleBlueprints/, or by downloading the example project here
    Blueprint Example
    Blueprint Examples
  • C++ Examples: The plugin has a seperate module for C++ examples called GenAIExamples.

Blueprint Usage Walkthrough

All Blueprint nodes are asynchronous (latent) and have a similar structure. Here is a walkthrough for a OpenAI multimodal chat request.

Blueprint Node Example

  1. Request OpenAI Chat Completion: This is the main node that initiates the API call.
  2. Settings: Use a "Make" node (e.g., Make Gen OpenAI Chat Settings) to configure the request. Here you set the model, temperature, max tokens, and importantly, the messages.
  3. OnComplete (Event): This event pin fires as soon as a response is received from the server. It provides the final Response string, an Error Message if something went wrong, and a `Success` boolean.

C++ Usage Walkthrough (OpenAI Chat)

The C++ workflow mirrors the Blueprint logic, using settings structs and response delegates.

#include "Data/OpenAI/GenOAIChatStructs.h"
#include "Models/OpenAI/GenOAIChat.h"

void ATOpenAIChatExample::TestTextChat()
{
	UE_LOG(LogGenAI, Log, TEXT("--- Testing OpenAI Text-Only Chat ---"));
	
	FGenOAIChatSettings ChatSettings;
	ChatSettings.Model = EOpenAIChatModel::GPT_4_1_nano; // Use the latest text model
	ChatSettings.Messages.Add(FGenChatMessage(TEXT("user"), TEXT("What is the distance between Earth and the Moon in kilometers?")));
	ChatSettings.MaxTokens = 100;

	UGenOAIChat::SendChatRequest(
	   ChatSettings,
	   FOnChatCompletionResponse::CreateLambda(
		  [this](const FString& Response, const FString& ErrorMessage, bool bSuccess)
		  {
			 if (!UTHelper::IsContextStillValid(this)) return;
			 
			 if (bSuccess)
			 {
				 UE_LOG(LogGenAI, Log, TEXT("OpenAI Text Chat Response: %s"), *Response);
			 }
			 else
			 {
				 UE_LOG(LogGenAI, Error, TEXT("OpenAI Text Chat Error: %s"), *ErrorMessage);
			 }
		  })
	);
}

7. Building Long Conversations & Chat Context

When building conversational AI systems, you often need to maintain context across multiple exchanges. Instead of sending isolated messages, you build up a conversation history that includes system prompts, user messages, and AI responses. This allows the AI to remember previous parts of the conversation and respond appropriately.

How Conversation Context Works

AI models are stateless - they don't remember previous conversations. To simulate memory, you must send the entire conversation history with each new request. The typical flow is:

  1. Initial Setup: Start with a system message that defines the AI's role and behavior
  2. User Input: Add the user's message to the conversation
  3. AI Response: Send the full context to get a response, then add that response to the history
  4. Continue: Repeat steps 2-3, always including the full conversation history

Conversation Structure Example

A typical conversation context grows like this:

Turn 1: [System], [User] → AI responds → [System], [User], [Assistant]
Turn 2: [System], [User], [Assistant], [User] → AI responds → [System], [User], [Assistant], [User], [Assistant]
Turn 3: [System], [User], [Assistant], [User], [Assistant], [User] → AI responds → ...and so on

C++ Implementation

Here's how to implement conversation context in C++, similar to the pattern shown in your example:

#include "Data/OpenAI/GenOAIChatStructs.h"
#include "Models/OpenAI/GenOAIChat.h"

class YOURGAME_API AConversationalNPC : public AActor
{
    GENERATED_BODY()

private:
    // Store the ongoing conversation context
    TArray<FGenChatMessage> ConversationContext;
    
public:
    void InitializeConversation();
    void HandlePlayerMessage(const FString& PlayerInput);
    void SendChatRequest();
};

void AConversationalNPC::InitializeConversation()
{
    // Start with a system message that defines the AI's role
    FGenChatMessage SystemMessage;
    SystemMessage.Role = TEXT("system");
    SystemMessage.Content = TEXT("You are a helpful NPC in a medieval village. You know about local events and can provide information to travelers. Respond in character.");
    
    ConversationContext.Add(SystemMessage);
}

void AConversationalNPC::HandlePlayerMessage(const FString& PlayerInput)
{
    // Add the player's message to our conversation history
    FGenChatMessage UserMessage;
    UserMessage.Role = TEXT("user");
    UserMessage.Content = PlayerInput;
    ConversationContext.Add(UserMessage);
    
    // Send the request with full conversation context
    SendChatRequest();
}

void AConversationalNPC::SendChatRequest()
{
    FGenOAIChatSettings ChatSettings;
    ChatSettings.Model = EOpenAIChatModel::GPT_4_1_mini;
    ChatSettings.Messages = ConversationContext; // Send the entire conversation history
    ChatSettings.MaxTokens = 150;
    
    TWeakObjectPtr<AConversationalNPC> WeakThis(this);
    
    UGenOAIChat::SendChatRequest(
        ChatSettings,
        FOnChatCompletionResponse::CreateLambda([WeakThis](const FString& Response, const FString& Error, bool bSuccess)
        {
            if (!WeakThis.IsValid()) return;
            
            AConversationalNPC* StrongThis = WeakThis.Get();
            
            if (bSuccess)
            {
                // Add the AI's response to our conversation context
                FGenChatMessage AssistantMessage;
                AssistantMessage.Role = TEXT("assistant");
                AssistantMessage.Content = Response;
                StrongThis->ConversationContext.Add(AssistantMessage);
                
                // Display or process the response
                UE_LOG(LogTemp, Log, TEXT("NPC Response: %s"), *Response);
            }
            else
            {
                UE_LOG(LogTemp, Error, TEXT("Chat Error: %s"), *Error);
            }
        })
    );
}

Advanced Context Management

Context Length Management

As conversations grow longer, they can exceed the model's context window (token limit). Consider these strategies:

  • Sliding Window: Keep only the most recent N messages while preserving the system message
  • Summarization: Periodically summarize older parts of the conversation and replace them with a summary
  • Important Message Retention: Keep key messages that contain important context while removing less important ones
void AConversationalNPC::TrimConversationContext(int32 MaxMessages)
{
    if (ConversationContext.Num() <= MaxMessages) return;
    
    // Always keep the system message (assumed to be first)
    FGenChatMessage SystemMessage = ConversationContext[0];
    
    // Keep only the most recent messages
    TArray<FGenChatMessage> TrimmedContext;
    TrimmedContext.Add(SystemMessage);
    
    int32 StartIndex = FMath::Max(1, ConversationContext.Num() - MaxMessages + 1);
    for (int32 i = StartIndex; i < ConversationContext.Num(); i++)
    {
        TrimmedContext.Add(ConversationContext[i]);
    }
    
    ConversationContext = TrimmedContext;
}

Multi-Character Conversations

For group conversations or multi-NPC scenarios, you can use custom roles or include character names in the content:

// Option 1: Use content formatting
FGenChatMessage NPCMessage;
NPCMessage.Role = TEXT("assistant");
NPCMessage.Content = FString::Printf(TEXT("[%s]: %s"), *NPCName, *Response);

// Option 2: Use custom role names (some models support this)
FGenChatMessage CustomMessage;
CustomMessage.Role = NPCName; // "Shopkeeper", "Guard", etc.
CustomMessage.Content = Response;

Best Practices for Conversation Context

  • Clear System Messages: Define the AI's role, personality, and knowledge clearly in the system message
  • Consistent Formatting: Maintain consistent message formatting throughout the conversation
  • Context Awareness: Include relevant world state or character information in system messages
  • Error Handling: Always check for API errors and have fallback responses ready
  • Performance: Monitor token usage and implement context trimming for long conversations
  • Memory Management: Consider persistent storage for conversations that should survive game sessions

8. Streaming

The plugin provides robust support for real-time streaming of AI responses, a crucial feature for creating dynamic and interactive experiences. Streaming allows you to receive the AI's response in small chunks (deltas) as they are generated, rather than waiting for the entire message to be completed. This is ideal for applications like live-narrated dialogue, real-time-generated character responses, or any feature where immediate feedback is essential.

How It Works

Streaming is implemented using Server-Sent Events (SSE), a standard protocol for pushing real-time updates from a server to a client. When a streaming request is made, the connection remains open, and the server sends a series of data packets. Each packet contains a small part of the ongoing response, such as a few words or a single sentence.

The plugin provides dedicated classes for handling streaming from different AI providers, such as UGenOAIChatStream, UGenDSeekChatStream, and UGenXAIChatStream. These classes manage the HTTP connection and parse the incoming SSE events, broadcasting them through delegates that your application can subscribe to.

Blueprint Implementation

In Blueprints, you can use the dedicated "Request...Stream" nodes for each provider. For example, Request OpenAI Chat Stream initiates a streaming request and provides an OnEvent delegate that fires for each received chunk. Here's an example of how to handle streaming events for OpenAI in Blueprints:

Blueprint streaming example
Blueprint streaming implementation example

C++ Implementation

The C++ implementation offers more fine-grained control and follows a similar pattern. You use the static SendStreamChatRequest function from the relevant stream class and provide a callback delegate to handle the events.

Here's an example of how to implement OpenAI streaming in C++:

#include "Models/OpenAI/GenOAIChatStream.h"

void AMyActor::TestStreamingChat()
{
    FGenOAIChatSettings ChatSettings;
    ChatSettings.Model = EOpenAIChatModel::GPT_4_1_nano;
    ChatSettings.Messages.Add(FGenChatMessage(TEXT("user"), TEXT("Tell me a story about a brave knight.")));
    ChatSettings.bStream = true; // Ensure streaming is enabled

    // Send the request with a lambda to handle the stream events
    ActiveRequestStreamingChat = UGenOAIChatStream::SendStreamChatRequest(
        ChatSettings,
        FOnOpenAIChatStreamResponse::CreateUObject(this, &AMyActor::OnStreamingChatEvent)
    );
}

void AMyActor::OnStreamingChatEvent(const FGenOpenAIStreamEvent& StreamEvent)
{
    if (StreamEvent.bSuccess)
    {
        if (StreamEvent.EventType == EOpenAIStreamEventType::ResponseOutputTextDelta)
        {
            // Log each piece of the story as it arrives
            UE_LOG(LogGenAI, Log, TEXT("Story chunk: %s"), *StreamEvent.DeltaContent);
        }
        else if (StreamEvent.EventType == EOpenAIStreamEventType::ResponseCompleted)
        {
            UE_LOG(LogGenAI, Log, TEXT("Story finished!"));
            ActiveRequestStreamingChat.Reset(); // Clean up the request handle
        }
    }
    else
    {
        UE_LOG(LogGenAI, Error, TEXT("Streaming Error: %s"), *StreamEvent.ErrorMessage);
        ActiveRequestStreamingChat.Reset(); // Clean up on error
    }
}

Best Practices

  • UI Updates: When updating UI elements like text boxes with streamed content, be mindful of performance. Appending text frequently can be expensive, so consider batching updates or using optimized text components.
  • Cancellation: Always store the TSharedPtr<IHttpRequest> returned by SendStreamChatRequest and call CancelRequest() in your object's EndPlay or Destroy method. This prevents orphaned requests and resource leaks if the actor is destroyed while a stream is active.
  • Error Handling: Implement logic to handle potential errors during the stream, such as network issues or API errors. The OnEvent delegate will provide detailed error messages.
  • Provider Differences: While the plugin standardizes the streaming interface, be aware of minor differences between AI providers. For example, DeepSeek offers a ReasoningUpdate event that provides "chain-of-thought" insights, which is unique to its models.

9. Structured Output

Structured Outputs is a feature that ensures the model will always generate responses that adhere to your supplied JSON Schema, so you don't need to worry about the model omitting a required key, or hallucinating an invalid enum value.

Blueprint Structured Output Example
Example of JSON response from ChatGPT API for NPCs

Some benefits of Structured Outputs include:

  • Reliable type-safety: No need to validate or retry incorrectly formatted responses
  • Explicit refusals: Safety-based model refusals are now programmatically detectable
  • Simpler prompting: No need for strongly worded prompts to achieve consistent formatting

Schema JSON example

{
"type": "object",
"properties": {
    "count": {
        "type": "integer",
        "description": "The total number of users."
    },
    "users": {
        "type": "array",
        "items": {
            "type": "object",
            "properties": {
                "name": { "type": "string", "description": "The user's name." },
                "heading_to": { "type": "string", "description": "The user's destination." }
            },
            "required": ["name", "role", "age", "heading_to"]
        }
    }
},
"required": ["count", "users"]
}

Using structured output in the C++:

FString MySchemaJson = R"(<insert your schema JSON here>)";

UGenAISchemaService::RequestStructuredOutput(
    TEXT("Generate a list of users and their details"),
    MySchemaJson,
    [](const FString& Response, const FString& Error, bool Success) {
       if (Success)
       {
           UE_LOG(LogTemp, Log, TEXT("Structured Output: %s"), *Response);
       }
       else
       {
           UE_LOG(LogTemp, Error, TEXT("Error: %s"), *Error);
       }
    }
);

In Blueprints:

Blueprint Structured Output Example
Blueprint Structured Output Example

10. Text-to-Speech and Transcription

Convert text to natural-sounding speech and speech back to text using OpenAI's advanced models. The plugin supports TTS with various voices and transcription with the Whisper model.

Text-to-Speech (TTS)

Easily bring your game to life by converting text into natural-sounding speech with OpenAI's advanced TTS models. The plugin supports a variety of voices and delivers high-quality 24kHz audio output ready for immediate use in Unreal Engine.

Use Cases

  • Dynamic Dialogue: Convert NPC dialogue text into natural speech in real-time
  • Narration: Create voiceovers for tutorials, story elements, or gameplay instructions
  • Accessibility: Provide audio versions of in-game text for visually impaired players
  • Localization: Generate voices for different languages without recording multiple voice actors
  • Procedural Content: Add voice to dynamically generated content like quest descriptions or item details

Blueprint Implementation

Using TTS in Blueprints is straightforward. You can configure the voice, model, and input text, then convert the resulting audio data into a playable USoundWave.

Blueprint TTS and Transcription Example
A simple Blueprint graph showing how to convert text to a playable sound.

The key nodes are:

  1. Request OpenAI Text To Speech: Kicks off the TTS request. Use the Make Gen OpenAI Text To Speech Settings node to configure it.
  2. Convert PCM Audio To Sound Wave: A helper node that takes the raw audio data from the API and turns it into a USoundWave.
  3. Create Sound 2D: A standard Unreal node to play the generated sound in the world. Set it to "Auto Destroy" to avoid memory leaks.

C++ Implementation

Here's a concise C++ example for converting text to speech and playing the audio.


#include "Models/OpenAI/GenOAITextToSpeech.h"
#include "Data/OpenAI/GenOAIAudioStructs.h"
#include "Utilities/GenAIAudioUtils.h"
#include "Components/AudioComponent.h"
#include "Kismet/GameplayStatics.h"

void AMyActor::SpeakText(const FString& Text)
{
    UE_LOG(LogTemp, Log, TEXT("Requesting speech for: '%s'"), *Text);

    // 1. Configure the TTS request
    FGenOAITextToSpeechSettings TTSSettings;
    TTSSettings.InputText = Text;
    TTSSettings.Model = EOpenAITTSModel::TTS_1;
    TTSSettings.Voice = EGenAIVoice::Nova;

    // 2. Send the request with a Lambda callback
    UGenOAITextToSpeech::SendTextToSpeechRequest(TTSSettings,
        FOnTTSCompletionResponse::CreateLambda([this](const TArray<uint8>& AudioData, const FString& ErrorMessage, bool bSuccess)
        {
            if (bSuccess && AudioData.Num() > 0)
            {
                UE_LOG(LogTemp, Log, TEXT("TTS successful! Received %d bytes of audio."), AudioData.Num());

                // 3. Convert data to a playable sound wave
                if (USoundWave* PlayableSound = UGenAIAudioUtils::ConvertPCMAudioToSoundWave(AudioData))
                {
                    // 4. Play the sound using an auto-destroying audio component
                    UAudioComponent* AudioComponent = UGameplayStatics::CreateSound2D(this, PlayableSound);
                    if(AudioComponent)
                    {
                        AudioComponent->bAutoDestroy = true;
                        AudioComponent->Play();
                    }
                }
            }
            else
            {
                UE_LOG(LogTemp, Error, TEXT("TTS request failed: %s"), *ErrorMessage);
            }
        })
    );
}

Speech-to-Text (Transcription)

Convert any audio source into text using OpenAI's powerful Whisper model. This feature is perfect for implementing voice commands, transcribing dialogue, or enabling any voice-driven interaction in your game. The plugin can process audio directly from a file or from raw audio data in memory.

Blueprint Implementation

The Blueprint node for transcription is just as simple as the TTS node. You provide the audio data and it returns the transcribed text.

Blueprint TTS and Transcription Example
A simple Blueprint graph showing how to convert text to a playable sound.

The key node is Request OpenAI Transcription From Data. As seen in the image in the TTS section, you can feed the audio data directly from the TTS output into the transcription node to perform a full round-trip conversion.

C++ Implementation

Here's how you can transcribe audio data from memory. This example assumes you already have the audio data in a TArray<uint8>.


#include "Models/OpenAI/GenOAITranscription.h"
#include "Data/OpenAI/GenOAIAudioStructs.h"

void AMyActor::TranscribeAudio(const TArray<uint8>& AudioData)
{
    if (AudioData.Num() == 0)
    {
        UE_LOG(LogTemp, Warning, TEXT("TranscribeAudio: Input audio data is empty."));
        return;
    }

    UE_LOG(LogTemp, Log, TEXT("Starting transcription for %d bytes of audio..."), AudioData.Num());

    // 1. Configure the transcription request
    FGenOAITranscriptionSettings TranscriptionSettings;
    TranscriptionSettings.Model = EOpenAITranscriptionModel::Whisper_1;
    TranscriptionSettings.Language = TEXT("en"); // Optional: Specify language for better accuracy

    // 2. Send the request with a Lambda callback
    UGenOAITranscription::SendTranscriptionRequestFromData(AudioData, TranscriptionSettings,
        FOnTranscriptionCompletionResponse::CreateLambda([](const FString& Transcript, const FString& ErrorMessage, bool bSuccess)
        {
            if (bSuccess)
            {
                UE_LOG(LogTemp, Log, TEXT("Transcription successful! Transcript: '%s'"), *Transcript);
            }
            else
            {
                UE_LOG(LogTemp, Error, TEXT("Transcription failed: %s"), *ErrorMessage);
            }
        })
    );
}

11. Image Generation

The GenAI for Unreal plugin integrates OpenAI's powerful image generation models, allowing you to create stunning, high-quality images from simple text prompts. This opens up incredible possibilities for procedural art, dynamic content, and rapid prototyping directly inside Unreal Engine.

Use Cases

  • Concept Art: Rapidly generate visual concepts for characters, environments, and props
  • Placeholder Assets: Create temporary artwork while waiting for final art assets
  • Procedural Content: Generate unique textures, landscapes, or decorative elements
  • Marketing Materials: Create promotional images and screenshots for your game
  • Prototyping: Quickly visualize game ideas and mechanics
  • User-Generated Content: Allow players to create custom imagery within your game

Models and Capabilities

The plugin provides access to OpenAI's state-of-the-art image models:

  • ✨ gpt-image-1: The latest and most advanced model, responsible for the popular "Ghibli trend." It excels at creating artistic, coherent, and high-fidelity images with remarkable prompt adherence.
  • 🎨 DALL-E 3: A powerful model offering excellent image quality and detail. A great all-rounder for a variety of styles.
  • ⚙️ DALL-E 2: A legacy model that is more cost-effective and supports generating multiple images in a single request, making it suitable for bulk generation or rapid iteration.

Blueprint Implementation

Blueprints make image generation incredibly accessible. You can easily configure a prompt, select your model, and save the resulting image to disk.

Blueprint Image Generation Example
A simple Blueprint graph for generating an image and saving it as a file.

The workflow is simple:

  1. Request OpenAI Image Generation: This is the main node that sends your request to the API. Use Make Gen OpenAI Image Generation Settings to configure it.
  2. Save Array to File: A utility node to save the raw image data (returned as a byte array) to a .png file.
  3. Load Texture 2D from File: (Optional) After saving, you can immediately load the image back into the engine as a UTexture2D to display it on a material.

C++ Implementation

For more control, you can call the image generation functions directly from C++. Here is a clean, focused example.


#include "Models/OpenAI/GenOAIImageGeneration.h"
#include "Data/OpenAI/GenOAIImageStructs.h"
#include "Misc/FileHelper.h"
#include "Misc/Paths.h"

void AMyActor::GenerateImage(const FString& Prompt)
{
    UE_LOG(LogTemp, Log, TEXT("Requesting image for prompt: '%s'"), *Prompt);

    // 1. Configure the image generation settings
    FGenOAIImageSettings ImageSettings;
    ImageSettings.Prompt = Prompt;
    ImageSettings.Model = EGenOAIImageModel::GPT_Image_1; // Use the latest model
    ImageSettings.Size = EGenAIImageSize::Size1024x1024;  // Standard high-res square
    ImageSettings.Quality = EGenAIImageQuality::HD;

    // 2. Send the request and handle the response in a Lambda
    UGenOAIImageGeneration::SendImageGenerationRequest(ImageSettings,
        FOnImageGenerationCompletionResponse::CreateLambda([this](const TArray<uint8>& ImageBytes, const FString& ErrorMessage, bool bSuccess)
        {
            if (bSuccess && ImageBytes.Num() > 0)
            {
                UE_LOG(LogTemp, Log, TEXT("Image generated successfully! Saving to disk..."));

                // 3. Save the image data to a file
                const FString FilePath = FPaths::ProjectSavedDir() / TEXT("AI_Generated_Image.png");
                if (FFileHelper::SaveArrayToFile(ImageBytes, *FilePath))
                {
                    UE_LOG(LogTemp, Log, TEXT("Image saved successfully to %s"), *FilePath);
                }
                else
                {
                    UE_LOG(LogTemp, Error, TEXT("Failed to save image to disk."));
                }
            }
            else
            {
                UE_LOG(LogTemp, Error, TEXT("Image generation failed: %s"), *ErrorMessage);
            }
        })
    );
}

// Don't forget to cancel any active requests when the actor is destroyed!
void AMyActor::EndPlay(const EEndPlayReason::Type EndPlayReason)
{
    if (ActiveRequest.IsValid() && ActiveRequest->GetStatus() == EHttpRequestStatus::Processing)
    {
        ActiveRequest->CancelRequest();
    }
    ActiveRequest.Reset();

    Super::EndPlay(EndPlayReason);
}

12. How to Run Tests

Disclaimer

Running this test will incur costs, as it makes API requests to all available model combinations across different organisations.

The plugin includes a suite of automation tests to verify that all API integrations are working correctly. To run them:

Blueprint Structured Output Example
  1. Ensure you have set valid API keys in the plugin settings for the providers you wish to test.
  2. In the Unreal Editor, go to Window > Session Frontend.
  3. In the "Automation" tab, you will see a list of available tests.
  4. Find the tests prefixed with GenAI. (e.g., GenAI.OpenAI.Chat).
  5. Check the boxes for the tests you want to run, and press the "Start Tests" button.

The tests will run and report their status. This is a great way to confirm your setup is correct or to debug integration issues.

14. Additional Notes

  • Unity Build Compatible: The codebase is fully compatible with Unreal's Unity Build and non-unity build workflows, ensuring it can be integrated into any project's compilation pipeline.

This documentation is based on the GenAI for Unreal plugin. Specific function names, struct members, and Blueprint node names may vary slightly based on the final plugin version. Always refer to the plugin's source code or example content for the most accurate implementation details.