GGUF Models
Embedded inference loads models in the GGUF format — a binary file that packages weights, tokenizer, and metadata into a single file that llama.cpp reads directly. This page covers where to get them, how to pick a size, and what the quantization suffixes (Q4_K_M, Q8_0, F16, etc.) actually mean in practice.
HTTP providers (Ollama, LM Studio) manage their own models — this page is for the Embedded (llama.cpp) path.
Where to Download GGUFs
All three sources below are free to browse. Most models are available in multiple quantizations (file sizes) per release.
| Source | What’s there | Notes |
|---|---|---|
| Hugging Face — GGUF search | Everything | Filter by model size, architecture, downloads |
| Qwen (Alibaba) | All Qwen sizes + quantizations | Best small-model family for games — 0.5B / 1.5B / 3B / 7B sizes |
| bartowski | GGUF conversions of most popular releases | Reliable, every quant variant published |
| Meta Llama | Official Llama 3.x / 4.x | License-gated — request access on Hugging Face first |
| Google Gemma | Official Gemma 2 GGUFs | Small, fast, good for dialogue |
| Mistral AI | Mistral 7B family | Strong multilingual performance |
How to download a single file: open the model’s “Files and versions” tab on Hugging Face, click the .gguf filename, then click Download. You don’t need to clone the whole repo.
Picking a Quantization
“Quantization” is how many bits represent each weight. Fewer bits = smaller file, faster inference, lower quality. All of these load with the same Load Embedded Model Blueprint node — the choice is just file-size vs quality.
| Suffix | Precision | Approx size (7B model) | When to use |
|---|---|---|---|
Q2_K |
2-bit | ~2.7 GB | Smoke tests, tiny models, mobile devices |
Q3_K_M |
3-bit | ~3.3 GB | Aggressive size reduction, noticeable quality loss |
Q4_K_M |
4-bit | ~4.1 GB | Recommended default — best quality/size trade |
Q5_K_M |
5-bit | ~4.8 GB | When you have extra RAM/VRAM |
Q6_K |
6-bit | ~5.5 GB | Near-full quality, still compact |
Q8_0 |
8-bit | ~7.2 GB | Near-lossless; desktop-only |
F16 |
16-bit | ~13 GB | Reference quality; needs a big GPU |
F32 |
32-bit | ~26 GB | Research use only |
Rule of thumb for shipping games: Q4_K_M unless you have a specific reason to change. It’s the de facto standard across the llama.cpp ecosystem — widely tested, well-supported, and the size/quality trade has held up across model generations.
Sizing by Target Hardware
RAM usage roughly equals the model file size + the KV cache (depends on context size). The table below is a starting guide, not a hard cutoff.
| Target hardware | Comfortable model size | Example |
|---|---|---|
| Mobile (2–4 GB RAM) | Under 1 GB | Qwen2.5-0.5B Q4_K_M (~400 MB) |
| Handheld / low-end laptop | 1–2 GB | TinyLlama 1.1B Q4_K_M (~637 MB) |
| Mid-range desktop | 2–5 GB | Phi-3-mini Q4_K_M (~2.2 GB) |
| Gaming desktop (16 GB RAM) | 5–10 GB | Llama 3.1 8B Q4_K_M (~4.9 GB) |
| High-end desktop (32 GB+) | 10–25 GB | Llama 3 70B Q2_K (~26 GB — slow, quality-first) |
| Data-center / research | 25+ GB | Llama 3 70B Q4_K_M (~40 GB) |
For GPU offload with GPU Layers > 0, the relevant limit is VRAM, not RAM. Most consumer GPUs have 8–16 GB VRAM — that’s the working budget for anything running on-device.
When the model is bigger than your VRAM
GPU Layers defaults to 99 (full offload). If the weights you ask to offload don’t fit in VRAM, llama.cpp aborts the process — it does not return a friendly error. This is a real risk for anyone shipping a game whose players have varying GPU sizes.
What the plugin does for you:
- For CPU-only loads (
GPU Layers = 0),Load Embedded Modelpre-flights the file size against available system RAM and returns a clear error before calling into llama.cpp. - For full-offload loads on large files (> 8 GiB), a warning is emitted to
LogGenAILlamatelling the user this might abort if VRAM is insufficient. - VRAM itself cannot be queried portably across Windows / macOS / Linux / consoles, so the plugin does not hard-block on it — the warning is advisory.
What you should do in your game code:
- Let the player pick a model size, or gate the large model behind a settings toggle.
- Start with
GPU Layers = 99— if loading fails, fall back to a partial offload (e.g.GPU Layers = 20) or CPU-only. - For shipped titles, prefer smaller quantizations (Q4_K_M at 8B is safer than Q8_0 at 13B) over chasing quality, because you’re deploying to hardware you can’t inspect.
Recommended Starter Models
All of these are well-tested and play nicely with the plugin’s pinned llama.cpp b8802 build.
| Model | Params | Q4_K_M Size | Best for | Link |
|---|---|---|---|---|
| Qwen2.5-0.5B-Instruct | 0.5B | ~400 MB | Ultra-lightweight, smoke tests | Hugging Face |
| TinyLlama 1.1B | 1.1B | ~637 MB | Lightweight NPC dialogue | Hugging Face |
| Gemma 2B | 2B | ~1.4 GB | Balanced size and quality | Hugging Face |
| Qwen2.5-3B-Instruct | 3B | ~1.9 GB | Sharper dialogue, still tiny | Hugging Face |
| Phi-3-mini | 3.8B | ~2.2 GB | Smarter game AI, desktop/console | Hugging Face |
| Llama 3.1 8B Instruct | 8B | ~4.9 GB | Full-quality chat, desktop | Hugging Face |
What GGUF Files Don’t Include
- Vision/audio capabilities — The plugin currently supports text-only embedded inference. Multimodal GGUFs (LLaVA, Llama 3.2 Vision, Qwen2-VL) will load but image/audio inputs are not yet wired into the embedded path. Use an HTTP provider like Ollama for vision models.
- Embeddings-only models — llama.cpp can run embedding models, but the plugin’s Blueprint nodes are built around chat completion, not embeddings.
- Tool-use fine-tunes — The underlying model may support structured output / function calling, but the plugin does not yet expose a dedicated tool-calling node. Prompt-based JSON coaxing works.
Placing Models in Your Project
By convention, drop your .gguf files under <YourProject>/Content/Models/. The Load Embedded Model Blueprint node accepts either:
- Relative path — resolved from the project root (e.g.
Models/qwen2.5-0.5b-instruct-q2_k.gguf). - Absolute path — any full path on disk (useful for shared model libraries across projects).
Packaging .gguf files into a shipped build
Important: .gguf files are not packaged by default. Unreal’s cooker only stages .uasset files automatically — raw binary files like GGUF models are skipped, so a build that loads fine in the editor will fail with Model file not found at runtime in a packaged game.
To ship them with your game, tell the cooker to copy the folder as raw (non-UFS) files. Two equivalent ways:
Option A — Project Settings (GUI):
- Open Edit → Project Settings → Packaging
- Click Show Advanced
- Find Additional Non-Asset Directories To Copy
- Add an entry with the path relative to
Content/, e.g.ModelsorBlueprintExamplesOllama/Models
Option B — Config/DefaultGame.ini:
[/Script/UnrealEd.ProjectPackagingSettings]
+DirectoriesToAlwaysStageAsNonUFS=(Path="Models")
The path is relative to your project’s Content/ directory. After repackaging, the .gguf files will land in <Build>/<Project>/Content/Models/ and Load Embedded Model will find them.
Why non-UFS? llama.cpp reads model files via native fopen, not Unreal’s asset/package system. Non-UFS staging copies the files to disk as-is rather than bundling them into a .pak.
Tip for large models: Rather than bloating your install size, ship the game without the model and download it on first launch into FPaths::ProjectPersistentDownloadDir(), then pass that absolute path to Load Embedded Model.
See Also
- Quick Start — Download a tiny model and run it in five minutes.
- Blueprint Nodes — How to wire up
Load Embedded Model. - Embedded Inference Setup — Drop llama.cpp binaries into the plugin.
- Troubleshooting — When a model fails to load.