llama.cpp on Windows 11

If you're following an older guide

serve and cli are subcommands, not filenames. Verify what you have:

llama --version
llama serve --help

Some packages may still ship the legacy llama-server.exe alongside the new binary for backward compatibility. Don't rely on it — the official documentation now covers only llama serve, and that's what new flags land in.

1. Install

winget — the documented Windows path

winget install llama.cpp
winget upgrade llama.cpp

This is the officially supported package manager for Windows and tracks new llama.cpp releases automatically. It handles PATH for you.

The one-line installer on llama.app (curl -LsSf https://llama.app/install.sh | sh) is a shell script for macOS and Linux. On Windows, use winget or the release ZIPs.

Verify

llama --version
llama serve --help

Then confirm the GPU is visible:

llama cli --list-devices

Example in my case, here is the results:

PS> llama cli --list-devices
Available devices:
  Vulkan0: Intel(R) Graphics (37025 MiB, 47805 MiB free)
  Vulkan1: NVIDIA GeForce RTX 5090 Laptop GPU (24137 MiB, 23369 MiB free)

If that shows only CPU on a machine with a GPU, see §3 — you may need a backend-specific build rather than the package-manager one.

2. Which Backend Do You Need?

Check what you have first:

Get-CimInstance Win32_VideoController | Select-Object Name, DriverVersion
nvidia-smi          # NVIDIA only — the CUDA version shown is your driver's maximum

3. Prebuilt Binaries from GitHub

Releases: github.com/ggml-org/llama.cpp/releases

Two channels: versioned releases (v0.x tags) and per-commit nightly builds (bNNNNN). The nightlies are the tip of master — new features arrive there first, and so do regressions.

Windows asset naming

llama-<build>-bin-win-<backend>-<arch>.zip

There is a single CPU package rather than separate AVX2/AVX512 downloads, because compute backends ship as loadable DLLs and the best one for your CPU is selected at startup.

The CUDA gotcha

The CUDA ZIP contains llama.cpp's binaries but not NVIDIA's runtime libraries. Download the matching runtime package too and extract it into the same folder:

Those are ~373 MB and hold cudart64_*.dll, cublas64_*.dll, cublasLt64_*.dll. Without them the binary fails at launch with a missing-DLL error. Skip only if you already have the full CUDA Toolkit on PATH.

Which CUDA version? The "CUDA Version" in nvidia-smi is the maximum your driver supports — pick a package at or below it. 12.4 is the broadly-compatible choice.

Download with PowerShell

$repo = "ggml-org/llama.cpp"
$tag  = (Invoke-RestMethod "https://api.github.com/repos/$repo/releases/latest").tag_name
$dest = "$env:USERPROFILE\llama.cpp"
New-Item -ItemType Directory -Path $dest -Force | Out-Null

$asset = "llama-$tag-bin-win-vulkan-x64.zip"
curl.exe -L -o "$env:TEMP\$asset" "https://github.com/$repo/releases/download/$tag/$asset"
Expand-Archive "$env:TEMP\$asset" -DestinationPath $dest -Force

CUDA, with the runtime extracted alongside:

$cuda = "12.4"
$bin  = "llama-$tag-bin-win-cuda-$cuda-x64.zip"
$rt   = "cudart-llama-bin-win-cuda-$cuda-x64.zip"

curl.exe -L -o "$env:TEMP\$bin" "https://github.com/$repo/releases/download/$tag/$bin"
curl.exe -L -o "$env:TEMP\$rt"  "https://github.com/$repo/releases/download/$tag/$rt"

Expand-Archive "$env:TEMP\$bin" -DestinationPath $dest -Force
Expand-Archive "$env:TEMP\$rt"  -DestinationPath $dest -Force   # same folder, deliberately

Use curl.exe, not curl — bare curl is a PowerShell alias for Invoke-WebRequest, which takes different arguments and is far slower on large files.

Verify

cd $dest
.\llama.exe --version
.\llama.exe cli --list-devices

--list-devices is the real test. A CUDA build with missing runtime DLLs or an old driver will still run — silently, at CPU speed.

4. PATH Setup

If you installed the ZIP rather than using winget:

$dest = "$env:USERPROFILE\llama.cpp"
$old  = [Environment]::GetEnvironmentVariable("Path", "User")
[Environment]::SetEnvironmentVariable("Path", "$old;$dest", "User")

Restart the terminal, then llama --version.

Keep the DLLs next to the EXE. Windows resolves dependent DLLs from the executable's own directory first, so llama.exe, ggml*.dll and any cudart64_*.dll must stay together. Copying just the EXE elsewhere breaks it.

Suggested layout:

%USERPROFILE%\llama.cpp\     <- binaries + DLLs, replaced wholesale on update
%USERPROFILE%\models\        <- GGUF files, untouched by updates

5. Build Prerequisites

Only needed if you're compiling. The prebuilt binaries are what the maintainers test, and building rarely buys meaningful speed — the shipped CPU package already selects optimised kernels at runtime.

Required

winget install Git.Git
winget install Kitware.CMake
winget install Ninja-build.Ninja
winget install Microsoft.VisualStudio.2022.BuildTools

Build Tools supplies MSVC, the actual compiler. The winget package installs the shell only — add the C++ workload via Visual Studio Installer → Modify → Desktop development with C++ (~7 GB), or non-interactively:

winget install Microsoft.VisualStudio.2022.BuildTools --override `
  "--quiet --add Microsoft.VisualStudio.Workload.VCTools --includeRecommended"

Backend-specific

CUDA — the Toolkit, not just the driver. The driver runs CUDA programs; the Toolkit contains nvcc, which compiles them. From developer.nvidia.com/cuda-downloads. Check with nvcc --version.

Vulkan — winget install KhronosGroup.VulkanSDK, or from vulkan.lunarg.com. Check with vulkaninfo --summary.

ROCm — AMD's HIP SDK for Windows. Limited to specific GPU architectures; check AMD's compatibility list first.

The build shell

MSVC needs its environment set. Either launch "Developer PowerShell for VS 2022" from the Start menu, or import it:

Import-Module "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\Common7\Tools\Microsoft.VisualStudio.DevShell.dll"
Enter-VsDevShell -VsInstallPath "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools" -DevCmdArguments "-arch=x64"

Skipping this is the most common first-build failure: CMake reports no C++ compiler found, even though Visual Studio is installed.

6. Clone and Build

git clone --depth 1 https://github.com/ggml-org/llama.cpp
cd llama.cpp

CPU:

cmake -B build
cmake --build build --config Release -j

CUDA:

cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

Targeting only your GPU's architecture cuts build time substantially — the default compiles kernels for every supported one:

cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES="89"

Vulkan:

cmake -B build -DGGML_VULKAN=ON
cmake --build build --config Release -j

Several backends in one binary:

cmake -B build -DGGML_CUDA=ON -DGGML_VULKAN=ON
cmake --build build --config Release -j

Then pick at runtime with -dev cuda:0 or -dev vulkan:0.

With Ninja:

cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON
cmake --build build

Output lands in build\bin\Release\ (Visual Studio) or build\bin\ (Ninja).

7. Build Flags Reference

-DGGML_NATIVE defaults on for local builds, tuning for the exact CPU you compile on. Turn it off if the binary must run elsewhere, or it can crash with an illegal-instruction fault on older CPUs.

8. Getting Models

llama.cpp runs GGUF files. Easiest path is to let it fetch from Hugging Face:

llama serve -hf ggml-org/gemma-4-e4b-it-GGUF:Q4_0

The :Q4_0 suffix selects the quantisation within the repo. Files are cached after the first run, so subsequent starts are instant. Set HF_TOKEN for gated repositories.

List what's already cached:

llama serve -cl

Direct download still works:

curl.exe -L -o "$env:USERPROFILE\models\model.gguf" `
  "https://huggingface.co/<user>/<repo>/resolve/main/<file>.gguf"

Choosing a quantisation

Rough sizing: file size + 1–2 GB for context ≈ VRAM for full offload. A larger model at Q4_K_M almost always beats a smaller model at Q8_0 of the same file size.

9. Running the Server

Minimum:

llama serve -m "$env:USERPROFILE\models\model.gguf"

Or straight from Hugging Face:

llama serve -hf ggml-org/gemma-4-e4b-it-GGUF:Q4_0

Defaults to http://127.0.0.1:8080. Open it in a browser for the built-in web UI.

Typical configuration:

llama serve `
  -m "$env:USERPROFILE\models\model.gguf" `
  -c 16384 `
  -ngl all `
  --host 127.0.0.1 `
  --port 8080

The backtick is PowerShell's line-continuation character and nothing may follow it on the line — not even a space.

• -ngl takes auto (default), all, or a number of layers. The old -ngl 999 idiom still works but all says what it means.
• -c 16384 sets context in tokens. -c 0 uses the model's full native context.
• Prompt caching is on by default, so repeated requests sharing a prefix (a system prompt, an ongoing chat) skip reprocessing.

A reusable launcher, serve.ps1:

param(
    [string]$Model = "$env:USERPROFILE\models\model.gguf",
    [int]$Port = 8080,
    [int]$Ctx = 16384
)
llama serve -m $Model --port $Port -c $Ctx -ngl all

10. Server Flags Reference

Core

Workload type

Multimodal models generally work with no extra flags — serve them like a text model and send images through the web UI or the standard OpenAI content format.

Reasoning and speculative decoding

Every option also has an environment-variable form, shown in --help — convenient for services and containers:

LLAMA_ARG_HOST, LLAMA_ARG_MODEL, LLAMA_ARG_CTX_SIZE, LLAMA_ARG_PORT, LLAMA_ARG_N_PARALLEL

The authoritative list for your build:

llama serve --help

11. Router Mode: Serving Several Models

Start llama serve with no model and it becomes a router that loads and unloads models on demand, forwarding each request to the right instance:

llama serve

Models come from three places:

llama serve                                    # 1. anything previously cached via -hf
llama serve --models-dir "$env:USERPROFILE\models"   # 2. a directory of GGUFs
llama serve --models-preset .\my-models.ini    # 3. a preset file with per-model settings

Requests must then name the model:

curl.exe -s http://localhost:8080/v1/models    # list available names

{ "model": "ggml-org/gemma-4-e4b-it-GGUF:Q4_0", "messages": [{"role": "user", "content": "Hello!"}] }

The server loads it on demand. Useful when VRAM only fits one model at a time but you want a single stable endpoint.

12. The CLI

llama cli -m model.gguf                        # interactive chat
llama cli -hf ggml-org/gemma-4-e4b-it-GGUF:Q4_0
llama cli -m model.gguf -sys "Answer in bullet points."
llama cli -m model.gguf -st -p "Give me a baklava recipe"   # single turn, then exit
llama cli -m model.gguf --image photo.png -p "Describe this."
llama cli --list-devices

Inside an interactive session, /image path and /audio path attach media — send the file first, then the prompt referring to it.

Sampling:

Other tools remain separate binaries — llama-bench, llama-quantize, llama-perplexity, llama-gguf-split, llama-imatrix, llama-tokenize.

13. Testing the API

OpenAI-compatible, so most OpenAI clients work by repointing base_url.

PowerShell

$body = @{
    model    = "local"
    messages = @(@{ role = "user"; content = "Say hello in five words." })
} | ConvertTo-Json -Depth 5

Invoke-RestMethod -Uri "http://127.0.0.1:8080/v1/chat/completions" `
    -Method Post -ContentType "application/json" -Body $body |
    Select-Object -ExpandProperty choices |
    ForEach-Object { $_.message.content }

Endpoints

Python

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="not-needed")
resp = client.chat.completions.create(
    model="local",
    messages=[{"role": "user", "content": "Hello"}],
)
print(resp.choices[0].message.content)

api_key must be a non-empty string — the library validates it locally even though the server ignores it unless --api-key is set.

14. Performance Tuning

Measure first

llama-bench -m "$env:USERPROFILE\models\model.gguf"

Reports pp (prompt processing) and tg (token generation) throughput. They behave differently and both matter.

Let auto-fit do its job

With --fit on (the default), llama.cpp sizes offload and slots to your hardware. Reach for manual tuning when you have a specific reason — a shared GPU, a fixed VRAM budget, or a measurement showing the automatic choice is wrong.

MoE models

Mixture-of-Experts models have enormous expert FFN weights of which only a fraction is active per token. Pushing those to CPU keeps the rest on GPU:

llama serve -m model.gguf -ngl all -cmoe

This is what makes a model that nominally exceeds your VRAM actually runnable. You trade some speed for the ability to run it at all — usually a good trade.

Concurrency

llama serve -m model.gguf -c 32768 -np 4

Each slot holds one conversation and context is shared across slots, so this gives each of four slots 8192 tokens, not 32768. Size -c accordingly.

Speculative decoding

A small draft model proposes tokens the large model verifies in batch:

llama serve -m big-model.gguf -md small-draft-model.gguf --spec-type draft-simple

Gains depend on how often the draft agrees, so it helps most on predictable text (code, structured output) and least on creative writing.

15. Background / Autostart

Start-Process -FilePath "llama" `
  -ArgumentList "serve","-m","$env:USERPROFILE\models\model.gguf","-c","16384" `
  -WindowStyle Hidden

Get-Process llama | Stop-Process        # stop it

Start at login via a scheduled task:

$action  = New-ScheduledTaskAction -Execute "llama.exe" `
           -Argument "serve -m $env:USERPROFILE\models\model.gguf -c 16384"
$trigger = New-ScheduledTaskTrigger -AtLogOn
Register-ScheduledTask -TaskName "llama-serve" -Action $action -Trigger $trigger

Run at logon, not at startup — GPU drivers and the user session need to be up, and a system-level task starting too early may not see the GPU at all.

Exposing on your network

llama serve -m model.gguf --host 0.0.0.0 --port 8080 --api-key "your-secret-key"

# elevated
New-NetFirewallRule -DisplayName "llama serve" -Direction Inbound `
  -LocalPort 8080 -Protocol TCP -Action Allow

16. Updating

winget upgrade llama.cpp

For ZIP installs, re-run the §3 download with a fresh $tag and extract over the same folder. Because models live elsewhere, this is cheap. For a rollback path, extract each build to llama.cpp-bNNNNN\ and point a junction at the current one:

New-Item -ItemType Junction -Path "$env:USERPROFILE\llama.cpp" `
  -Target "$env:USERPROFILE\llama.cpp-b10456" -Force

From source:

cd $env:USERPROFILE\llama.cpp
git pull
cmake --build build --config Release -j

If a pull changes CMake files significantly, delete build\ and reconfigure — stale cache entries produce failures that look like source errors.

Pin a version for anything you depend on. GGUF format changes are backward-compatible in practice, but flags get renamed — as the llama-server → llama serve change demonstrates — and a script that worked last month can break on a pull.

17. Troubleshooting

llama-server is not recognised

That binary no longer exists in current releases. Use llama serve. See the table at the top.

Missing DLL on launch

ggml*.dll must be beside llama.exe, and CUDA builds also need the cudart package extracted alongside. A dialog naming a specific DLL tells you which package is absent.

Get-ChildItem "$env:USERPROFILE\llama.cpp\*.dll" | Select-Object Name

Runs, but slowly — is the GPU used?

llama cli --list-devices

Only CPU on a GPU build means old drivers, missing CUDA runtime DLLs, or you downloaded the CPU package. The startup log also states what it offloaded — worth reading.

Out of memory

Less common now that auto-fit is on, but if it happens: lower -c, add -cmoe for MoE models, set an explicit -ngl below all, or move to a smaller quantisation.

Garbled or looping output

Chat-template mismatch. Modern builds read the template from the GGUF automatically; if the file has none, name one with --chat-template. Older guides tell you to pass --jinja — check llama serve --help for whether your build still needs it.

cmake can't find a C++ compiler

Not in a Developer PowerShell, or the C++ workload wasn't installed. See §5.

Port already in use

Get-NetTCPConnection -LocalPort 8080 | Select-Object OwningProcess
Get-Process -Id <pid>

18. Quick Reference

# Install
winget install llama.cpp

# Confirm the GPU is visible
llama cli --list-devices

# Serve from Hugging Face
llama serve -hf ggml-org/gemma-4-e4b-it-GGUF:Q4_0

# Serve a local file
llama serve -m .\models\model.gguf -c 16384 -ngl all --port 8080

# Router mode: all cached models, loaded on demand
llama serve

# Interactive chat
llama cli -m .\models\model.gguf

# Benchmark
llama-bench -m .\models\model.gguf

# Health check
Invoke-RestMethod http://127.0.0.1:8080/health

Sources

• Official docs: llama.app/docs
• Server guide: llama.app/docs/serve
• CLI guide: llama.app/docs/cli
• Build guide: docs/build.md
• Full server flag reference: tools/server/README.md
• Releases: github.com/ggml-org/llama.cpp/releases