llama.cpp vs Ollama vs vLLM: Picking a Local Inference Server

Somebody in a forum thread asks which local inference server they should use, and gets forty replies naming three projects, all of them correct. This happens because the question has a hidden variable: what does your hardware look like, and how many people are asking it questions at once? Answer those and the choice makes itself.

I’ve run all three in the same cupboard over the last few months, on the same hardware, serving the same models. Here’s what actually separates them.

The one-paragraph version

llama.cpp is a C++ inference library that will run a model on almost anything with a CPU, and will use your GPU if you let it. Ollama wraps llama.cpp in a daemon, a model registry and an HTTP API, so you type one command instead of six. vLLM is a Python serving engine built for GPUs with enough VRAM to hold the whole model, and it exists to serve many concurrent requests efficiently.

The dividing line between the first two and the third is quantisation. llama.cpp and Ollama are built around GGUF, a format designed so a 13B model fits in 8 GB of VRAM at some cost to quality. vLLM is built around models in their native fp16, where a 13B needs about 26 GB before you’ve stored a single token of context. That single architectural fact predicts most of the rest.

llama.cpp: the engine everything else is built on

Georgi Gerganov’s project started as an experiment in running LLaMA on a MacBook and turned into the substrate for the entire local-model ecosystem. Its defining trick is aggressive quantisation plus the ability to split a model across CPU and GPU — you tell it how many layers your VRAM can hold, and the rest run on CPU.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make LLAMA_CUBLAS=1 -j$(nproc)

# 35 of 41 layers on the GPU, the rest on CPU
./server \
  -m models/mistral-7b-instruct-v0.1.Q4_K_M.gguf \
  --n-gpu-layers 35 \
  --ctx-size 4096 \
  --host 127.0.0.1 --port 8080 \
  --threads 8

That --n-gpu-layers flag is the whole value proposition. With 8 GB of VRAM and a 13B model, you can’t fit everything, so you fit what you can and the CPU handles the remainder at maybe a fifth of the speed. The result is slow. The result also exists, which is more than the alternatives manage on that hardware.

The reasons to use llama.cpp directly rather than through a wrapper come down to control. Every knob is exposed: rope scaling, KV cache type, batch size, NUMA placement, grammar-constrained output via GBNF. If you want the model to emit only valid JSON, llama.cpp can enforce it at the sampler level, which is a stronger guarantee than asking politely in a prompt. I’ve written more about what the engine is doing internally, and that detail is exactly what a wrapper hides.

The reasons to avoid it: you build it from source, you track a project that lands breaking changes weekly, and you manage GGUF files by hand. The build flags change. The server’s API has changed. This is a project moving fast enough that “I updated it and something broke” is a normal Tuesday.

Ollama: llama.cpp with the sharp edges filed off

Ollama bundles llama.cpp, a model registry, automatic layer-offload sizing, and a REST API into a single daemon. The pitch is that everything above becomes:

1

ollama run mistral

and it works out how many layers fit in your VRAM without asking you. For most people, most of the time, its guess is as good as the number you’d have picked.

What you get beyond convenience:

• Model management. ollama pull, ollama list, ollama rm. The registry hosts sensible default quantisations, so mistral gets you Q4_0 without a trip to Hugging Face to work out which of nineteen files you want.
• Modelfiles. A Dockerfile-shaped way to bake a system prompt and sampler settings into a named model. Genuinely useful — a model called summarise that’s Mistral plus a fixed prompt is nicer than repeating the prompt in six scripts.
• Automatic unloading. Idle models get evicted from VRAM after five minutes, which matters if you use the same card for anything else.
• A stable-ish API. /api/generate and /api/chat have kept their shape while llama.cpp’s server has moved around.

What you give up: the knobs. Ollama exposes a fraction of llama.cpp’s parameters, and when its offload heuristic guesses badly, arguing with it is harder than passing a flag. It also has no authentication whatsoever, which is fine on loopback and becomes your problem the moment you put it on the network.

The other Ollama caveat worth knowing: it serves requests against a loaded model one at a time. Two people asking questions means the second one waits. For a household, that’s invisible. For anything else, it’s the wall you hit.

vLLM: the one that assumes you have real hardware

vLLM comes out of Berkeley and its contribution is PagedAttention — managing the KV cache in fixed-size blocks the way an OS manages virtual memory pages, instead of pre-allocating one contiguous slab per request. The consequence is that memory fragmentation stops wasting most of your VRAM, and continuous batching becomes practical: new requests join the running batch at the next token step rather than queueing behind a whole generation.

1
2
3
4
5
6
7
8

pip install vllm

python -m vllm.entrypoints.openai.api_server \
  --model mistralai/Mistral-7B-Instruct-v0.1 \
  --dtype float16 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 4096 \
  --port 8000

That’s an OpenAI-compatible endpoint, so anything written against the OpenAI SDK points at it with a changed base URL:

1
2
3
4
5
6
7

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistralai/Mistral-7B-Instruct-v0.1",
    "messages": [{"role": "user", "content": "Summarise PagedAttention in two sentences."}],
    "max_tokens": 128
  }'

The hard requirement: the model lives entirely in VRAM. There is no CPU offload, no partial loading, no graceful degradation. A 7B at fp16 needs about 15 GB plus cache, so a 16 GB card is tight and a 24 GB card is comfortable. vLLM has AWQ 4-bit support, which brings that down considerably, though AWQ checkpoints are less widely available than GGUF and the quality trade-offs are less well documented.

--gpu-memory-utilization 0.90 is worth understanding: vLLM claims that fraction of VRAM up front and manages it internally. If the card also drives your display, or another container wants it, 0.90 will end your evening. On a shared card I run 0.70 and accept the smaller cache.

Where vLLM wins is under concurrency. One request against a 7B on a 24 GB card is roughly comparable to Ollama on the same hardware — vLLM ahead, though not by an amount that changes your life. Put eight simultaneous requests against it and the picture inverts completely: vLLM’s aggregate throughput climbs while per-request speed barely moves, because they’re all riding the same batched forward pass. Ollama, meanwhile, is serving request one and making the other seven wait their turn.

The comparison table I wish I’d had

What I actually measured, and why you should distrust it

Numbers first, caveats immediately after. Same box — an 8-core CPU, 32 GB of RAM, a 12 GB consumer card — same model family, same prompt, 256 tokens requested:

The single-request column is a wash, and vLLM losing it slightly is the expected shape — batching machinery costs something when there’s nothing to batch. The concurrent column is the entire argument. And the last row is the other argument: a 13B on a 12 GB card is only possible at all because llama.cpp will spill to CPU, and it costs you five-sixths of your speed to do it.

Distrust all of this. Tokens per second depends on the quantisation, the prompt length, the sampler settings, the CUDA version, whether the card thermally throttled halfway through, and what the model was asked. My numbers describe my cupboard. The ratios travel; the absolute values do not. Run your own before you make a purchasing decision on the strength of a table on the internet, including this one.

The ones I left out

text-generation-webui wraps several backends including llama.cpp and offers a UI with every sampler exposed as a slider. It’s a superb exploration tool and an awkward thing to build a service on, since the API is a bolt-on to something designed as an interface.

LM Studio is a desktop application, closed source, genuinely pleasant for trying models on a laptop. It has a local server mode. I don’t run closed-source software on the machine in the cupboard, which is a preference rather than an argument.

TGI (Hugging Face’s Text Generation Inference) occupies roughly vLLM’s niche — continuous batching, GPU-resident models, production serving. Its licence changed to a restrictive one in mid-2023, and while that changes nothing for home use, it changed which project the ecosystem’s momentum went to, and momentum is why vLLM’s list of supported architectures grows faster.

Troubleshooting the three of them

llama.cpp: CUDA error: out of memory at a layer count that worked yesterday. The KV cache scales with --ctx-size, so raising context silently raises VRAM. Drop --n-gpu-layers by two or three before touching anything else.

llama.cpp: builds fine, runs at CPU speed anyway. LLAMA_CUBLAS=1 wasn’t picked up, usually because a stale object file survived. make clean first. The startup banner tells you the truth — if it doesn’t mention your device, the build is CPU-only.

Ollama: model runs far slower than expected. Check ollama ps for how much sits on GPU versus CPU. The heuristic reserves headroom, and on an 8 GB card that reservation can push a model that nearly fits into partial offload, which costs you most of the speed. Setting num_gpu in a Modelfile overrides the guess.

Ollama: 500 errors after a version upgrade. The registry occasionally re-publishes tags with different quantisations. ollama pull the model again rather than debugging the daemon.

vLLM: ValueError: The model's max seq len is larger than the maximum number of tokens that can be stored in KV cache. vLLM computed how much cache fits and your requested context doesn’t. Lower --max-model-len or raise --gpu-memory-utilization if there’s genuinely headroom.

vLLM: installs, then fails on import with a CUDA version mismatch. The wheels are built against specific CUDA versions and pip will cheerfully install one that doesn’t match your driver. This is the single most common vLLM complaint and the reason I run it in a container with the toolchain pinned rather than in a virtualenv on the host.

vLLM: first token takes ten seconds. It’s compiling CUDA graphs on the first request. Warm it at startup, or pass --enforce-eager to skip graph capture at some throughput cost.

Which one, then

Ollama if you have one GPU, a household of users, and other things to do with your weekend. This is most people, including me for most workloads. The automatic sizing is good, the model registry removes a genuinely tedious chore, and the API is stable enough to build against. It’s where I’d point anyone starting out with local models.

llama.cpp if your hardware is marginal or your requirements are odd. No GPU at all, or 6 GB of VRAM and a stubborn desire to run something bigger than fits, or a need for grammar-constrained output. The build is an evening; the control is permanent.

vLLM if more than a couple of requests can arrive at once and you have 24 GB of VRAM. A retrieval pipeline hitting the model per chunk, a batch job over a thousand documents, a service several people use simultaneously — that’s where continuous batching stops being a benchmark curiosity and starts being the difference between a job finishing overnight and finishing next week.

The failure mode to avoid is picking vLLM because it benchmarks fastest and then discovering your card holds a 7B at fp16 with no room for context. Throughput numbers from a datacentre card describe a machine you don’t own. Work out your VRAM first, then read the table.