vLLM: Serving Local Models Fast Enough to Use

I ran Ollama on a spare box with a 3090 for the better part of a year and thought I understood local LLM performance. Then I pointed three people at the same endpoint at once and watched throughput fall off a cliff. Ollama is brilliant for a single user pulling one model and asking it one thing at a time. It is not built for concurrency, and pretending otherwise wastes a GPU that cost more than the rest of the rack combined.

vLLM exists to fix exactly that problem. It is an inference server built around two ideas — PagedAttention and continuous batching — that together let one GPU serve many simultaneous requests without the throughput collapse you get from naive batching. If you are running a model for more than yourself, or building anything that fires off concurrent completions (a RAG pipeline, a Slack bot with more than one user, a batch-summarisation job), vLLM is the difference between a GPU that earns its keep and one that mostly idles waiting for the previous request to finish.

Why naive serving falls apart under load

To understand why vLLM matters, it helps to know what a naive server does wrong. Every transformer generates tokens one at a time, and at each step it needs the key/value (KV) cache — the attention state — for every token generated so far. The KV cache grows linearly with sequence length, and a framework that allocates it the simple way reserves a contiguous block of GPU memory sized for the maximum possible sequence length, for every request, whether or not the request ever gets that long.

That is enormously wasteful. A chat request that finishes in 200 tokens still ties up memory reserved for 4,096 tokens for its entire lifetime. Multiply that across ten concurrent requests and you run out of VRAM long before you run out of compute, so the server can only handle a couple of requests at once regardless of how much raw throughput the GPU actually has.

The second naive-serving failure is static batching. If a server batches requests together to keep the GPU busy, it typically waits for a full batch to arrive, runs it, and only starts the next batch once every request in the current one has finished — including the slow ones. One user asking for a 2,000-token essay blocks everyone else’s short completions until it’s done. That’s the opposite of what you want from a shared service.

PagedAttention: memory management borrowed from operating systems

vLLM’s core contribution is PagedAttention, which treats the KV cache the way an OS treats virtual memory: instead of one contiguous allocation per request, the cache is split into fixed-size blocks (pages) that get allocated on demand and can live anywhere in GPU memory. A logical sequence is mapped to a list of physical blocks via a lookup table, exactly like a page table maps virtual addresses to physical ones.

This has two practical consequences. First, memory is only allocated as a sequence actually grows, so a short completion doesn’t reserve space for a long one it will never need — vLLM’s own benchmarks showed 2–4x more requests fitting in the same VRAM compared with naive allocators. Second, because blocks are addressed indirectly, vLLM can share blocks between sequences that have identical prefixes — the system prompt in a RAG pipeline, or the shared prefix in beam search — without copying the KV cache for each one. For a self-hosted setup where every one of your requests reuses the same system prompt, that sharing alone can be worth a meaningful chunk of your throughput budget.

Continuous batching: never wait for the slowest request

The second piece is continuous (sometimes called “in-flight”) batching. Instead of running fixed batches to completion, vLLM’s scheduler evaluates the batch at every generation step. As soon as a request finishes, its slot is freed and a new request from the queue is inserted immediately, mid-batch. The GPU never sits idle waiting for one long-running generation to catch up with everything else, and short requests are not held hostage by long ones.

Combined with PagedAttention’s efficient memory use, this is what actually gives you the throughput numbers vLLM is known for: several times the requests-per-second of naive Hugging Face transformers serving, and comfortably ahead of Ollama’s llama.cpp-based backend once you have more than one concurrent user. Ollama is still the right tool for a single-user chat box on a laptop — it isn’t trying to solve this problem, and shouldn’t be judged against a server that is.

Getting it running

vLLM ships an OpenAI-compatible server out of the box, which is the detail that made switching painless for me — anything already speaking to /v1/chat/completions (a UI, a script, LangChain, whatever) keeps working with a base-URL change.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# on a box with an NVIDIA GPU and recent drivers
python3 -m venv vllm-env
source vllm-env/bin/activate
pip install vllm

# serve a model straight from the Hugging Face hub (or a local path)
vllm serve mistralai/Mistral-7B-Instruct-v0.3 \
  --host 0.0.0.0 \
  --port 8000 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192 \
  --dtype auto

That last flag, --gpu-memory-utilization, controls how much of the card’s VRAM vLLM pre-allocates for the KV cache pool — set it too high and you’ll starve anything else running on the same GPU; too low and you cap your concurrent-request ceiling. I run 0.85–0.90 on a dedicated inference box and drop to 0.6 on a card I’m sharing with a Jellyfin transcode job.

A docker-compose deployment is closer to what I actually run in the homelab, fronted by a reverse proxy for TLS:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

services:
  vllm:
    image: vllm/vllm-openai:latest
    restart: unless-stopped
    ports:
      - "127.0.0.1:8000:8000"
    volumes:
      - ./models:/root/.cache/huggingface
    environment:
      - HF_TOKEN=${HF_TOKEN}
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    command: >
      --model meta-llama/Llama-3.1-8B-Instruct
      --gpu-memory-utilization 0.85
      --max-model-len 8192

Bind it to 127.0.0.1 and put a reverse proxy like Caddy in front if you need it reachable outside the host — vLLM’s server has no auth of its own beyond an optional static API key, and you don’t want an inference endpoint with no rate limiting sitting on the open internet.

Querying it is identical to querying OpenAI, just with a different base URL:

1
2
3
4
5
6
7

curl http://mylab.local:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Summarise PagedAttention in two sentences."}],
    "max_tokens": 200
  }'

VRAM realities

This is where I have to be honest about the trade-off, because vLLM’s throughput advantage comes with a real memory cost. Continuous batching and the KV cache pool mean vLLM wants to reserve a large, mostly-full chunk of VRAM up front rather than growing usage lazily the way llama.cpp does. An 8B model in FP16 needs roughly 16GB just for weights, and vLLM’s default memory-utilisation target adds a substantial cache pool on top — on a 24GB card that’s workable for one 8B model with headroom for a handful of concurrent requests, but you will not casually run two model instances alongside it the way you might juggle model swaps in Ollama.

The practical guidance: if you have a single user and a single conversation at a time, Ollama’s simplicity and lower idle footprint win — there’s no reason to fight vLLM’s memory pre-allocation for a workload that never has more than one request in flight. If you have concurrent users, an application making several calls in parallel, or you’re running structured batch jobs (classifying a folder of documents, say), vLLM’s throughput lead is large enough to matter, and quantised weights (GPTQ or AWQ rather than GGUF, which vLLM doesn’t consume natively) get the VRAM ceiling back down to something a single consumer GPU can carry.

Troubleshooting

Server starts, then OOMs on the first real request. --gpu-memory-utilization is a target covering weights plus the KV cache pool, which vLLM sizes based on --max-model-len — it’s easy to assume it only accounts for weights. Drop --max-model-len before you drop the utilisation fraction; a smaller context window shrinks the cache pool far more than shaving a few percent off the utilisation target does.

CUDA out of memory on model load, but nvidia-smi shows the card is free. Something else claimed the memory and then crashed without releasing it cleanly — a leftover Python process from a previous vLLM run is the usual suspect. nvidia-smi will show the process still holding memory even if it’s zombied; kill -9 the PID and retry.

Throughput is fine with one request but doesn’t scale with concurrency. Check you’re actually sending concurrent requests — a naive test script that awaits each curl sequentially will never trigger continuous batching, because there’s only ever one request in flight. Use a proper load tool (hey, wrk, or a Python script with asyncio.gather) to fire genuinely parallel requests.

Model loads but generation is much slower than the benchmarks suggested. Check --dtype. If you left it on auto and your GPU doesn’t support bfloat16 well (older Pascal-generation cards, for instance), vLLM may fall back to a slower path. Also check you’re not accidentally running in FP32 because a config file specified it explicitly.

Multiple models needed, but VRAM only fits one at a time. vLLM doesn’t do dynamic model swapping the way Ollama does — each vllm serve invocation loads one model and holds it. Run separate instances on separate ports if you have the VRAM, or put LiteLLM in front to route between multiple backend servers, only one of which needs to be vLLM.

Tuning beyond the defaults

Once the basic server is stable, a handful of flags make the difference between “works” and “works well” under real load. --max-num-seqs caps how many sequences vLLM will admit into a batch simultaneously — the default is generous, but on a card with limited VRAM you’ll hit memory pressure before you hit that cap, so it’s worth lowering explicitly rather than relying on the scheduler to discover the ceiling itself. --swap-space controls how much CPU RAM vLLM can use to page out KV cache blocks under memory pressure rather than rejecting a request outright; a few gigabytes here smooths out bursty traffic without needing more VRAM.

For prefix-heavy workloads — a RAG pipeline where every request shares the same retrieved-context template, or an agent loop that resends a long system prompt each turn — --enable-prefix-caching turns on the block-sharing behaviour PagedAttention makes possible. I saw a noticeable latency drop on a document-QA setup where the retrieved chunks were often identical across nearby queries; the shared prefix’s KV cache only had to be computed once. It costs nothing when prefixes don’t actually match, so there’s little reason to leave it off.

Watch nvidia-smi dmon or vLLM’s own Prometheus metrics endpoint (/metrics) while under load rather than guessing — the server exposes queue depth, cache utilisation, and time-to-first-token directly, and tuning blind against those numbers beats tuning against vibes.

Verdict

vLLM is worth the extra setup complexity the moment more than one thing talks to your inference server at once. It is not a drop-in Ollama replacement for a laptop or a single-user box — the memory pre-allocation and lack of automatic model swapping make it a worse fit there, and Ollama’s simplicity wins fair and square for that use case. But if you’re serving a household, a small team, or an application making concurrent calls, the throughput gap is not subtle, and PagedAttention’s memory efficiency is the reason a single GPU can carry that load at all. I run it now for anything that isn’t purely “me, one chat window, one question at a time” — which turned out to be most of what I actually use local LLMs for.