llama.cpp Deep Dive: Inference on a CPU

The box that runs my local LLM experiments most often isn’t the one with the GPU. It’s an old Xeon file server with sixteen threads and a pile of RAM that was otherwise sitting there doing nothing more strenuous than serving SMB shares, and it happily runs a 7B model at a pace that’s genuinely usable for anything that isn’t real-time chat. That’s llama.cpp’s whole pitch: a C++ inference engine written to run transformer models efficiently on hardware that was never meant for machine learning, no CUDA required, no GPU purchase required, no cloud bill required.

It’s easy to undersell how significant that is. Before llama.cpp, running an LLM locally more or less meant owning a capable GPU, because the reference implementations (Hugging Face transformers and friends) are built around GPU tensor operations and treat CPU execution as an afterthought fallback path. llama.cpp is written from the ground up to make CPU inference actually fast, and it’s the engine underneath Ollama — if you’ve run Ollama, you’ve run llama.cpp, just with a friendlier wrapper on top.

Why CPU inference works at all

The obvious objection is that CPUs are much slower than GPUs at the matrix multiplications a transformer is built from, which is true — a GPU has thousands of cores doing simple arithmetic in parallel, while a CPU has a handful of cores doing much more general work. What makes llama.cpp viable anyway is a combination of aggressive quantisation and low-level optimisation that closes the gap further than you’d expect.

Quantisation — covered in more detail in the quantisation explainer — does double duty here. It’s a memory saver too; a 4-bit weight needs to move a quarter of the memory bandwidth a 16-bit weight does, and memory bandwidth is very often the actual bottleneck for LLM inference generating one token at a time, ahead of raw compute. A CPU with a fast memory subsystem running 4-bit weights can end up compute-bound less often than you’d think, because the weights simply have less distance to travel per token generated.

On top of that, llama.cpp is hand-optimised at a level most inference frameworks don’t bother with: hand-written SIMD kernels (AVX2, AVX-512, NEON on ARM) for the core matrix operations, careful cache-aware memory layout, and a build system that compiles specifically for the CPU features your hardware actually has rather than shipping a lowest-common-denominator binary.

GGUF and mmap: why loading is nearly instant

GGUF, llama.cpp’s model format, stores quantised weights in a layout designed to be memory-mapped directly rather than parsed and copied into a separate in-memory structure. When llama.cpp opens a GGUF file, it uses mmap() to map the file straight into the process’s address space — the OS handles paging weight data in from disk as it’s actually needed, rather than the application reading the entire multi-gigabyte file into RAM up front.

This has two real benefits. Model load time is close to instant regardless of file size, because mmap() doesn’t actually read the file — it just sets up the address mapping and lets page faults pull data in on demand. And multiple processes loading the same model file share the underlying page cache, so if you’re running several llama.cpp instances against the same GGUF file (unusual, but it happens when testing), the OS doesn’t duplicate that memory.

The trade-off is that the first pass through a model’s weights involves real disk I/O as pages fault in, so the very first inference after a cold start is slower than subsequent ones once the pages are cached in RAM. On a system with enough spare RAM to hold the whole model in page cache, this cost is paid once and forgotten; on a RAM-constrained system, you can end up paying it repeatedly as pages get evicted and re-faulted.

Threads, offload, and getting the configuration right

Three flags control most of what matters for CPU performance, and getting them wrong is the single biggest reason people conclude “CPU inference is too slow” when it’s actually just misconfigured.

1
2
3
4
5
6
7

./llama-server \
  -m ./models/llama-3.1-8b-instruct-Q4_K_M.gguf \
  --threads 8 \
  --threads-batch 8 \
  --ctx-size 4096 \
  --n-gpu-layers 0 \
  --port 8080

--threads should roughly match your physical core count rather than your logical thread count (hyperthread/SMT siblings share execution resources and rarely help matrix-heavy workloads the way a genuinely separate core does). Setting it too high causes threads to contend for cache and memory bandwidth, and I’ve measured worse throughput from an over-threaded configuration than a correctly-sized one on the same box.

--n-gpu-layers is the offload control, and it’s the reason “CPU inference” and “GPU inference” aren’t actually a binary choice with llama.cpp — you can offload some transformer layers to a GPU while the rest run on CPU, letting a small or older GPU that can’t hold the whole model still contribute meaningfully. Setting this to 0 forces pure CPU; setting it to a high number like 999 offloads as many layers as VRAM allows and runs the remainder on CPU, which is the sensible default on a mixed system rather than an all-or-nothing choice.

--ctx-size matters more on CPU than people expect, because the KV cache for the context window competes with the model weights for both RAM and memory bandwidth. A smaller context window isn’t just about fitting the conversation — it measurably speeds up generation on CPU-bound setups by reducing the working set that has to move through cache on every token.

Building for your actual hardware

The prebuilt binaries are fine for trying llama.cpp out, but building from source with the right CPU feature flags is worth the five minutes it takes, because the project’s build system will detect and enable instruction sets a generic binary won’t assume are present:

1
2
3
4
5
6
7
8

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

cmake -B build -DGGML_NATIVE=ON
cmake --build build --config Release -j $(nproc)

# quick sanity check it picked up the right instruction sets
./build/bin/llama-cli --version

-DGGML_NATIVE=ON tells the build to target the exact CPU it’s compiled on, enabling whatever the newest available SIMD instruction set is (AVX-512 on a recent Intel/AMD chip, for instance) rather than the safer AVX2 baseline a generic build assumes. On the Xeon box I mentioned earlier, a native build measurably outperformed the generic prebuilt binary on the same model and quant level — not a marginal difference, closer to 20-30% depending on the prompt.

Where CPU inference actually breaks down

This is the section that matters most, because CPU inference has real limits and pretending otherwise sets the wrong expectations. Token generation on CPU is fundamentally memory-bandwidth-bound, and CPU memory bandwidth — even on a good dual-channel or quad-channel system — is an order of magnitude below a GPU’s. A GPU with several hundred GB/s (or over a terabyte, on recent cards) of memory bandwidth will generate tokens far faster than a CPU with 40-80GB/s, and no amount of thread tuning closes that gap because it’s a fixed hardware ceiling.

In practical terms: a 7B model in Q4 on a decent modern CPU generates somewhere in the range of 5-15 tokens per second depending on the chip and memory configuration — usable for a single interactive conversation, tedious for anything expecting near-instant responses, and hopeless if you’re serving more than one concurrent request. That last point matters a lot: CPU inference has none of vLLM’s continuous-batching advantage, because the bottleneck it optimises around (concurrent GPU compute utilisation) doesn’t apply the same way to a bandwidth-bound CPU workload. Two concurrent requests on a CPU roughly halve the tokens-per-second each one gets, rather than sharing spare compute the way a GPU batch can.

Larger models make the honest case against CPU inference even starker. A 70B model, even heavily quantised, needs enough RAM to hold tens of gigabytes of weights and enough memory bandwidth to move them repeatedly per token — technically possible on a well-specced server, but the tokens-per-second at that point drop low enough that it stops being interactive and becomes a batch job you check back on.

Batch size and prompt processing versus generation

It’s worth separating two phases of inference that behave very differently on CPU, because conflating them is a common source of “my benchmark doesn’t match yours” confusion. Prompt processing — chewing through the tokens you send in, before the model starts producing output — is compute-bound and parallelises well, because every token in the prompt can be processed against the same set of weights simultaneously. Generation, producing the response one token at a time, is memory-bandwidth-bound for the reasons already covered, because each new token depends on the one before it and the weights have to be re-read from memory on every single step. This is why a long prompt against a short expected answer (summarising a big log file into one sentence, say) behaves very differently on CPU than a short prompt with a long expected answer — the first is dominated by the fast, parallel prompt-processing phase, the second by the slow, serial generation phase. If your benchmarking only measures tokens-per-second on generation, you’re getting an accurate but incomplete picture; a workload that’s mostly prompt processing can feel considerably faster in practice than the generation-only number suggests.

--batch-size and --ubatch-size control how prompt processing is chunked, and increasing them (within the limits of available RAM) generally speeds up that phase specifically without touching generation speed at all, since it’s a different code path exercising a different hardware bottleneck. This is one of the more common misunderstandings I see: someone tunes --batch-size expecting faster token generation and gets no improvement, because they tuned the phase that was never the bottleneck for their workload.

A concrete comparison worth running yourself

Numbers from someone else’s benchmark are a starting point, not a substitute for testing your own hardware, because memory channel count, clock speed, and even how populated your DIMM slots are will shift the result meaningfully. The comparison I’d actually recommend running before committing to a CPU-only setup: load the same GGUF model at Q4_K_M and Q8_0 on your target box, and measure both cold-start latency and steady-state tokens-per-second on a representative prompt length for your use case. Q8_0 roughly doubles the memory footprint and bandwidth demand of Q4_K_M for a modest quality gain, and on a bandwidth-constrained box that trade rarely earns its keep — but “rarely” isn’t “never”, and a workload that’s unusually sensitive to quantisation-induced quality loss (structured extraction from messy input, say, rather than casual chat) might justify the slower, higher-precision option. Run both, time both, and decide with your own numbers rather than a rule of thumb from a blog post — including this one.

Troubleshooting

Generation is far slower than the model’s reported benchmarks. Check --threads against nproc --all versus physical cores — on a hyperthreaded system these numbers differ, and setting --threads to the SMT-inflated count is the most common single mistake. Also verify the build actually used GGML_NATIVE or the appropriate -march flag; a generic binary quietly leaves real performance on the table.

RAM usage keeps climbing during a long conversation. The KV cache grows with context length, and a long-running chat session accumulates it across every turn — this is expected growth rather than a leak. Reduce --ctx-size if it’s a problem, or restart the session periodically if you’re running near your RAM ceiling.

Model loads instantly but the first response takes far longer than subsequent ones. That’s the mmap() cold-start cost — pages are faulting in from disk on first touch. If it stays slow well past the first request after a fresh boot, check available RAM; a system tight on memory evicts cached pages and pays that cost repeatedly instead of once.

Offloading some layers to GPU makes generation slower. This usually means the GPU is a weak or memory-starved one and the constant data transfer between CPU and GPU per token is costing more than the GPU’s compute saves. Try --n-gpu-layers 0 as a baseline comparison — if pure CPU wins, the GPU in that box isn’t worth using for this model.

Build fails or crashes with an illegal instruction error on an older CPU. GGML_NATIVE=ON targets the build machine’s CPU exactly — if you built on one machine and are running the binary on an older one, it may have compiled in instructions the target CPU doesn’t support. Build on the target machine, or specify feature flags explicitly rather than NATIVE.

Verdict

llama.cpp is the reason “I don’t have a GPU” stopped being a hard blocker for running local LLMs, and for a single-user, single-conversation-at-a-time workload on a 7B-or-smaller model, CPU inference through llama.cpp is genuinely pleasant to use on decent modern hardware. Where it stops being the right tool is concurrency and scale — memory bandwidth is a hard ceiling that no configuration tuning gets around, and anyone serving multiple users or running larger models is better served putting an actual GPU behind vLLM instead. But as the engine quietly running underneath Ollama on every CPU-only homelab box out there, it’s already doing more work than it gets credit for.