Self-Hosted-Ai - vo.rs

llama.cpp Deep Dive: Inference on a CPU

Tue, 18 Nov 2025 11:20:00 +0000

The box that runs my local LLM experiments most often isn’t the one with the GPU. It’s an old Xeon file server with sixteen threads and a pile of RAM that was otherwise sitting there doing nothing more strenuous than serving SMB shares, and it happily runs a 7B model at a pace that’s genuinely usable for anything that isn’t real-time chat. That’s llama.cpp’s whole pitch: a C++ inference engine written to run transformer models efficiently on hardware that was never meant for machine learning, no CUDA required, no GPU purchase required, no cloud bill required.

Quantisation Explained: GGUF, GPTQ and Small VRAM

Tue, 14 Oct 2025 08:45:00 +0000

The first time I tried to run a 13B model on a card with 12GB of VRAM, I hit an out-of-memory error before the model had finished loading. The maths is unforgiving: a 13-billion-parameter model in the FP16 precision it was likely trained and released in needs roughly 26GB just to hold the weights, before you’ve allocated a single byte for the context window. My card had less than half that. Quantisation is the technique that made it possible anyway, and understanding what it actually does — rather than treating it as a magic “make it smaller” checkbox — is the difference between picking a sensible trade-off and quietly running a model that’s dumber than it needs to be.

LiteLLM: One API Gateway for Every Model

Tue, 09 Sep 2025 10:00:00 +0000

Somewhere around my fourth self-hosted model backend, I noticed every one of my scripts had a slightly different way of talking to it. The Ollama box used its native API for one script and the OpenAI-compat shim for another. The vLLM server took a different base URL and a different model-name format. And two of my tools still called out to a cloud provider directly whenever the local box was overloaded, with their own hard-coded API key sitting in a config file I kept forgetting was there. None of this was hard, individually. Collectively, it was the kind of accumulated friction that means you stop experimenting because switching backends is annoying.

vLLM: Serving Local Models Fast Enough to Use

Tue, 12 Aug 2025 09:30:00 +0000

I ran Ollama on a spare box with a 3090 for the better part of a year and thought I understood local LLM performance. Then I pointed three people at the same endpoint at once and watched throughput fall off a cliff. Ollama is brilliant for a single user pulling one model and asking it one thing at a time. It is not built for concurrency, and pretending otherwise wastes a GPU that cost more than the rest of the rack combined.