LiteLLM: One API Gateway for Every Model

Somewhere around my fourth self-hosted model backend, I noticed every one of my scripts had a slightly different way of talking to it. The Ollama box used its native API for one script and the OpenAI-compat shim for another. The vLLM server took a different base URL and a different model-name format. And two of my tools still called out to a cloud provider directly whenever the local box was overloaded, with their own hard-coded API key sitting in a config file I kept forgetting was there. None of this was hard, individually. Collectively, it was the kind of accumulated friction that means you stop experimenting because switching backends is annoying.

LiteLLM is the fix: a lightweight proxy that sits in front of every model you can reach — local or cloud — and presents them all through one OpenAI-compatible API. Every client in your homelab talks to one URL, one API key format, and one model naming scheme, and LiteLLM handles routing the request to whichever backend actually serves that model. It’s the kind of infrastructure that feels unnecessary until you have three backends, at which point it feels overdue.

Why a gateway earns its keep

The appeal isn’t abstract API purity, it’s operational. Once every request goes through one proxy, you get several things for free that would otherwise mean writing (and maintaining) glue code in every application:

A single endpoint and key for anything that speaks OpenAI’s API shape — which by now is most tooling, from LangChain to a raw curl script — means adding a new backend model doesn’t touch any client code. You add it to LiteLLM’s config and every consumer sees it immediately.

Fallbacks mean a request to your local vLLM box can fail over to a cloud model automatically if the local server is down, overloaded, or the model isn’t loaded — useful if you run local-first but don’t want an outage to take down whatever depends on it. You configure the fallback chain once, centrally, instead of wiring retry logic into every script.

Cost tracking matters even in a homelab, because “self-hosted” doesn’t mean every request is free — you’re still paying a cloud provider for the requests you fall back to, and LiteLLM logs token usage and estimated spend per key, per model, and per user, so you can actually see where the cloud bill is coming from instead of guessing at the end of the month.

Per-key rate limits and budgets let you hand out scoped access — a key for a script that should never spend more than a couple of dollars a month, a separate one for interactive use — without running separate proxy instances.

Running it

LiteLLM ships as a Python package with a proxy server mode, and the docker-compose route is the one I’d actually recommend for anything long-running:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

services:
  litellm:
    image: ghcr.io/berriai/litellm:main-stable
    restart: unless-stopped
    ports:
      - "127.0.0.1:4000:4000"
    volumes:
      - ./litellm-config.yaml:/app/config.yaml
    environment:
      - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY}
      - DATABASE_URL=postgresql://litellm:${DB_PASSWORD}@postgres:5432/litellm
    command: ["--config", "/app/config.yaml"]
    depends_on:
      - postgres

  postgres:
    image: postgres:16-alpine
    restart: unless-stopped
    environment:
      - POSTGRES_USER=litellm
      - POSTGRES_PASSWORD=${DB_PASSWORD}
      - POSTGRES_DB=litellm
    volumes:
      - litellm-pg:/var/lib/postgresql/data

volumes:
  litellm-pg:

The Postgres backend isn’t optional if you want persistent key management, spend tracking, or a usage dashboard — LiteLLM can run stateless against just a YAML file for routing alone, but you lose the accounting features that make the gateway worth deploying in the first place.

The config file is where the actual routing logic lives:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

model_list:
  - model_name: local-fast
    litellm_params:
      model: openai/mistral-7b-instruct
      api_base: http://vllm.mylab.local:8000/v1
      api_key: "not-needed"

  - model_name: local-general
    litellm_params:
      model: ollama/llama3.1
      api_base: http://ollama.mylab.local:11434

  - model_name: cloud-fallback
    litellm_params:
      model: anthropic/claude-haiku-4-5
      api_key: os.environ/ANTHROPIC_API_KEY

router_settings:
  routing_strategy: usage-based-routing
  fallbacks:
    - local-fast: ["cloud-fallback"]
    - local-general: ["cloud-fallback"]

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL

Every client now points at http://litellm.mylab.local:4000/v1 with the master key (or a scoped virtual key you generate through LiteLLM’s admin API) and requests model: "local-fast" or whichever alias you’ve defined. The fallbacks block means a local-fast request that errors out — vLLM down, model not loaded, whatever — gets retried against cloud-fallback automatically, and the client never sees the failure.

1
2
3
4
5
6
7

curl http://litellm.mylab.local:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-your-virtual-key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local-fast",
    "messages": [{"role": "user", "content": "Draft a one-line changelog entry."}]
  }'

Virtual keys and budgets

Generating a scoped key for a specific script or user is a single API call once the proxy is up, and it’s the piece that turns LiteLLM from “convenient routing” into something closer to real access control:

1
2
3
4
5
6
7
8
9

curl -X POST http://litellm.mylab.local:4000/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "models": ["local-fast", "local-general"],
    "max_budget": 5,
    "budget_duration": "30d",
    "key_alias": "home-assistant-summariser"
  }'

That key can call the two local models but has no access to cloud-fallback, and it’s hard-capped at $5 of tracked spend a month even though the local models cost nothing — the budget field is really about giving each integration a blast radius as much as tracking cloud dollars. If a script goes rogue and starts hammering the API in a loop, it hits its budget and gets a 429 instead of quietly running up a bill or hammering your GPU indefinitely.

Load balancing and usage-based routing

The routing_strategy setting in the earlier config is doing more work than it might look like. usage-based-routing distributes requests across multiple deployments of the same model alias based on how much each is already handling, which matters the moment you have more than one instance capable of serving a given request. If you run two Ollama boxes on different hardware, both loaded with the same model, you can register both under one model_name and let LiteLLM spread the load rather than picking one and leaving the other idle:

1
2
3
4
5
6
7
8
9

model_list:
  - model_name: local-general
    litellm_params:
      model: ollama/llama3.1
      api_base: http://ollama-a.mylab.local:11434
  - model_name: local-general
    litellm_params:
      model: ollama/llama3.1
      api_base: http://ollama-b.mylab.local:11434

Two entries sharing a model_name is how LiteLLM expresses “these are interchangeable” — clients still just ask for local-general and never need to know which physical box answered. This is also where LiteLLM earns its keep during maintenance: pull one Ollama box out of the pool to update it, and traffic quietly shifts to the other without touching a single client config.

Latency-based routing is the other strategy worth knowing about — instead of balancing by request count, it tracks recent response times per deployment and biases new requests towards whichever backend has been answering fastest. On a homelab where one box is noticeably beefier than another, this tends to produce better real-world results than a naive round-robin, because it adapts to hardware asymmetry instead of assuming every deployment behind an alias is equally capable.

Observability

Beyond the spend and budget tracking already mentioned, LiteLLM exposes a Prometheus-compatible metrics endpoint and can log every request/response pair to a callback of your choosing — a webhook, a logging service, or a local file. For a homelab this is mostly useful for catching silent failures: a model that starts returning empty completions, or a backend that’s technically up but answering with garbage, doesn’t always throw an obvious error the way a connection refused does. Piping request logs into whatever you’re already using for monitoring means you notice a degraded model before a person does.

1
2
3

litellm_settings:
  success_callback: ["prometheus"]
  failure_callback: ["prometheus"]

That two-line addition to the config is enough to start exporting request counts, latencies, and error rates per model and per key, scrapeable by anything already pointed at your other services.

Keeping the proxy off the public internet

The compose file above binds LiteLLM to 127.0.0.1:4000 on purpose. It’s tempting to just publish the port and reach it from anywhere on the LAN, but a proxy that holds your only cloud API key and can be asked, on demand, to spend real money deserves the same caution you’d give a database. My own setup puts LiteLLM behind a reverse proxy that terminates TLS and only forwards requests carrying a valid client certificate or a Tailscale-originated connection, so the master key is never the only thing standing between “trusted request” and “anyone who guesses the URL”. A leaked virtual key is annoying and gets revoked in one API call; a leaked master key means rotating every downstream integration at once, because the master key can mint or delete any other key.

Rotation is worth practising before you need it in a hurry. LITELLM_MASTER_KEY is an environment variable, not a value baked into the config file, so rotating it is a restart with a new secret rather than an edit-and-redeploy. Virtual keys generated through the admin API can be revoked individually without touching the master key at all, which is the whole point of generating a scoped key per integration instead of handing every script the same credential — a compromised key for one automation doesn’t mean re-keying the rest of your homelab.

Streaming and tool calls

Most of what makes a modern LLM client usable — token-by-token streaming so a response starts appearing immediately, and function/tool calling so a model can request a lookup mid-conversation — passes through LiteLLM unmodified, because the proxy is deliberately thin at the protocol level rather than rewriting request bodies. A client that streams against vLLM directly streams identically against local-fast through the gateway; LiteLLM relays server-sent events rather than buffering a full response and re-emitting it, so there’s no perceptible added latency to the first token. Tool-calling support is more backend-dependent than streaming is: LiteLLM translates the OpenAI tool-call schema against each provider’s own equivalent where one exists, but a backend that doesn’t support structured tool calls at all — an older local model served through a minimal API shim, say — won’t gain the capability just because it’s sitting behind LiteLLM. The gateway can normalise the shape of a feature across providers; it can’t invent a feature a given model was never trained to use.

Troubleshooting

Requests to a local model return a connection error, but curling the backend directly works fine. Check the api_base in your model config against the network LiteLLM’s container actually sees — a localhost reference inside a docker-compose service resolves to LiteLLM’s own container rather than your host. Use the host’s LAN address or a shared docker network with the backend’s service name.

Cost tracking shows $0 for everything, including cloud calls that should have a price. LiteLLM’s pricing tables are model-name-keyed and need an exact match against its internal catalogue. A typo’d or very new model name (a cloud provider’s latest release, before LiteLLM’s next package update) falls through to untracked. Check the LiteLLM changelog for pricing-table updates before assuming your setup is broken.

Fallback never triggers even when the primary model is clearly down. Fallbacks only fire on errors LiteLLM’s router recognises as retryable — a hard connection refused counts, but a slow response that eventually times out at the client instead of the proxy might not, depending on your client’s own timeout being shorter than LiteLLM’s. Set explicit timeout values in litellm_params so the proxy detects the failure before your client gives up first.

Postgres grows unexpectedly large after a few weeks. Every request’s usage record is logged for the cost-tracking and analytics features. If you don’t need long-term per-request history, set up a retention job to prune old rows — LiteLLM doesn’t do this for you by default.

Virtual key works for one model but not another it should have access to. Double-check the models array on key generation — it’s an explicit allowlist, so a key created before you added a new model to model_list won’t automatically gain access to it.

Caching duplicate requests

One more feature that pays for itself quickly in a homelab context: LiteLLM supports response caching, backed by Redis or an in-memory store, keyed on the request content. If several of your automations end up asking near-identical questions — a home-automation script that re-summarises the same sensor log format every hour, or a documentation bot answering the same handful of common questions repeatedly — caching means the second identical request never touches the model at all.

1
2
3
4
5
6
7

litellm_settings:
  cache: true
  cache_params:
    type: redis
    host: redis.mylab.local
    port: 6379
    ttl: 3600

An hour’s TTL is a reasonable default for anything where slightly stale answers are harmless; drop it much lower for anything time-sensitive. This is one of the few LiteLLM features that saves money on cloud fallbacks and GPU cycles on local backends simultaneously, since a cache hit skips inference entirely regardless of which backend would have served it.

Verdict

LiteLLM is worth deploying the moment you have more than one model backend, which in practice is most homelabs running local AI seriously — even a setup with just Ollama and an occasional cloud fallback benefits from not hard-coding two different client integrations. The Postgres dependency and the extra hop add a small amount of latency and operational surface, and if you only ever run one model on one backend it’s genuinely unnecessary complexity — point your script straight at vLLM or Ollama and skip the proxy. But the day you add a second backend, or want to hand a scoped key to a script you don’t fully trust, the gateway pattern pays for itself immediately.