A Local RAG Stack: Chatting With Your Own Documents Offline

I have roughly a decade of accumulated documentation rotting in a folder: network diagrams I drew and forgot, config notes for services I no longer run, receipts, manuals, a wiki export, and about four hundred markdown files of the sort you write at 1am and never look at again. Searching it with grep works if I remember the exact word I used. Most of the time I don’t. What I actually want is to ask “how did I wire up the backup job for the media box?” in plain English and get an answer that quotes my own notes back at me.

That is what retrieval-augmented generation (RAG) does. A language model on its own is a confident stranger who has never seen your files; it will happily invent an answer about your backup job because inventing plausible text is the only thing it does. RAG fixes that by fetching the relevant passages from your documents first and handing them to the model as context, so the model summarises what it was given rather than hallucinating from thin air. The interesting part, for anyone who cares about where their data goes, is that every piece of this can run on a box under your desk with the network cable pulled out.

Why bother running it locally

The obvious answer is privacy. The documents I most want to query are the ones I would least like to paste into someone else’s web form: infrastructure notes, financial records, half-finished writing, correspondence. A hosted assistant means uploading all of that to a company whose business model I don’t control and whose retention policy I can’t audit. A local stack means the embeddings, the index, and the model weights all sit on hardware I own, and I can verify the isolation by unplugging the machine and watching it keep working.

The less obvious answer is that local RAG makes you honest about how the technique actually behaves. When you pay per token you are tempted to treat the model as an oracle and move on. When you are watching your own GPU fan spin up, you pay attention to what got retrieved, why the answer was wrong, and how much of the context window you just wasted on irrelevant chunks. You learn the machinery, and the machinery is where all the quality lives.

The pieces, and what each one is for

A RAG stack is four moving parts and some glue. Understanding what each does saves you from cargo-culting a framework you can’t debug.

A language model runner. This loads the model weights and answers prompts. Ollama is the least painful option for a homelab because it wraps llama.cpp, pulls quantised models with one command, and exposes an HTTP API. On a machine with a 12GB GPU I run a 7-billion-parameter model like Mistral 7B comfortably; on CPU alone it works but you will be waiting.

An embedding model. This is the part people skip past and then wonder why retrieval is useless. An embedding model turns a piece of text into a vector — a list of a few hundred numbers — positioned so that passages about similar things land near each other. nomic-embed-text runs under Ollama and is a sensible default. The rule that bites everyone: you must embed your documents and your questions with the same model, because two different models produce coordinates in two incompatible spaces.

A vector store. This holds the embeddings and answers the question “which twenty chunks are nearest to this query vector?” Qdrant runs as a single container and is happy with millions of vectors. For a few thousand documents, a local SQLite-backed store or Chroma is plenty. I cover the retrieval side in more depth in running embeddings locally for self-hosted search; the same index that powers a search box powers RAG.

An ingest and orchestration layer. This walks your files, splits them into chunks, embeds each chunk, and stores it. At query time it embeds the question, fetches the nearest chunks, stuffs them into a prompt, and calls the model. You can write this yourself in about a hundred lines of Python, or lean on LlamaIndex or LangChain if you want batteries included. I have done both. The framework saves you a day and then charges it back the first time you need to understand why a retrieval went wrong.

Standing up the infrastructure

The two long-running services — the model runner and the vector store — sit in a compose file. Everything else is a script you run on demand.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

services:
  ollama:
    image: ollama/ollama:latest
    volumes:
      - ./ollama:/root/.ollama
    ports:
      - "11434:11434"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

  qdrant:
    image: qdrant/qdrant:latest
    volumes:
      - ./qdrant:/qdrant/storage
    ports:
      - "6333:6333"

Bring that up, then pull the two models you need:

1
2
3

docker compose up -d
docker compose exec ollama ollama pull mistral
docker compose exec ollama ollama pull nomic-embed-text

At this point nothing on the machine needs the internet again. Pull the cable, run your ingest, and ask questions offline — the whole reason we are here.

Ingesting documents without making a mess

The ingest step is where RAG quality is won or lost, and the culprit is almost always chunking. Embed a whole 4,000-word document as one vector and you get a blurry average of everything it says, so retrieval finds it for every vaguely related question and never for the specific fact you wanted. Split it into overlapping windows of a few hundred tokens and each chunk carries one coherent idea, which is exactly what nearest-neighbour search rewards.

Here is the shape of a minimal ingest against the stack above. It is deliberately plain so you can see every step rather than trusting a framework to do it invisibly:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

import glob, requests
from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct

OLLAMA = "http://localhost:11434"
client = QdrantClient(url="http://localhost:6333")

client.recreate_collection(
    "docs",
    vectors_config=VectorParams(size=768, distance=Distance.COSINE),
)

def embed(text):
    r = requests.post(f"{OLLAMA}/api/embeddings",
                      json={"model": "nomic-embed-text", "prompt": text})
    return r.json()["embedding"]

def chunk(text, size=1200, overlap=200):
    step = size - overlap
    return [text[i:i + size] for i in range(0, len(text), step)]

point_id = 0
for path in glob.glob("docs/**/*.md", recursive=True):
    with open(path, encoding="utf-8") as f:
        for piece in chunk(f.read()):
            client.upsert("docs", [PointStruct(
                id=point_id,
                vector=embed(piece),
                payload={"path": path, "text": piece},
            )])
            point_id += 1

Chunking on character count is crude; splitting on paragraph or heading boundaries produces cleaner chunks because it respects where one idea stops and the next begins. Start crude, measure the retrieval quality, and refine only the part that is actually letting you down.

Asking a question

Querying is the mirror image: embed the question, fetch the nearest chunks, and build a prompt that tells the model to answer from the supplied context and say so if the context is silent. That last instruction is what keeps a grounded system honest.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

def ask(question):
    hits = client.search("docs", query_vector=embed(question), limit=5)
    context = "\n\n---\n\n".join(h.payload["text"] for h in hits)
    prompt = (
        "Answer using only the context below. "
        "If the answer is not in the context, say you don't know.\n\n"
        f"Context:\n{context}\n\nQuestion: {question}"
    )
    r = requests.post(f"{OLLAMA}/api/generate",
                      json={"model": "mistral", "prompt": prompt, "stream": False})
    return r.json()["response"]

print(ask("how did I wire up the backup job for the media box?"))

If you would rather have a chat window than a Python REPL, point a front end at the same Ollama instance. Open WebUI as a private ChatGPT front end has a document feature that does a lightweight version of all this in the browser, which is a fine way to try the idea before you commit to building the pipeline yourself.

Troubleshooting: where local RAG actually goes wrong

Almost every RAG failure is a retrieval failure wearing a generation costume. The model looks like it lied; usually it was handed rubbish and did its best.

The answer is confidently wrong. Print the retrieved chunks before you blame the model. Nine times in ten the right passage was never fetched, so the model answered from context that didn’t contain the fact. Fix retrieval — better chunking, more results, a better embedding model — before you touch the prompt.

Retrieval returns nothing relevant. Check that you embedded documents and queries with the same model. Mixing nomic-embed-text for ingest and something else for queries produces vectors in incompatible spaces, and cosine similarity between them is meaningless noise. If they match and results are still poor, your chunks are probably too large; shrink them and re-ingest.

The prompt overflows the context window. A 7B model with a small context window will silently truncate an over-stuffed prompt, dropping exactly the chunk you needed. Count your tokens. Retrieving five chunks of 1,200 characters plus the question and instructions should fit comfortably; retrieving twenty will not, and more retrieved chunks is not automatically better.

Everything is correct and it’s just slow. On CPU, generation is the bottleneck and there is no clever fix beyond a smaller or more heavily quantised model. On GPU, first check the model actually loaded onto the card — nvidia-smi should show VRAM in use while it answers. If it didn’t, Ollama fell back to CPU and you’re paying for a GPU you aren’t using.

Re-ingesting duplicates everything. The naive script above appends on every run. Key each chunk by a stable hash of its file path plus position so a re-ingest overwrites rather than piling up near-duplicates that crowd out real results.

Is it worth it, and who for

This is worth building if you have a real corpus you actually query and a genuine reason to keep it off other people’s servers — personal notes, homelab documentation, records you consider nobody else’s business. The payoff is a private assistant that answers from your own words and improves as you refine the ingest.

It is not worth it if you have fifty files and a good memory; plain full-text search will serve you better and cost you nothing. It is also a poor fit if you need answers about current events or general knowledge, because a small local model grounded only in your documents knows nothing outside them, and that is the point rather than a bug.

For me the tradeoff lands firmly on the build side. The stack is three containers and a script, it runs on hardware I already own, and it turns a decade of forgotten notes into something I can interrogate in a sentence — with the network cable on the floor where I can see it.