Running Embeddings Locally for Self-Hosted Search

Keyword search fails in a very specific and irritating way. I search my notes for “server won’t boot” and find nothing, because the note I wrote three years ago says “the box refuses to POST after a power cut”. Same problem, no shared words, and full-text search — which matches strings — has no idea the two are related. Every self-hosted search box I’ve run has this hole: it finds documents that contain your words and misses documents that contain your meaning. Embeddings are how you close it, and the model that produces them will run happily on the same homelab hardware as everything else.

The pitch for local semantic search is the same as for the rest of a private AI setup. The content you most want to search well is usually the content you’d least like to index on someone else’s servers — personal notes, homelab documentation, a family wiki, years of writing. A hosted embedding API means streaming all of that text to a third party every time you re-index. A local embedding model means the vectors are computed on hardware you own, stored in a database you run, and queried offline. This is the piece that sits underneath both a search box and a retrieval-augmented chatbot, so it’s worth understanding on its own terms before you build anything on top of it.

What an embedding actually is

An embedding model takes a chunk of text and returns a vector — a fixed-length list of numbers, a few hundred of them — positioned in a high-dimensional space so that texts about similar things land near each other. “The server won’t boot” and “the box refuses to POST” produce vectors that sit close together even though they share no words, because the model was trained on enough language to learn that they mean roughly the same thing. Search becomes geometry: embed the query, find the nearest stored vectors, return the documents they came from.

The number that matters is the dimension — how many numbers are in each vector. nomic-embed-text, a sensible homelab default that runs under Ollama, produces 768-dimension vectors. Smaller models like all-MiniLM-L6-v2 produce 384 and are faster and lighter; larger models like the bge and gte families go higher and score better on benchmarks at the cost of memory and speed. The dimension is fixed per model and it must match everywhere: the column in your database, the index, and every vector you ever compare. Mixing dimensions is the first mistake everyone makes, and it fails loudly, which is a small mercy.

The rule that fails quietly, and therefore hurts more, is that you must embed your documents and your queries with the same model. Two different models produce coordinates in two incompatible spaces, so the distance between a document embedded by one and a query embedded by another is meaningless noise that happens to be a number. If your search suddenly returns garbage after you “upgraded” the embedding model, this is why: you changed the model for queries and forgot to re-embed the documents.

Why local, and why this is cheap to run

Embedding is far lighter than text generation, which is the pleasant surprise. Generating a paragraph makes a model produce hundreds of tokens one after another; embedding a chunk is a single forward pass that spits out one vector. A modest GPU embeds thousands of chunks a minute, and even on CPU a small embedding model is perfectly usable for a home-sized corpus. You do not need the hardware that a chat model wants, which makes this the cheapest local-AI project on the shelf.

Running it locally also removes the recurring cost that kills hosted semantic search for hobby use. Re-indexing is not a one-off — you re-embed whenever you change the model, change your chunking, or add a batch of content — and a hosted API charges you every time. Locally, re-indexing the whole corpus overnight costs electricity and nothing else, so you experiment with chunk sizes and models the way you can only experiment when the meter isn’t running.

Storing vectors where you already keep data

You need somewhere to put the vectors that can answer “which stored vectors are nearest to this one?” quickly. A dedicated vector database like Qdrant does this well, and I use it for large indexes. For a homelab that already runs PostgreSQL, though, the pgvector extension is the pragmatic choice: it adds a vector column type and nearest-neighbour operators to a database you already back up, monitor, and know how to tune. One fewer moving part is worth a lot.

Enable the extension and create a table with a vector column matching your model’s dimension:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE documents (
    id       bigserial PRIMARY KEY,
    path     text NOT NULL,
    chunk    text NOT NULL,
    embedding vector(768)          -- must match nomic-embed-text's 768
);

-- HNSW index for fast approximate nearest-neighbour search by cosine distance.
CREATE INDEX ON documents
    USING hnsw (embedding vector_cosine_ops);

The index choice matters. Without an index, pgvector does an exact search that scans every row — fine for a few thousand chunks, slow at a hundred thousand. The HNSW index (available in pgvector 0.5 and later) builds a graph that finds approximate nearest neighbours fast, trading a sliver of recall for a large speed-up. Pick the operator class to match how you’ll query: vector_cosine_ops for cosine distance, which is the usual choice for text embeddings, and query with the matching <=> operator. Get the operator and the index out of sync and Postgres silently ignores the index and scans the table, which looks like “pgvector is slow” and is actually “the index is never used”.

Now embed and store. This uses Ollama’s embedding endpoint and writes straight into pgvector:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

import glob, requests, psycopg2

OLLAMA = "http://ollama.mylab.local:11434"
db = psycopg2.connect("dbname=search user=search host=db.mylab.local")

def embed(text):
    r = requests.post(f"{OLLAMA}/api/embeddings",
                      json={"model": "nomic-embed-text", "prompt": text})
    return r.json()["embedding"]        # a 768-element list

def chunk(text, size=1200, overlap=200):
    step = size - overlap
    return [text[i:i+size] for i in range(0, len(text), step)]

cur = db.cursor()
for path in glob.glob("notes/**/*.md", recursive=True):
    with open(path, encoding="utf-8") as f:
        for piece in chunk(f.read()):
            cur.execute(
                "INSERT INTO documents (path, chunk, embedding) VALUES (%s, %s, %s)",
                (path, piece, embed(piece)),
            )
db.commit()

Querying is the mirror image: embed the search phrase, ask Postgres for the nearest chunks by cosine distance, and let the HNSW index do the work.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

def search(query, k=5):
    vec = embed(query)
    cur.execute(
        "SELECT path, chunk FROM documents "
        "ORDER BY embedding <=> %s::vector LIMIT %s",
        (vec, k),
    )
    return cur.fetchall()

for path, chunk in search("server won't boot"):
    print(path)

That query now finds the note that says “refuses to POST”, because the geometry decides what’s near whatever words the note happened to use. The <=> operator is cosine distance; ORDER BY ... LIMIT is the nearest-neighbour search, and with the HNSW index in place it stays fast as the table grows.

Hybrid search, because semantic alone isn’t enough

Pure semantic search has a blind spot of its own: exact tokens. Search for an error code, a part number, or a person’s surname and vector similarity can drift to things that are conceptually near and miss the document containing the literal string. Keyword search is perfect at exactly this and hopeless at meaning, so the strong self-hosted search boxes run both and blend the results.

Postgres makes this convenient because it has full-text search built in. Run the tsvector keyword query and the pgvector semantic query, then combine the two ranked lists — reciprocal rank fusion is a simple, effective way to merge them without hand-tuning weights. You get the surname and the paraphrase, which is what people actually expect from a search box that feels smart. Building this on one database you already run is the quiet argument for pgvector over a separate vector service.

Troubleshooting: where local search goes wrong

Every result is irrelevant after a change. You almost certainly embedded documents and queries with different models, or different versions of one model. Cosine distance across incompatible embedding spaces is noise. Re-embed the whole corpus with the exact model you query with, and pin the model version so an Ollama update doesn’t silently swap it under you.

Inserting a vector throws a dimension error. Your column is vector(768) and the model returned a different length, or vice versa. Print len(embed("test")) and make the column, the index, and the model agree. This one fails immediately, which is the good kind of failure.

Search is slow on a big table. The index isn’t being used. Run EXPLAIN on the query — if you see a sequential scan instead of an index scan, your query’s distance operator doesn’t match the index’s operator class. A cosine index (vector_cosine_ops) is only used by the <=> operator; query with <-> and Postgres scans the whole table. Match them.

Recall is poor — the right document exists but never surfaces. Two usual causes. Your chunks are too large, so each vector is a blurry average of several ideas and matches nothing sharply; shrink the chunks and re-index. Or the HNSW index is tuned for speed over accuracy — raising the search-time ef_search parameter trades a little latency for finding more true neighbours.

Cosine results look wrong despite everything matching. Some models expect their vectors normalised to unit length before comparison. nomic-embed-text and most sentence-transformer models handle this sanely, but if you’ve wired up a model that doesn’t, un-normalised vectors make cosine distance misbehave. Normalise on the way in and stay consistent.

Re-running the indexer duplicates everything. The naive insert above appends on every run. Key each chunk by a stable identifier — a hash of path plus position — and upsert, so re-indexing overwrites rather than piling up near-duplicates that crowd the top of every result list.

Where this sits in the bigger picture

Local embeddings are the foundation that several other projects stand on. The same index that powers this search box is the retrieval half of a chatbot grounded in your own files — a local RAG stack for chatting with your own documents reuses everything here and adds a language model on top. If you take the pgvector route, the search quality and speed you get depend on the database being configured sensibly for your hardware, which is exactly what Postgres tuning for homelabbers walks through — a badly-tuned Postgres makes even a well-built index feel sluggish.

Is it worth it, and who for

Add local semantic search if you have a real corpus you search often and keyword matching keeps letting you down — notes, a wiki, documentation, a personal archive. It’s the lightest local-AI project going, it runs on hardware you already own, and the payoff is a search box that finds things by what they mean, offline, with nothing indexed on anyone else’s servers.

Skip it if your content is small enough that grep or your existing full-text search already finds everything, because embeddings add moving parts for a problem you don’t have. And be honest that semantic search complements keyword search rather than replacing it; the setups that feel genuinely good run both, so budget for the hybrid step if you want it to impress.

For me this earned its keep the day it found a note I’d have sworn I never wrote, from a phrase that shared not one word with the query. The model runs under my desk, the vectors live in a database I already back up, and the whole thing answers with the network cable on the floor where I can see it.