Adding RAG to Local LLMs: A Practical Guide

A local LLM running on your own hardware answers from training data alone. It does not know your contracts, your tickets, or last week's pricing decision. Retrieval-augmented generation (RAG) fixes that by letting the model search your documents and answer from them. This guide walks through the indexing, embedding, and retrieval pieces, the Qdrant vs pgvector decision, and the mistakes that quietly wreck accuracy.

6 min read8 sectionsJune 22, 2026

Kostadin Stamboliev

Founder and Imaginator

LinkedIn

Founder and Imaginator at Pineido. Practitioner across IT consulting, app and web development, technical SEO, and AI / automation. Ships production code in 30-day sprints with fixed scope and clear ownership.

What RAG actually does for a local model

A local LLM (run through Ollama, LM Studio, or similar) only knows what it learned during training. Ask it about your internal runbook or a customer's service history and it either refuses or invents an answer. RAG closes that gap. Before the model writes a single token, the system searches a knowledge base built from your documents, pulls the few passages most relevant to the question, and pastes them into the prompt as context. The model then answers from that evidence rather than from memory.

The payoff for running this locally is that private documents never leave your hardware. There are no per-token API fees, no data-sharing terms to negotiate with a vendor, and latency is dominated by your own GPU or CPU rather than a network round trip. For a service business sitting on contracts, SOPs, and historical tickets that cannot go to a third party, this is often the only acceptable way to put an LLM near sensitive data. The trade-off is that you now own the full pipeline, and the pipeline is where the quality lives.

The pipeline: ingest, chunk, embed, index, retrieve

Every RAG system follows the same shape: ingestion, chunking, embeddings, indexing, retrieval, reranking, context construction, then generation. Skip or botch a step early and no amount of model quality recovers it. The widely repeated rule is that if you only fix one thing, fix chunking, because it defines the units of knowledge the system can ever retrieve.

Ingestion loads raw files (PDFs, Markdown, database rows) and strips them to clean text. Chunking splits that text into retrievable pieces. Embedding turns each chunk into a vector of numbers that captures its meaning. Indexing stores those vectors so they can be searched fast. Retrieval finds the chunks closest to the question. An optional reranking step reorders the candidates with a more precise model. Finally, the top few chunks are assembled into the prompt and handed to the local LLM.

The mental trap is treating this as a one-time script. In practice it is a loop you tune against a fixed set of test questions, changing one variable at a time and measuring whether retrieval improved.

Chunking: the highest-leverage decision

Chunks that are too large drown the model in irrelevant text; chunks that are too small lose the context that made a passage meaningful. The 2026 consensus default is a recursive splitter at roughly 400-512 tokens with 10-20% overlap, which handles the majority of use cases before you need anything fancier. Most systems perform well in the 200-500 token range, and factoid-style questions tend to favor the smaller end of that.

Overlap exists to stop a key sentence from being cut in half at a chunk boundary; a 500-token chunk with 50-100 tokens of overlap is a reasonable starting point. But treat overlap as a tunable parameter, not a law. A January 2026 analysis found cases where overlap added indexing cost with no measurable retrieval benefit, so validate it against your own questions rather than assuming it always helps.

Whatever you choose, attach metadata to every chunk: document title, section, page number, and a chunk index. That metadata lets you filter retrieval ("only search the 2026 contracts") and lets the model cite where an answer came from, which is what turns a demo into something a team will trust.

Choosing a local embedding model

The embedding model decides whether semantically similar text actually lands near each other in vector space. For local setups in 2026, three names dominate. Nomic-embed-text is the safe CPU-friendly default: around 137M parameters, roughly 274MB on disk, an 8,192 token context window, and by far the most-pulled embedding model on Ollama. One configuration gotcha is that its native context is 8,192 tokens but the Ollama model card lists 2,048, so set num_ctx if you want the full window.

BGE-M3 has become the production default for serious RAG because it supports dense and sparse retrieval from a single model, enabling hybrid search without a separate keyword index, and it covers 100-plus languages. If you have the VRAM and want the highest open-source quality, Qwen3-Embedding (the 8B variant scored 70.58 on the MTEB multilingual leaderboard, ranking first as of June 2025) now competes with commercial embedding APIs.

One caveat worth internalizing: lightweight English-focused models like nomic and mxbai-embed-large score near zero on cross-lingual tasks and degrade on very long documents. If your corpus is multilingual or document-heavy, reach for BGE-M3 or Qwen3 rather than the smallest option.

Qdrant or pgvector: how to actually decide

This is the question people overthink. For a typical local RAG setup under roughly 5 million vectors, the database is almost never your bottleneck. A 2026 benchmark on a few thousand documents found Qdrant and pgvector effectively tied on latency when both use HNSW indexes, with run-to-run variance larger than the gap between them. The real cost is embedding and generation, not search. A vector query in the 45-80ms range disappears inside an LLM response that takes one to three seconds.

So the decision is operational. Choose pgvector if you already run PostgreSQL: same backups, same monitoring, same credentials, no second service to restart at 2am. It is the pragmatic default for most teams under a few million vectors. Choose Qdrant when you are vector-first and at scale: it is purpose-built in Rust, benchmarks at the top of ANN throughput tests on million-vector workloads, ships built-in quantization to cut memory, and its localhost dashboard is genuinely useful for debugging what is in your index.

The sane migration path is to start on pgvector if Postgres is already in your stack, and move to Qdrant only if you cross into tens of millions of vectors or need heavy concurrent query throughput. That migration is tractable; premature optimization is not.

Retrieval quality: hybrid search and reranking

Pure vector search has a real failure mode: it conflates precise terms, IDs, and acronyms that happen to be semantically similar. Hybrid search fixes this by combining keyword search (BM25 or SPLADE) with vector search, then fusing the two ranked lists, most commonly with Reciprocal Rank Fusion. This is exactly why BGE-M3's built-in sparse support is attractive: one model serves both halves.

Reranking is the step that pushes a system from "topically related" to "actually answers the question." First-stage retrieval is a fast bi-encoder that scores query and passages separately; a cross-encoder reranker scores them jointly and reorders the candidates, which can meaningfully lift retrieval quality. The catch is latency: cross-encoders are expensive, so retrieve a wide net (say the top 30 from each method) and rerank down to a small final set. In one production system, passing about 5 documents to generation was the sweet spot: enough context for multi-part questions without cluttering the prompt or inviting noise.

Do not add all of this on day one. Layer it in, measure each change against a set of golden questions using Recall@K or NDCG, and stop when accuracy is good enough.

A workable local stack and its gotchas

A common 2026 local stack is Ollama for both embedding and generation inference, LlamaIndex for the ingestion and indexing pipeline (it ships hundreds of document connectors and handles chunking cleanly), and pgvector or Qdrant as the store. For orchestration beyond simple retrieval, many teams pair LlamaIndex retrieval with LangGraph. For a single-user static corpus, FAISS with no server process is the lightest option; Chroma is friendlier when multiple processes need access.

The gotchas are mundane and they bite hard. Ollama needs two separate models pulled, one for embeddings and one for generation, and forgetting to pull the embedding model (or pointing at the wrong base URL) produces silent empty retrievals that look like a broken database. On 8GB of RAM, drop the generation model to something like a 3B variant; embedding quality is unaffected and only complex reasoning suffers slightly. And plan for production realities from the start: retrieval drift as the corpus changes, no evaluation loop, and no way to trace why a given answer happened are the failure modes that show up after launch, not before.

Where this connects to the rest of your AI setup

RAG is the layer that makes a local model useful on your own data, but it sits on top of decisions you ideally made earlier: which model, what hardware, how it is served, and how it stays private. If you have not stood up the base local model yet, that comes first; RAG is the second move, not the first.

The parts that consume the most time in practice are not glamorous. Cleaning and chunking source documents, picking and testing an embedding model against your actual questions, and building an evaluation loop so you can tell whether a change helped are where weeks go. That is also where a fixed-scope sprint earns its keep: a focused 30-day engagement can stand up the indexing pipeline, wire in Qdrant or pgvector, tune retrieval against your real corpus, and hand you something your team can run, rather than a clever demo that quietly degrades the moment the documents change.

Ready to ship?

Tell us where you are stuck. We will review it and send a focused recommendation within one business day.

Get operations insights weekly

Join service-business operators who get one actionable insight on ops, automation, and growth every week.

Checking that you're human. You can send in a moment.

No spam. Unsubscribe anytime.

You’re subscribed! Look for our next issue in your inbox.

Frequently asked questions

Do I need a GPU to run RAG on a local LLM?

Not strictly. Embedding models like nomic-embed-text run on a laptop CPU, and small generation models work on modest hardware (a 3B model fits in 8GB of RAM). A GPU mainly speeds up generation and lets you run larger, more capable models. For embedding throughput on a big corpus, a GPU helps but is not required for a working system.

Should I use Qdrant or pgvector for a local RAG system?

Under roughly 5 million vectors the two perform similarly, so the choice is operational. If you already run PostgreSQL, use pgvector to avoid a second service. Choose Qdrant when you are vector-first and at scale (tens of millions of vectors, heavy concurrent queries, or you want built-in quantization). Starting on pgvector and migrating later is a tractable path.

Which embedding model is best for a local RAG setup?

Nomic-embed-text is the safe CPU-friendly default. BGE-M3 is the 2026 production pick because it handles dense and sparse retrieval and 100-plus languages from one model. Qwen3-Embedding offers the highest open-source quality if you have the VRAM. Avoid lightweight English-only models for multilingual or very long documents, where they degrade sharply.

What chunk size and overlap should I start with?

Start with a recursive splitter at about 400-512 tokens with 10-20% overlap; that handles most cases. Smaller chunks improve precision for factoid questions, larger ones preserve context. Treat overlap as a tunable parameter, since some workloads see no benefit from it, and always attach metadata like title, section, and page to every chunk.

My local RAG gives bad answers. Where do I start debugging?

Almost always retrieval, not the model. Confirm the embedding model was actually pulled and is reachable, since a wrong base URL produces silent empty retrievals. Check that chunking is not cutting context in half, add hybrid search if precise terms get missed, and add a cross-encoder reranker passing about five documents to generation. Measure each change against fixed test questions.

Is a local RAG setup truly private?

Yes, when every component runs locally: the LLM, the embedding model, and the vector database all on your own hardware. No documents, queries, or embeddings leave your machine, and there are no per-token fees. The trade-off is that you own the full pipeline, including indexing, tuning, and an evaluation loop to keep accuracy from drifting as documents change.