The RAG That Searched the Wrong Space

  • Right source in the top five: from 17% to 94%
  • 5.5× more questions answered from the right document
  • Trusted by the support team again in 2 weeks
  • 17% to 94% Questions with the right source in the top five
  • 5.5× More questions answered from the right document
  • 2 weeks Until the support team trusted it again

What changed

Questions with the right source in the top five
17% to 94%
More questions answered from the right document
5.5×
Until the support team trusted it again
2 weeks
Places that set the embedding model and size
3 to 1
Real questions in the guard test set
120
Full re-embedding on their own database
Overnight

What we walked into

A software company built an assistant for its support team: ask a question, get an answer drawn from the product documentation and past tickets, with the sources linked. Retrieval-augmented generation, RAG. The demo had looked good. In production the assistant answered confidently and cited documents that had nothing to do with the question. Support agents stopped trusting it within two weeks. The team spent a month rewriting prompts, trying a larger model and adjusting how many documents to retrieve. Each change moved the answers around without making them right.

What was actually broken

When a RAG assistant cites the wrong sources, the language model is usually not the culprit. It can only answer from what retrieval hands it, and here retrieval was handing it passages that were barely related. The prompt work and the bigger model were treating the last step of the pipeline for a fault in the first one.

We checked the vectors themselves. The documents had been embedded months earlier with a model configured to produce 2048-dimensional vectors. Later, the database column and the query code were set to 1024 dimensions, to save storage and speed up search. The ingestion job had been adjusted to cut each stored vector down to fit the column, while the questions were embedded natively at 1024. Stored documents and incoming questions were now vectors from two different spaces. The search still returned its “nearest” passages, because a distance can always be computed, but nearness between those two spaces meant very little.

Nothing failed loudly. No error, no crash, every search returned results in milliseconds. That is what made it expensive: the system looked healthy on every dashboard while the one thing users cared about, finding the right document, was close to random. The embedding model and its dimensions were set in three different places, and they had drifted apart.

Step by step

Test retrieval on its own

We collected 120 real questions from the support team and, for each, the document an expert would cite. Then we measured retrieval alone, with no language model involved: does the right document appear in the top five results? For only 17% of the questions. That number, not the answers, became the measure of every fix.

One owner for the embedding setting

The embedding model and its dimension count now live in a single configuration that both ingestion and querying read. The code refuses to store or search with a vector whose length does not match that setting, so the two sides can never drift apart silently again.

Re-embed and rebuild the index

We re-embedded the whole corpus with the same model and the same 1024-dimension setting the queries use, then rebuilt the vector index on the new column. It ran overnight on the company’s own Supabase database, with the old index kept until the new one passed the test set.

Keep the test set as a guard

The question set now runs whenever the embedding setup, the chunking or the documents change. If retrieval quality drops, the change does not ship. Retrieval went from 17% to 94% on the same 120 questions. Only then did we go back to the prompts, and most of the earlier prompt changes turned out not to be needed.

17% to 94% Questions with the right source in the top five

Get in touch

Tell us what you want built. We tell you where to start and what it takes.

The questions that decide if we are right for you

How can different embedding dimensions break retrieval without an error?

A vector search always returns the closest matches it can compute. If the stored documents and the incoming questions come from different embedding setups, the distances are still numbers, so results come back fast and look normal. They are just not meaningfully related. Only a check of whether the right document comes back reveals it.

Is a smaller embedding size worse?

Not necessarily. Many current embedding models are trained to work at several sizes, and 1024 dimensions is plenty for most document sets. What breaks things is mixing sizes or models between the documents and the questions. Pick one setting, use it on both sides, and measure retrieval on your own questions.

Do we need to re-embed everything after changing the model?

Yes. Vectors from different models or settings are not comparable, so a change to the embedding model or its size means re-embedding the whole corpus and rebuilding the index. Done on your own database, it is usually an overnight job, and a test set tells you when the new index is ready to take over.

ByFounder and Imaginator

Published