When a RAG assistant cites the wrong sources, the language model is usually not the culprit. It can only answer from what retrieval hands it, and here retrieval was handing it passages that were barely related. The prompt work and the bigger model were treating the last step of the pipeline for a fault in the first one.
We checked the vectors themselves. The documents had been embedded months earlier with a model configured to produce 2048-dimensional vectors. Later, the database column and the query code were set to 1024 dimensions, to save storage and speed up search. The ingestion job had been adjusted to cut each stored vector down to fit the column, while the questions were embedded natively at 1024. Stored documents and incoming questions were now vectors from two different spaces. The search still returned its “nearest” passages, because a distance can always be computed, but nearness between those two spaces meant very little.
Nothing failed loudly. No error, no crash, every search returned results in milliseconds. That is what made it expensive: the system looked healthy on every dashboard while the one thing users cared about, finding the right document, was close to random. The embedding model and its dimensions were set in three different places, and they had drifted apart.