A team I spoke with last month was convinced their agentic customer support system was underperforming because their embedding model wasn’t good enough. They switched from text-embedding-ada-002 to a newer model, saw marginal improvement, and moved on—never realizing that their retrieval pipeline was stuffing 40 chunks into every prompt and actively confusing the model. The embedding model wasn’t the problem. Their assumptions about RAG were.
These misconceptions aren’t hypothetical. They show up in real architectures, cause real quality degradation, and persist because RAG seems to work well enough in demos that the failure modes stay hidden until production. Let’s clear them up.
Myth #1: More Retrieved Chunks Always Improve Accuracy
The intuition here makes sense: give the model more context and it has more to work with. In practice, this is one of the most reliably damaging things you can do to a RAG pipeline.
The problem has a name—lost in the middle—and it’s been documented empirically. Models like GPT-4 and Claude are significantly better at using information at the beginning and end of a long context window than they are at using information buried in the middle. When you retrieve 30 chunks and concatenate them, you’re not giving the model 30 opportunities to get the answer right. You’re creating a long context where the most relevant passage might sit at position 14 and get systematically underweighted.
What actually works:
- Retrieve more, pass fewer. Use a reranker—Cohere’s Rerank API or a cross-encoder like those available through
sentence-transformers—to retrieve 20 candidates and pass the top 4-6 to the model. - Test with ground truth. Build a small evaluation set and measure answer quality as you increase chunk count. The curve almost always peaks well before you’d expect.
- In agentic contexts, retrieve iteratively. An agent that retrieves 5 chunks, reasons about them, then decides whether to retrieve more almost always outperforms one that dumps 25 chunks upfront.
The right mental model isn’t “more context is better.” It’s “the right context, in the right position, in the right amount.”
Myth #2: RAG Eliminates Hallucination
This one causes the most damage because it leads teams to remove the guardrails they’d otherwise build. RAG reduces certain types of hallucination—specifically, confabulation about facts the model was never trained on. It does not prevent the model from hallucinating about the retrieved content itself.
I’ve seen production systems where a model is handed a retrieved passage that says “the refund window is 30 days” and responds to the user with “you have 45 days to request a refund.” The model didn’t ignore the context—it partially processed it and blended it with prior training beliefs. This is especially common when retrieved content contradicts something the model learned during pretraining, when the retrieved passage is long and the relevant detail is a specific number or date, or when multiple retrieved chunks contain superficially similar but subtly different information.
RAG also does nothing about retrieval failures. If the right document isn’t in your index, or your chunking strategy split a critical table across two chunks that are never retrieved together, the model will often generate a plausible-sounding answer from priors rather than saying it doesn’t know.
The actual mitigation stack: faithfulness evaluation (using tools like RAGAS or TruLens to measure whether the answer is grounded in the retrieved context), explicit “I don’t know” training or prompting, and citation enforcement that makes hallucination visible when it does occur.
Myth #3: Vector Similarity Equals Relevance
Cosine similarity between embedding vectors is a proxy for semantic relatedness. It is not the same thing as “this chunk will help the model answer this question.”
Consider a query like “why did our API latency spike on November 12th?” A vector search might surface chunks about API latency in general, historical performance discussions, or architecture documents—all semantically close—while missing the actual incident postmortem that answers the question, because the postmortem was written in a different register (“on 11/12 we observed degraded response times due to…”) that creates distance in embedding space.
Keyword overlap matters. Recency matters. Document structure matters. The relationship between a question and its answer is not always the same as the relationship between two similar pieces of text.
What this means practically:
- Hybrid search is usually better than pure vector search. Combining BM25 (keyword) with dense retrieval—as Elasticsearch, Weaviate, and Qdrant all support natively—consistently outperforms either approach alone on real-world corpora.
- Metadata filtering can do more than embedding can. If you know a query is about an incident from a specific date, filtering by document date before embedding search is more reliable than hoping the date is encoded in the semantic similarity.
- Cross-encoders see the query and chunk together. Unlike bi-encoders that embed query and document separately, cross-encoders evaluate them jointly and consistently outperform similarity-based ranking when relevance depends on the specific relationship between question and answer.
Myth #4: RAG and Fine-Tuning Solve the Same Problem
Teams treat this as a budget question: “Should we fine-tune or build RAG?” They’re not the same solution. They address fundamentally different failure modes.
RAG is about knowledge access. Use it when the model needs information it doesn’t have—proprietary documents, recent events, user-specific data, large knowledge bases that can’t fit in a context window.
Fine-tuning is about behavior and format. Use it when the model knows the relevant facts but isn’t responding in the right way—wrong tone, wrong output structure, wrong reasoning pattern, inconsistent adherence to domain conventions.
A legal AI that needs to cite specific case law needs RAG. A legal AI that keeps writing in casual language when it should write in formal legal prose needs fine-tuning. A legal AI that needs to do both needs both—and the teams that combine them, using fine-tuning to improve how the model reasons over retrieved context, consistently outperform teams that treat the choice as either/or.
In agentic systems specifically, fine-tuning on tool use patterns and retrieval decision-making—when to retrieve, how to formulate retrieval queries—is often more valuable than fine-tuning on domain knowledge. The knowledge lives in the index. The reasoning lives in the model.
The Common Thread
Every one of these myths comes from treating RAG as a black box that you plug in and trust. In production agentic systems, retrieval quality is not a one-time configuration decision—it’s a continuous measurement problem. Build evaluation into your pipeline early, instrument what’s actually being retrieved, and test your assumptions about chunk count, similarity thresholds, and retrieval triggers with real queries before they hit users.
The teams getting the most out of RAG aren’t the ones using the newest embedding models. They’re the ones who stopped assuming and started measuring.
