When RAG Isn't Enough

I The Retrieval Quality Ceiling

Retrieval-Augmented Generation changed the game. Instead of hallucinating answers, the model looks things up first. But "looking things up" is harder than it sounds.

A basic RAG pipeline — split documents into chunks, embed them, retrieve the top-K by cosine similarity — gets you surprisingly far. For FAQ-style questions where the answer lives in a single clean paragraph, it works beautifully. The trouble starts when it doesn't: when the answer spans two paragraphs, when the user's phrasing doesn't match the document's vocabulary, when the most relevant passage ranks sixth out of five retrieved.

The ceiling is retrieval quality. Give the model perfect context and even a mediocre model produces excellent answers. Give a frontier model garbage context and you get confidently wrong answers with citations. Garbage in, eloquent garbage out. And the stakes go up fast: HBR draws a sharp line between content risk (the AI says something wrong) and execution risk (the AI does something wrong). A RAG system answering questions sits in content-risk territory. The moment that same system starts taking actions based on its own retrieved context — filing tickets, issuing refunds, updating records — bad retrieval becomes execution risk, and the severity class changes entirely.

The rest of this article covers the four techniques that close the gap: chunking strategies, hybrid search, reranking, and query rewriting. Each one addresses a different failure mode. Together, they turn a decent RAG system into a great one.

II Chunking Strategies

Before you can retrieve anything, you have to decide how to slice your documents. This is chunking, and it matters more than most teams realize.

Fixed-size chunkingis the default: split every 200 characters or 500 tokens regardless of content. It's fast and predictable, but it splits sentences mid-thought and frequently slices the answer to a question across two separate chunks. When the retriever pulls chunk 7 but the answer actually spans chunks 7 and 8, the model gets half the context it needs.

Sentence-boundary chunkingis a step up. Each chunk ends at a sentence boundary, so you never split a thought mid-sentence. But individual sentences are often too small to be useful — a standalone sentence like "The return window is 30 days" has no surrounding context to help the embedder understand what product or policy it refers to. This is what researchers call context fragmentation — enterprise data is contradictory and versioned, and a chunk without its surrounding policy context can lead the model to act on outdated or incomplete information. That's not a hallucination; it's a retrieval mistake with real consequences.

Semantic chunkinggroups related sentences together by measuring embedding similarity between consecutive sentences. When two adjacent sentences are semantically similar, they stay in the same chunk. When there's a topic shift, a new chunk begins. This produces chunks that map to coherent ideas rather than arbitrary boundaries.

The right strategy depends on your content. Technical documentation with clear section headers benefits from structure-aware chunking. Conversational transcripts work well with semantic chunking. Legal contracts might need overlapping windows so no clause gets orphaned. Try it yourself:

III Hybrid Search

Most RAG tutorials use vector search — embed the query, find the nearest document embeddings, return the top-K. This works well for semantic similarity: "How do I cancel my subscription?" matches a document about "ending your membership" even though they share no keywords. But it fails on the exact queries that users expect to work best.

Search for "error code E-4012" and vector search might return documents about error handling in general rather than the specific error code. Search for "HIPAA compliance requirements" and you'll get results about healthcare privacy that never mention HIPAA by name. Embeddings capture meaning, not exact terms.

Keyword search (BM25)is the opposite. It excels at exact matches: product names, error codes, legal terms, acronyms. It has decades of engineering behind it and is fast. But it can't bridge vocabulary gaps — if the user says "cheap flights" and the document says "budget airfare," keyword search returns nothing.

Hybrid search runs both in parallel and fuses the results. The most common approach is Reciprocal Rank Fusion (RRF): each search method ranks its results, and the final score is the sum of inverse ranks. A document that ranks #1 in keyword and #5 in vector gets a combined score that reflects both signals. This way, exact-match queries are handled by BM25, semantic queries are handled by vectors, and ambiguous queries benefit from both.

The implementation cost is minimal — you're already running vector search, and BM25 is a well-supported algorithm in every search library. The quality improvement is typically 10–25% on recall benchmarks. Explore the difference below:

IV Reranking

Initial retrieval — whether keyword, vector, or hybrid — is designed to be fast. It scans millions of documents in milliseconds using approximate methods. The tradeoff is precision. The top 10 results are "probably relevant," but their ordering is rough. The best answer might sit at position 7.

A rerankeris a second-stage model — typically a cross-encoder — that takes the query and each retrieved document as a pair and scores their relevance with much higher accuracy. Cross-encoders are too slow to run over your entire document store (they process one pair at a time), but they're perfect for rescoring 10–20 candidates. The most relevant result jumps from position 7 to position 1. Marginal results drop to the bottom.

The pattern is simple: retrieve broadly, then rerank precisely. First-stage retrieval casts a wide net (top-20 or top-50). The reranker narrows it down to the 3–5 best results that actually go into the LLM's context. This two-stage approach consistently outperforms single-stage retrieval, often by 15–30% on precision metrics.

Watch reranking in action:

V Query Rewriting

Every technique so far improves what happens after the search runs. Query rewriting improves what goes intothe search. Users are terrible at writing search queries. They use ambiguous pronouns ("How do I fix it?"), abbreviations ("What's the ETA on the SOW?"), and compound questions that should be three separate searches ("Compare the pricing, features, and support tiers of our enterprise plan vs. the competitor").

A query rewritersits between the user and the retrieval pipeline. It takes the raw user query and transforms it into one or more optimized search queries. The simplest version expands abbreviations and adds context from the conversation: "How do I fix it?" becomes "How do I fix the authentication timeout error in the mobile app?" when the conversation history mentions that error. More advanced rewriters decompose complex questions into sub-queries and run each one independently.

The four most common rewriting patterns:

1. Expansion.Add context from conversation history. "What about pricing?" becomes "What is the pricing for the Enterprise plan discussed earlier?"
2. Decomposition. Break compound queries into atomic searches. One question becomes three, each with its own retrieval pass.
3. Hypothetical Document Embedding (HyDE). Ask the LLM to write a hypothetical answer, then embed that instead of the question. A fake answer is often closer in embedding space to the real answer than the question is.
4. Step-back prompting.Abstract the query to a higher level. "Why did revenue drop in Q3?" becomes "What factors affect quarterly revenue?" to retrieve background context before answering the specific question.

VI What's Next

You now know how to build retrieval that's genuinely good — not just "works in the demo" good, but "handles real user queries in production" good. Better chunking, hybrid search, reranking, and query rewriting are the tools that get you there.

But even with perfect retrieval, the model can still go wrong. In Part 3: Hallucinations, Safety & Trust, we tackle the output side: detecting and preventing hallucinations, building safety guardrails, establishing trust with users, and the human-in-the-loop patterns that keep AI systems accountable. Grounding gets the right information in. Guardrails keep the wrong information out.