Grounding Models in Your Data

I The Grounding Problem

Ask ChatGPT to explain quantum entanglement and you'll get a polished, accurate answer. Ask it to explain your company's Q3 refund policy and you'll get something that sounds plausible but is entirely fabricated. The model speaks with the same confidence either way.

This is the grounding problem. LLMs are trained on public data — books, websites, code repositories, Wikipedia. They know the world in general. They do not know your world in particular: your internal docs, your customer data, your product specs, your Confluence pages that haven't been updated since 2022 but somehow remain the canonical source of truth for half the company.

The gap between "general knowledge" and "your knowledge" is where hallucinations thrive. The model doesn't know what it doesn't know, so it fills the void with plausible-sounding fiction. For a consumer chatbot, that's an annoyance. For an enterprise product handling customer data, billing logic, or compliance rules, it's a liability. HBR calls this content risk — the danger that a generative AI system might saysomething wrong — and it's the defining risk category for any retrieval-based product.

Grounding is the practice of connecting a model to authoritative data sources so its answers are rooted in fact rather than fabrication. And the most practical way to do it today is called RAG.

II The RAG Pipeline

Retrieval-Augmented Generation — RAG — is a deceptively simple idea: instead of asking the model to recall information from memory, you look up the relevant information first and paste it into the prompt. The model generates its answer based on what you gave it, not what it memorized during training.

Three stages make it work:

1. Embed.Convert your documents (and the user's query) into numerical representations called embeddings — arrays of numbers that capture meaning.
2. Search.Find the documents whose embeddings are closest to the query's embedding. "Closest" means most semantically similar.
3. Generate.Stuff the retrieved documents into the prompt alongside the user's question. The LLM reads everything and produces a grounded answer.

The beauty of RAG is that it separates knowledge from reasoning. The LLM provides the reasoning. Your data provides the knowledge. Update a document and the model's answers change immediately — no retraining, no fine-tuning, no waiting weeks for a new model version. You ship a doc update, and the next query reflects it.

That immediacy matters more than it sounds. Enterprise data is contradictory and versioned — a 2022 policy doc might still sit in your knowledge base alongside the 2024 replacement. As HBR researchers Telang, Hydari, and Iqbal point out, an agent acting on a stale policy isn't hallucinating — it's making a retrieval mistake with legal consequences. RAG gives you the mechanism to fix this: swap the doc, and the answers update in real time.

III Embeddings Explained

Embeddings are where language becomes math. An embedding model takes a piece of text — a sentence, a paragraph, a document chunk — and converts it into a list of numbers, typically 768 or 1,536 of them. These numbers encode meaning, not just keywords. That distinction is everything.

Consider: "How do I return a product?" and "What is your refund policy?" share zero words in common. A keyword search would miss the connection entirely. But their embeddings will be nearly identical, because the meaning is the same.

Conversely, "Java programming language" and "Java island in Indonesia" share the same keyword but have very different embeddings, because the meaning diverges.

Explore how this works in the visualization below:

Notice how the clusters form naturally. Cooking phrases group together regardless of whether they mention "saute," "grill," or "oven." Programming terms cluster even when they reference different languages. The embedding model has learned that "debug a Python script" and "fix a JavaScript bug" are semantically adjacent — something a keyword index would never catch.

IV Vector Search

Once every document chunk is an embedding — a point in high-dimensional space — searching becomes a geometry problem: find the points closest to the query. This is vector search, also called nearest-neighbor search.

The math is straightforward. You compute the cosine similarity between the query embedding and every document embedding. Cosine similarity measures the angle between two vectors: 1.0 means identical direction (same meaning), 0.0 means unrelated, and -1.0 means opposite. In practice, most useful results fall between 0.7 and 0.95.

To do this at scale — millions of document chunks, sub-second latency — you need a vector database. Tools like Pinecone, Weaviate, Qdrant, and pgvector (a PostgreSQL extension) are purpose-built for this. They use approximate nearest-neighbor algorithms that trade a tiny bit of accuracy for dramatic speed improvements.

See the full RAG pipeline in action:

V Building Your First RAG System

The conceptual pipeline — embed, search, generate — fits on a napkin. The engineering details are where teams spend months. Here's the practical checklist that separates a working RAG system from a frustrating demo.

1. Choose an embedding model. OpenAI's text-embedding-3-smallis the default choice: cheap, fast, and good enough for most use cases. Cohere's embed-v4 and open-source models like bge-large are strong alternatives. The key decision is dimension count — higher dimensions capture more nuance but cost more to store and search.

2. Pick a vector database.If you're already on PostgreSQL, start with pgvector. If you need managed infrastructure at scale, Pinecone or Weaviate. If you're prototyping, an in-memory store like ChromaDB gets you running in minutes. You can always migrate later — the embedding format is portable.

3. Design your chunking strategy. This is the most underrated decision in the entire pipeline — and the one most teams get wrong on the first try. Documents need to be split into chunks before embedding. Too large (entire pages) and the embeddings become vague, like asking someone to summarize an entire chapter in a single word. Too small (single sentences) and you lose context. A chunk size of 200–500 tokens with 50-token overlap between chunks is a solid starting point. Adjust based on your retrieval accuracy.

4. Write your prompt template. The prompt that wraps the retrieved documents matters enormously — and this is where many teams get sloppy. Tell the model to answer onlybased on the provided context. Tell it to say "I don't know" if the context doesn't contain the answer. Without these instructions, the model will happily blend its training data with your retrieved documents, and you'll have no idea which facts came from your data and which it made up. That's not grounding. That's a confidence game.

Test how text similarity works at the core of this pipeline:

VI What's Next

RAG is the workhorse of enterprise AI — practical, deployable, and effective for the majority of grounding use cases. But it has real limitations. What happens when your documents are out of date? When the answer requires reasoning across ten different sources? When retrieval accuracy isn't high enough for compliance-sensitive domains?

In Part 2, we'll confront the cases where RAG falls short and explore the strategies that pick up where retrieval leaves off: fine-tuning, hybrid search, reranking, and the emerging pattern of agentic RAG. RAG gets you 80% of the way there. The next 20% is where product differentiation lives.