Graph RAG & Structured Retrieval

I Where Vector RAG Breaks

You have built a RAG pipeline. Embeddings, vector search, reranking — the full stack from Parts 1 and 2. For straightforward questions, it works. "What is our return policy?" retrieves the right paragraph and the model nails the answer. But then someone asks a different kind of question:

"Which team lead approved the budget for the project that caused last quarter's compliance incident?"

This is a multi-hop question. The answer requires connecting three separate facts: the compliance incident, the project it came from, and the person who approved that project's budget. Each fact might live in a different document. Vector search finds documents that are semantically similarto the query — but semantic similarity and informational relevance are not the same thing. The budget approval memo doesn't mention compliance. The compliance report doesn't mention budget approvals. The embedding space puts them far apart, even though they are causally linked through a shared project.

This is the fundamental limitation of vector RAG: it retrieves by similarity, not by relationship.Cosine similarity measures how much two texts "sound like" they're about the same thing. It cannot follow a chain of relationships across documents. It cannot answer "who reports to the person who manages the team that built this feature?" because that answer requires traversing a graph of connections, not finding a similar paragraph.

Three failure patterns show up repeatedly:

1. Multi-hop reasoning.The answer spans 2–5 documents connected by shared entities. Vector search retrieves each document independently but can't stitch the chain together.
2. Structural queries."Who reports to whom?" "What depends on what?" These questions are about the topology of your data, not its content. Embeddings don't encode organizational structure.
3. Aggregation across relationships."How many projects does this team lead manage that are over budget?" requires traversing from person to projects to budgets, then counting. Vector search has no concept of traversal or aggregation.

If your users only ask "what does document X say about topic Y?" — vector search is all you need. The moment they start asking about connections between things, you've outgrown it.

II What Graph RAG Is

Graph RAG adds a knowledge graph to the retrieval pipeline. Instead of just embedding document chunks, you also extract entities (people, projects, policies, departments) and relationships (manages, approved, caused, depends-on) from your data. These get stored as nodes and edges in a graph database. At query time, you traverse the graph to find connected information, then feed the results to the LLM alongside any vector-retrieved context.

The extraction step is where the LLM earns its keep before the user even asks a question. You run your documents through a model with a prompt like: "Extract all entities and their relationships from this text. Output as (entity1, relationship, entity2) triples." A budget memo becomes (Sarah Chen, approved, Project Atlas), (Project Atlas, budget, $2.4M), (Project Atlas, team, Platform Engineering). An incident report becomes (Compliance Incident CI-2024-17, caused_by, Project Atlas), (CI-2024-17, reported_by, Legal).

Now the multi-hop question from Section I becomes trivial. "Which team lead approved the budget for the project that caused last quarter's compliance incident?" turns into a graph traversal: start at the incident node, follow the caused_by edge to the project, follow the approved edge to the person. Two hops, one answer. No embedding similarity required.

The pipeline has two phases:

1. Indexing (offline). Extract entities and relationships from documents using an LLM. Store them in a graph database (Neo4j, Amazon Neptune, or even a simpler structure). Optionally, also embed the document chunks in a vector store for hybrid retrieval.
2. Querying (online).Parse the user's question to identify entities and the type of relationship being asked about. Traverse the graph to gather connected context. Optionally, also run a vector search for additional semantic context. Combine everything and generate the answer.

Microsoft Research's GraphRAG paper (2024) formalized this approach and showed that graph-based retrieval significantly outperforms vector-only retrieval on questions requiring synthesis across multiple documents — precisely the failure mode we identified in Section I. Their key innovation was using community summaries: clustering related nodes in the graph and pre-computing summaries of each cluster, so the system can answer broad questions ("What are the main themes in this dataset?") without retrieving every individual document.

Try it yourself — see how the same question gets answered differently by each retrieval approach:

III When to Use It

Graph RAG is not a universal upgrade. It is a specialized tool for a specific class of problems, and applying it to the wrong problem will cost you months of engineering for marginal improvement. The decision framework is straightforward: does your use case require reasoning about relationships between entities?

Good fit for graph RAG:

• Multi-hop questions."Who reports to the person who approved this budget?" requires traversing from budget to approver to their manager. Every layer of indirection adds another hop that vector search cannot follow.
• Compliance and audit trails.Regulated industries need to trace decisions back through chains of approvals, modifications, and authorizations. "Show me every change to this policy and who authorized each one" is a graph problem.
• Interconnected data. Org charts, supply chains, codebases, product dependency trees, customer relationship maps. Any domain where the structure of the data carries as much information as the content.
• Broad synthesis queries."What are the main risks across all our active projects?" requires aggregating information from many documents. Microsoft's community summary approach handles this by pre-clustering the graph.

Poor fit for graph RAG:

• Simple factual Q&A."What is our return policy?" lives in one paragraph. Vector search retrieves it perfectly. Adding a graph adds complexity for zero improvement.
• Low-relationship data.If your documents are independent articles, blog posts, or FAQs with few cross-references, there isn't enough connective tissue to build a useful graph.
• Rapidly changing data with no entity stability. Graphs require stable entity identifiers. If your entities change names or merge frequently, graph maintenance becomes a full-time job.

IV Structured Retrieval Beyond Graphs

Graphs are the highest-profile structured retrieval method, but they are far from the only one. Many retrieval problems don't need a knowledge graph — they need a database query. The broader category is structured retrieval: any technique that uses the structure of your data to find answers, rather than relying solely on semantic similarity.

Text-to-SQL.The user asks a question in natural language. The LLM translates it into a SQL query. The query runs against your database. The results come back as structured data. "What were our top 5 customers by revenue last quarter?" becomes SELECT customer_name, SUM(revenue) FROM orders WHERE quarter = 'Q1-2025' GROUP BY customer_name ORDER BY SUM(revenue) DESC LIMIT 5. No embeddings. No vector search. Just a database query that returns exactly the right answer. Text-to-SQL is the most underappreciated retrieval pattern in production — if your data is already in a relational database, it's often the fastest path to accurate answers.

Metadata filtering.Before running vector search, narrow the search space using structured metadata: document type, department, date range, access level. "What did the engineering team publish about authentication in Q4?" filters by team and date before running semantic search on the remaining documents. This is not a replacement for vector search — it's a force multiplier. Metadata filtering is cheap, fast, and dramatically improves precision by eliminating irrelevant results before they compete for top-K positions.

Table extraction.Many enterprise documents contain tables: financial reports, spec sheets, comparison matrices. Embedding a table row by row loses the column headers. Embedding the entire table makes the embedding too vague. Table-aware extraction preserves the structure, allowing the system to answer "What was the Q3 margin for the Enterprise segment?" by finding the right cell, not the most similar paragraph.

Faceted search. Combine multiple filter dimensions — category, author, date, status — with keyword or semantic search. This is the retrieval pattern behind every e-commerce search bar: the user narrows by price, brand, and rating before browsing results. The same pattern works for document retrieval when users know what kindof thing they're looking for but not the exact content.

Describe your query type below and see which retrieval strategy fits:

V The Build vs Buy Decision

You've decided your use case warrants graph RAG. Now comes the infrastructure question, and this is where teams spend either $50K or $500K depending on how honestly they assess the decision.

Option 1: Full graph database. Neo4j, Amazon Neptune, or ArangoDB. You get a dedicated graph engine with Cypher or Gremlin query languages, ACID transactions, and mature tooling. This is the right choice if your knowledge graph is a core product asset — something you will maintain, evolve, and depend on for years. The cost is operational: graph databases need a team that understands graph modeling, query optimization, and schema evolution. Most teams underestimate this.

Option 2: Graph layer on existing stores.Store triples in PostgreSQL with a lightweight graph query library, or use a property graph overlay on your vector database (Weaviate and some others support this). Lower operational overhead, less powerful graph queries. Good for teams that need basic relationship traversal without a dedicated graph database. The tradeoff: you won't get the query performance or modeling flexibility of a real graph database, and complex multi-hop queries may be slow.

Option 3: Managed graph RAG services.Microsoft's GraphRAG library, LlamaIndex's knowledge graph module, LangChain's graph integrations. These abstract the graph construction and querying into higher-level APIs. The fastest path from zero to working prototype. The tradeoff: you're locked into their extraction and querying patterns, which may not match your specific data model.

The honest assessment: graph RAG is operationally expensive.Entity extraction is noisy — LLMs will produce inconsistent entity names ("Sarah Chen" vs "S. Chen" vs "the VP of Engineering"), and you'll need entity resolution to merge them. Relationship extraction is even noisier. The graph needs ongoing maintenance as your source documents change. And graph query patterns require a different mental model than SQL or vector search.

VI Hybrid Architectures

The best production systems don't choose between vector search and graph traversal. They combine them. The pattern is called a retrieval router: classify the query type, then route it to the retrieval strategy most likely to produce a good answer.

A retrieval router works in three steps:

1. Classify the query.Use a lightweight classifier (or the LLM itself) to determine the query type. Is it a factual lookup? A relational question? An analytical query? A broad synthesis request? The classification doesn't need to be perfect — it needs to be good enough to route to the right retrieval strategy most of the time.
2. Route to the right strategy. Factual lookups go to vector search. Relational questions go to graph traversal. Analytical questions go to text-to-SQL. Broad synthesis queries hit graph community summaries. Some queries may trigger multiple strategies in parallel.
3. Merge and generate.Combine the retrieved context from all strategies, deduplicate, and feed it to the LLM. The model doesn't need to know which retrieval method found each piece of context — it just synthesizes the answer from everything it's given.

The power of this architecture is that each retrieval method covers the others' weaknesses. Vector search provides broad semantic coverage — it catches relevant documents that don't share entities with the query. Graph traversal provides relational precision — it follows connection chains that vector search can't see. Metadata filters provide scoping — they eliminate irrelevant results before the expensive retrieval methods run. Together, they handle the full spectrum of user queries.

A practical example: a customer support AI for a SaaS product. "What does the Enterprise plan include?" routes to vector search (single-document factual lookup). "Which customers on the Enterprise plan have open support tickets about the billing API?" routes to graph traversal (connecting customers, plans, tickets, and features). "How many Enterprise customers renewed last quarter?" routes to text-to-SQL (analytical query against the CRM). One product, three retrieval strategies, one unified experience for the user.