Great question. The short answer: Knowledge Graph + LLM (GraphRAG) is still top-tier for specific problem shapes, but it's no longer the universal default. The state of the art in 2026 is broader — hybrid, agentic, and context-aware RAG, with graphs as a powerful specialized component rather than the single best architecture.

Here's the full picture:

Where Plain (Flat) RAG Stands Now

The original "chunk → embed → retrieve → generate" pipeline is now the baseline, not best practice. It's well understood, easy to deploy, but struggles with:

• Questions requiring multi-hop reasoning across documents
• Synthesizing themes or relationships across a corpus
• Fragmented context from fixed-size chunking ("Lost in the Middle" problem)

Where Knowledge Graph + LLM (GraphRAG) Fits

Microsoft's GraphRAG (open-sourced 2024) proved that entity-relationship graphs dramatically improve reasoning over flat retrieval, particularly for global/corpus-level questions like "What are the compliance risks across all our vendor contracts?"arxiv.org

GraphRAG is still actively used and very strong for:

• Multi-hop entity reasoning ("which suppliers for critical components have quality issues?")
• Cross-document synthesis and theme discovery
• High-stakes domains needing deterministic accuracy — GraphRAG combined with carefully curated taxonomies and ontologies can push search precision to as high as 99%squirro.com

But it has real costs:

• 3–5× more LLM calls vs. baseline RAG for graph construction
• Entity recognition accuracy only 60–85% depending on domain — automatically extracted entities often contain significant noise and redundancy
• Knowledge fragmentation: even after graph algorithms discover related communities, generating final answers from discrete fragments places high demands on the LLMragflow.io

What Has Evolved Around and Beyond GraphRAG

The field hasn't replaced KG+LLM so much as it has embedded it into richer architectures:

1. 🔀 Hybrid Multi-Representation Systems (Current Production Standard)

Rather than choosing between vector DBs or knowledge graphs, state-of-the-art enterprise systems maintain multiple knowledge representations simultaneously:

• Dense vector embeddings (semantic search)
• Sparse lexical search (BM25/SPLADE)
• Knowledge graphs (relationship traversal)
• Hierarchical/tree indexes (document structure navigation)

Results are merged via Reciprocal Rank Fusion (RRF), boosting documents that score highly across multiple retrieval methods.applied-ai.com

A 2025 ACL paper, HybGRAG, argues many real-world questions are inherently "hybrid" — requiring both textual and relational retrieval — and that neither pure vector RAG nor pure graph RAG alone is optimal.aclanthology.org

2. 🌲 TreeRAG / Hierarchical RAG

Uses LLMs during ingestion to build a multi-level tree summary of documents (Chapter → Section → Key Paragraph). At query time: precise small-chunk retrieval first, then expand outward using the tree as a navigation map. This elegantly solves the "fragmented vs. complete" chunking tradeoff without the cost of full graph extraction.ragflow.io

Microsoft's LazyGraphRAG (a later iteration) explicitly positions itself as combining the advantages of vector RAG and GraphRAG at lower cost — another sign that pure-graph is being hybridized.arxiv.org

3. 🤖 Agentic RAG (The Hottest Current Frontier)

The biggest shift: moving from fixed "retrieve-then-generate" pipelines to autonomous orchestration.arxiv.org

The LLM becomes the decision-maker, dynamically choosing which retrieval tool to invoke (vector search, SQL, knowledge graph, API), evaluating if retrieved context is sufficient, and looping until it has a complete answer.

Key techniques here include Self-RAG (model decides when to retrieve, critiques its own output) and ReAct-style agents that plan multi-step retrieval strategies. Production deployments report 25–40% reduction in irrelevant retrievals — but also new failure modes like retrieval loops.nstarxinc.com

4. 🧠 Stateful Memory Architectures

Traditional RAG (including GraphRAG) is stateless — every query starts fresh. 2026 frontier systems add persistent memory layers that:

• Track how information evolves over time (marking old nodes "superseded" rather than returning contradictory facts)
• Maintain user-scoped context across sessions
• Bridge document retrieval with long-term conversational context

vardhmanandroid2015.medium.com

5. 🧩 Context Engineering (The Emerging Meta-Paradigm)

How to dynamically and intelligently assemble the most effective context for different tasks became the defining technical challenge of late 2025, giving rise to Context Engineering as a distinct discipline. The core insight: bluntly cramming all potentially relevant data into the context window is both costly and actively harmful to LLM reasoning. Intelligent filtering, ordering, and stitching of context is essential.ragflow.io

Anthropic's Contextual Retrieval (adding chunk-specific context before indexing) is one concrete technique in this space that improves recall significantly.anthropic.com

What About Long Context as a RAG Replacement?

This was a heated 2024–2025 debate. The verdict: complementary, not competitive. Stuffing entire documents into context suffers from "Lost in the Middle" degradation and non-linear cost growth. Improved long-context capability has instead prompted "retrieval-first, long-context containment" synergy — use retrieval to find the right material, then use a large context window to hold coherent chunks of it for reasoning.cloud.google.com +1

Practical Decision Framework

Bottom Line

Knowledge Graph + LLM for RAG is still excellent — especially for entity-rich, relationship-heavy, or multi-hop reasoning tasks. But the state of the art in 2026 is hybrid and agentic: graphs are one powerful retrieval layer within a broader system, not the universally optimal architecture. If you're building now, the practical advice is to architect with an agentic, multi-representation framework and bring in graph retrieval where the query type specifically benefits from relationship traversal.

Great question, Greg — yes, these four aren't alternatives. They're different layers of one system, each solving a distinct sub-problem:

The Full Stack, End to End

User Query    │    ▼┌─────────────────────────────────────────┐│  LAYER 1: Query Understanding           ││  • Classify query type                  ││  • Decompose multi-hop → sub-queries    ││  • HyDE / query rewriting               │└───────────────────┬─────────────────────┘                    │                    ▼┌─────────────────────────────────────────┐│  LAYER 2: Agentic Router                ││  LLM decides which tools to invoke:     ││                                         ││  ┌──────────┐ ┌──────────┐ ┌─────────┐ ││  │  Dense   │ │  Sparse  │ │  Graph  │ ││  │  Vector  │ │BM25/SPLADE│ │   DB   │ ││  └────┬─────┘ └────┬─────┘ └────┬────┘ ││       └────────────┘            │      ││            │  RRF fusion        │      ││            ▼                    ▼      ││    Fused text candidates   Graph facts ││            └────────────────────┘      │└───────────────────┬─────────────────────┘                    │                    ▼┌─────────────────────────────────────────┐│  LAYER 3: Reranking + Quality Gate      ││                                         ││  Cross-Encoder scores all candidates    ││                                         ││  High quality ──────────► Layer 4       ││  Low quality  ──► rewrite query / web   ││                   search / loop back    │└───────────────────┬─────────────────────┘                    │                    ▼┌─────────────────────────────────────────┐│  LAYER 4: Context Engineering           ││                                         ││  <system_instructions>                  ││    Role, constraints, output format     ││  </system_instructions>                 ││  <retrieved_knowledge relevance="0.94"> ││    Top reranked passage(s)              ││  </retrieved_knowledge>                 ││  <graph_context query_type="multi-hop"> ││    Entity relationships, KG summary     ││  </graph_context>                       ││  <episodic_memory compressed="true">    ││    Prior conversation summary           ││  </episodic_memory>                     ││  <tool_definitions> ... </tool_defs>    │└───────────────────┬─────────────────────┘                    │                    ▼┌─────────────────────────────────────────┐│  LLM Generation                         ││  Self-verify: answer contradict sources?││  If yes → Reflexion loop back           │└─────────────────────────────────────────┘

Layer-by-Layer Breakdown

🏗️ Layer 0: Ingestion (Context Engineering starts here, not at generation)

Most teams apply context engineering only at prompt time — that's too late. Before a single query is ever run, you should:

1. Contextual chunking — before embedding each chunk, pass the whole document to a cheap LLM to generate a brief summary, then prepend that to every chunk. A chunk that says "revenue dropped" becomes "[Q3 2025 Acme Corp Financial Report]: revenue dropped."anthropic.com This dramatically closes the asymmetric matching problem where user queries are brief but documents are verbose.

2. Multi-index storage — store each contextual chunk in both a vector DB (dense embeddings) and a BM25/lexical index (sparse). Dense retrieval captures synonyms and implied intent; sparse retrieval is extremely sensitive to exact keywords like product SKUs, legal codes, and proper nouns.

3. Graph extraction — simultaneously run an LLM over the chunks to extract entity/relationship triples (e.g., Acme Corp → ACQUIRED → Beta Startup) and build a Knowledge Graph in parallel. This is how GraphRAG works under the hood: it uses the Leiden algorithm for community detection, divides the graph into hierarchical community levels from specific events up to macro-themes, and pre-generates community summaries.microsoft.github.io

🤖 Layer 2: The Agentic Router — the whole system's control plane

This is the integration point that makes everything else work together. The agent (using frameworks like LangGraph or LlamaIndex Workflows) classifies each query and routes it:

For a query like "How did the acquisition affect Q3 compliance risk?", the agent doesn't run a single search — it decomposes it into sub-queries, fans them out in parallel to the vector DB, BM25 index, and graph, then collects results.learn.microsoft.com

The key design pattern here is the CRAG (Corrective RAG) self-check loop: after retrieval, a lightweight LLM evaluator grades relevance. If high → proceed. If low → rewrite the query, try an alternate source, or trigger a web search. This closed-loop mechanism makes the system robust to edge cases outside the knowledge base, significantly improving robustness compared to one-shot pipelines.

🔀 Layer 2a: Hybrid Retrieval + RRF in detail

RRF is the right fusion method because BM25 scores and cosine similarity scores live on completely different scales — you can't normalize them directly. Instead, RRF works purely on rank position:

where $k$ is typically 60, $R$ is the set of ranked lists (dense, sparse, etc.), and $\text{rank}_r(d)$ is document $d$'s rank in list $r$. Documents appearing near the top of multiple retrieval lists float to the top of the fused list.learn.microsoft.com

Production-proven numbers: Vanguard's financial RAG system got a 12% retrieval accuracy boost from adding weighted BM25 + vector fusion via hybrid search, directly eliminating the need for seasonal support staff. Databricks' Mosaic AI Vector Search saw Recall@10 jump from 74% to 89% after adding a Cross-Encoder reranking step with only ~1.5 seconds of added latency.synthimind.net

Important note on graph + RRF: don't feed raw graph nodes directly into RRF alongside text chunks — they're not the same unit. Instead, convert graph results into evidence cards (subgraph summary + key relationship paths + provenance IDs), then fuse those cards with text chunks through the reranker. This is exactly how GraphRAG local search already works: it combines structured graph data and associated text units into a single bounded evidence set.microsoft.github.io

🌐 Layer 2b: Knowledge Graph — a specialist tool, not a replacement

The KG fits cleanly as one of several retrieval tools the agent can invoke. GraphRAG supports four distinct query modes you can expose as separate agent tools:

• Local search — entity-centric, multi-hop reasoning ("Which suppliers are linked to Company X?")
• Global search — corpus-level holistic questions using Map-Reduce over community summaries ("Summarize the main compliance risks across all documents")
• DRIFT search — hybrid blend of local and global context
• Basic search — baseline vector search as fallback

LinkedIn's production GraphRAG deployment is a strong proof point: by building a KG with tickets as nodes and citation relationships as edges, they improved retrieval MRR by 77.6% and reduced median issue resolution time by 28.6%.synthimind.net

The two integration patterns:

• Option A (KG as parallel retriever): vector top-20 + BM25 top-20 + graph local top-10 evidence cards → fuse with RRF → rerank → answer. Good when graph evidence can be grounded back to source passages.
• Option B (KG as selective specialist tool): agent invokes graph retrieval only when the query involves entity disambiguation, relationship traversal, or cross-document synthesis. This is usually the better default — lower cost, lower complexity, full power when actually needed.

🧠 Layer 4: Context Engineering at assembly time

After retrieval and reranking, don't just stuff results into the prompt. Over 70% of errors in modern LLM applications stem not from insufficient model capability but from incomplete, irrelevant, or poorly structured context.meta-intelligence.tech

Four strategies that compound together:

1. Hierarchical structure — most relevant reranked results at top; supplementary background in the middle; tool definitions at the bottom. LLMs pay more attention to the beginning and end of context ("Lost in the Middle" problem is real even at 200K tokens).

2. Structured XML tagging — tag context by source so the LLM can attribute answers and you can audit them:

<context>  <graph_facts source="neo4j" query_type="multi-hop">    Acme Corp acquired Beta Startup on Aug 1.    Beta Startup has active vendor contract with Delta Compliance LLC.  </graph_facts>  <document_excerpt source="Q3_report" relevance="0.98">    [Q3 2025 Acme Financial Report]: Post-acquisition integration    revealed overlapping compliance liabilities...  </document_excerpt>  <episodic_memory compressed="true">    User previously asked about cryptocurrency regulatory framework...  </episodic_memory></context>

3. Dynamic compression — compress earlier conversation turns into summaries; distill long retrieved docs into key passages. Balances information completeness with token budget.

4. Selective injection — not everything needs to be in the window simultaneously. Route financial queries to financial docs + conversation history; technical queries to technical docs. Swap context dynamically.

💾 The Memory Dimension

Context engineering also spans across sessions via three memory layers:

The episodic layer is what makes an AI assistant feel like it "remembers you" across sessions — summaries of prior conversations are retrieved by relevance and injected into working memory when needed.meta-intelligence.tech

Practical Build Recommendation

Start minimal, layer in complexity:

Tooling:

The One-Line Summary

The agent controls the loop. RRF + the KG supply the raw material. Context engineering shapes how that material reaches the model. None of them hit their ceiling without the others.

Yes — and these three pillars aren't just included, they are the control layer that makes the entire retrieval stack functional. Without them, you have a pipeline. With them, you have an autonomous knowledge system.

Here's the complete mapping:

How All Three Pillars Map In

🧠 1. Memory — What the system knows and retains

Memory splits into four distinct types (most implementations only build three):

Procedural memory is the most commonly forgotten type. It's what separates an agent that uses tools competently from one that fumbles — encoding strategies like "always decompose questions involving more than one entity before querying the graph."

Memory write is just as important as memory read. After generation, the loop should:

• Compress the conversation turn → episodic memory
• Detect new entities/relationships → write back to the KG
• Mark superseded facts as stale (don't just add — prune)

The Knowledge Graph is your semantic memory layer. It isn't just another retriever — it's where the system's stable world knowledge lives.

📋 2. Planning — What the system decides to do before acting

This is the most under-specified pillar in most RAG systems. A non-planning RAG pipeline fires a query straight at the vector DB. A planning-capable system does this first:

Step 1 — Classify the query type: factual / relational / analytical / procedural

Step 2 — Decompose if multi-hop:

"How did the acquisition affect Q3 compliance risk?"

• Sub-q1: When was the acquisition, and what entities does it involve?
• Sub-q2: What compliance risks exist for those entities?
• Sub-q3: How do they intersect in Q3 specifically?

Step 3 — Generate a tool execution plan before invoking anything:

[parallel]    graph_local_search(sub-q1)[parallel]    hybrid_retrieval(sub-q2)[sequential]  synthesize → sub-q3 after the above complete[conditional] if confidence < 0.7 → escalate to web_search

Step 4 — Reflexion gate after generation: Does the answer contradict any retrieved source? Are all sub-questions addressed? If not — revise the plan and re-retrieve before the answer reaches the user.

Three planning patterns complement each other:

• ReAct — reason aloud before each tool call; inspectable and auditable
• Plan-then-Execute — generate the full plan upfront, then execute; enables parallel tool calls and lower token cost
• Reflexion — a critic pass after generation catches hallucinations and triggers re-retrieval

🔧 3. Tool Orchestration — How the system acts on its plan

The router described earlier is the beginning of orchestration, but full orchestration goes further:

A typed tool registry — the agent selects tools by function calling with defined schemas:

{ "tools": [    { "name": "hybrid_retrieval",       "description": "...", "params": {...} },    { "name": "graph_local_search",     "description": "...", "params": {...} },    { "name": "graph_global_search",    "description": "...", "params": {...} },    { "name": "sql_query",              "description": "...", "params": {...} },    { "name": "web_search",             "description": "...", "params": {...} },    { "name": "episodic_memory_recall", "description": "...", "params": {...} }]}

Parallel vs. sequential execution graphs — independent sub-queries fan out simultaneously; dependent steps wait (fan-in). This is exactly what LangGraph and LlamaIndex Workflows model as DAGs.

Multi-agent delegation — at scale, a "tool" can itself be a specialized sub-agent:

Orchestrator Agent  ├── Retrieval Agent   (owns hybrid search + RRF fusion)  ├── Graph Agent       (owns KG traversal + evidence cards)  ├── Analyst Agent     (owns SQL + numerical reasoning)  └── Synthesis Agent   (owns context assembly + generation)

Each sub-agent has its own memory scope and toolset. The orchestrator only sees their outputs, isolating failure modes and allowing independent scaling.

The Complete Picture

The agent control plane then drives the retrieval stack:

User Query    │    ▼┌─────────────────────────────────────┐│  MEMORY READ                        ││  episodic + semantic + procedural   │└──────────────┬──────────────────────┘               │               ▼┌─────────────────────────────────────┐│  PLANNING                           ││  classify → decompose → build plan  ││  → reflexion gate after generation  │└──────────────┬──────────────────────┘               │               ▼┌─────────────────────────────────────┐│  TOOL ORCHESTRATION                 ││  parallel: dense + sparse + graph   ││  → RRF fusion → reranker            ││  → CRAG quality gate                ││  → fallback / rewrite if needed     │└──────────────┬──────────────────────┘               │               ▼┌─────────────────────────────────────┐│  CONTEXT ENGINEERING                ││  prioritize, compress, structure    │└──────────────┬──────────────────────┘               │               ▼          LLM Generation               │               ▼┌─────────────────────────────────────┐│  MEMORY WRITE                       ││  compress → episodic                ││  new entities → KG                  ││  stale facts → pruned               │└─────────────────────────────────────┘

The hybrid retrieval (RRF, KG, reranking) and context engineering are the bottom half. Memory, Planning, and Tool Orchestration are the top half — the cognitive infrastructure driving when, how, and how many times retrieval gets invoked. Neither half works well without the other.