Your RAG pipeline just returned the wrong answer. Again.
The query was simple. The doc had the answer. You watched the top-5 chunks come back and none of them held it. The right paragraph sat two pages away from anything your embedding model thought was "similar." So the LLM hallucinated something plausible, your user trusted it, and you spent the afternoon tuning chunk_size from 512 to 1024 and back.
This is not a tuning problem. This is the architecture.
Vector RAG was a clever hack for a 2022 problem: LLMs had tiny context windows, so we shredded documents into chunks, embedded them, and prayed cosine similarity would find the right shred. It worked well enough on FAQ pages and Notion dumps that we called it a pattern and shipped it into production at every company on earth.
Then we pointed it at a 200-page 10-K. A Kubernetes operator manual. A pharma SOP. A legal contract. And it fell apart.
The Chunking Lie
Vector RAG rests on four assumptions. All four break on real documents.
Assumption 1: Similarity equals relevance. It doesn't. "revenue declined" and "revenue grew" sit next to each other in embedding space — opposite meanings, near-identical vectors. Domain jargon drifts even worse. "margin" means one thing in finance, another in typography, another in ML. Your general-purpose embedding model doesn't know which doc it's in.
Assumption 2: The right answer fits in k chunks. Why is k=5? Because someone's tutorial said so. Real answers span sections, reference earlier definitions, sit inside tables that got shredded on ingest. Top-k is a guess dressed up as a hyperparameter.
Assumption 3: Chunk boundaries preserve meaning. They don't. A 512-token window slices mid-sentence, mid-table, mid-recipe. The chunk that contains the answer loses the heading that gives it context. The LLM sees "...bake at 425°F for 40 minutes" and has no idea which dish "it" belongs to.
Assumption 4: Documents are bags of text. They aren't. A 10-K has a structure. A manual has chapters. A contract has clauses that reference other clauses. Vector RAG throws all of that away on ingest, then acts surprised when retrieval can't find its way back.
Every RAG failure I've debugged in the last two years traces back to one of these four. Bigger embedding models don't fix them. Reranking papers over them. Hybrid BM25 just postpones them. The problem isn't the retriever — it's the premise.
Humans don't read a 200-page doc by cosine similarity. They open the table of contents.
How You Actually Find an Answer
Forget databases for a second. Picture a 400-page cookbook, and you want the roast chicken recipe.
You don't read all 400 pages looking for the passage most "similar" to chicken. You flip to the contents, find Poultry, turn to that chapter, run your finger down to Roast Chicken, and read the whole recipe — ingredients, oven temp, and timing all together, in order.
Now imagine doing it the vector-RAG way: tear every page out, shuffle them into a bin, and pull the five loose pages whose words look closest to your question. You might grab the ingredients list but miss the oven temperature on the next page. You might grab a chicken stock recipe by mistake. And you'd have no idea which chapter any page came from.
That's the difference. Humans use reasoning over structure; vector RAG uses similarity over shredded text. The table of contents isn't decoration — it's a map of meaning the author already drew for you, and chunking throws it in the trash.
So what if retrieval kept the map?
Meet the Killer
It's called PageIndex.
Instead of embedding chunks, PageIndex builds a tree from the document's structure — chapters, sections, sub-sections — mirroring the table of contents. (No table of contents? An LLM generates the tree from the page text.) It keeps the cookbook intact instead of shredding it. Each node knows what it contains. No vectors. No cosine similarity. No k.
At query time, an LLM walks the tree the way a human walks a TOC. It starts at the root, reasons about which branch is most likely to hold the answer, descends, re-evaluates, and either drills deeper or backs out. When it lands on the right node, it reads the full, intact section — headers, tables, and context still attached.
Two things fall out of this for free:
1. Context stays whole. The answer arrives with the section that frames it. That orphaned "...bake at 425°F for 40 minutes" from earlier now carries its heading — Roast Chicken — instead of floating free.
2. Retrieval is explainable. You get the reasoning trace — the exact path from root down to the answer node. When it's wrong, you see where it turned wrong. Try debugging a cosine score.
Does it actually work?
PageIndex is built and open-sourced by Vectify AI (docs). And the numbers on structured docs are hard to argue with. On FinanceBench — QA over real financial filings — Vectify's Mafin 2.5, built on PageIndex, reports:
98.7% accuracy — vs 31% for ChatGPT-4o with search, and 45% for Perplexity, on the same documents.
Worth being honest: that's a vendor-reported number, not an independent replication. But the pattern holds — structured documents are exactly where similarity search hurts most, and where a tree wins biggest.
Vector RAG searches for text that looks like the answer. PageIndex reasons about where the answer lives.
Same Question, Same Doc
Enough theory. Back to that 400-page cookbook, one real query: "What temperature do I roast the chicken at, and for how long?"
Vector RAG returns the top-5 chunks by similarity. Four are about chicken stock, chicken salad, chicken stir-fry — they all say "chicken." One catches "425°F for 40 minutes" but lost its heading on chunking, so the model can't tell if that's the roast chicken or the roast potatoes two pages over. Answer: confident, wrong.
PageIndex walks the tree — root → Poultry → Roast Chicken → the intact recipe, temperature and timing under their own heading, ingredients and steps still attached. It reads the answer in context and gets it right.
Same book. One retriever guessed at surface text; the other navigated to where the answer actually lived — the same way you would.
The Part Everyone Misses: Cross-References
Real documents are full of pointers. "Prepare the brine as described on page 12." "See the pan-searing technique in Chapter 2." "Use the spice blend from the recipe above." The roast chicken recipe only makes sense alongside a brine three chapters back.
Vector RAG is structurally blind to this. It fetches the top-k chunks that look like your query — and a cross-reference looks like nothing. The chunk that says "brine the bird first" scores well; the chunk three chapters away that actually explains how to make the brine doesn't, so it never comes back. The model answers with half the recipe and no idea the other half exists.
PageIndex sees the whole tree at once. When the node it lands on points elsewhere — "see the brine on page 12" — the navigating LLM can follow that pointer and pull the referenced node too. It isn't a dedicated link-resolver; it's a side effect of reasoning over the full structure instead of a bag of disconnected fragments. But the result is what matters: the dependency gets read, not dropped.
A chunk is an island. A tree node knows its neighbors.
But I Have a Whole Shelf, Not One Book
Fair — real systems aren't one cookbook, they're a library of thousands. The instinct is to reach for vectors again here: embed every book, do a similarity search to pick candidates, then navigate inside. And you can — a vector first pass narrows the shelf, then PageIndex walks each book to the exact recipe. It's fast, and it's the current default in Vectify's own API, mostly because walking a giant tree with an LLM costs more calls than a millisecond vector lookup.
But notice what just happened: you dragged the shredder back into your pipeline for the one job it was never bad at — a coarse "which books are roughly about Italian food?" And now you're maintaining an embedding store, a vector index, and a reasoning layer to answer one question.
The cleaner idea is to go structure all the way up: the whole shelf becomes one big tree. Cuisine → book → chapter → recipe, one hierarchy the LLM navigates top-down — no embeddings, no separate vector store, no re-indexing when documents change. Vectify ships this as their "file system" layer for corpus-scale search. It's the newer path and still maturing, but it's the honest end state of the argument. If structure beats similarity inside a document, it beats it across a collection too.
The lesson isn't "add vectors for scale." It's don't reach for a similarity index at every level as a reflex — that's overhead masquerading as architecture. Use it as a temporary speed shortcut if you must, and drop it the moment structure can do the job. Either way, the shredder never comes back.
Which One, When
PageIndex is not free. A vector lookup is ~50ms and costs cents; walking a tree with an LLM takes seconds — sometimes tens of seconds — and multiple model calls. You're trading milliseconds and cents for accuracy, whole-context answers, and an audit trail. Whether that's worth it is entirely about your workload:
| Workload | Reach for | Why |
|---|---|---|
| Dense filings, contracts, SOPs, manuals | Page Index | real structure to navigate; answers need whole-section context + an audit trail |
| Steps or clauses that reference other steps | Page Index | follows cross-references vector search is blind to — the difference between a right answer and a confidently wrong one |
| A million structured docs | Page Index | one hierarchy, navigated top-down — vectors optional as a speed prefilter, not a requirement |
| 100k tickets, FAQs, chat logs | Vector RAG | genuinely flat — no structure to navigate, sub-second latency, cents per query |
| Real-time chatbot autocomplete | Vector RAG | ~50ms lookup; a tree-walk is far too slow |
Vectors still win where the data is genuinely flat and latency is everything — chat logs, FAQs, autocomplete. But the moment your documents have structure, that's PageIndex's turf, one book or a million.
So Is RAG Actually Dead?
What's dead is the reflex to reach for vector-only RAG as the default for every retrieval problem. That default was never right for structured documents; we just didn't have a better option worth the effort. Now we do.
Retrieval is turning from a similarity-search problem into a reasoning problem. Vectors were the hack we needed when models were small and context was scarce. Neither is true anymore.
Point your worst-performing RAG query — the one you've been re-chunking for a week — at a tree instead. The code is open source: github.com/VectifyAI/PageIndex. Then tell me what breaks. Change my mind: what breaks PageIndex?
If you're building at the seam between real apps and LLMs, that's most of what I write about here — follow along if it's your kind of problem.
Raj Kumar is a Toronto-based tech lead. Building software since 2011, AI since 2014 (before the buzzword). Ruby, Python, and system architecture; currently working the seam between production apps and LLM agents.
Comments
Post a Comment