RAGLens Research
What happened when we added a knowledge graph
We tested a set of questions against a small handbook to see how different retrieval methods affected AI-generated answers.
We compared semantic search, version filtering, and knowledge-graph approaches. The version filter did most of the work. The graph fixed one important miss, but the way we used it caused misses elsewhere.
The result was not “metadata beats knowledge graphs.” It was narrower: for this corpus and this retrieval design, keep the version filter and rethink how the graph is used.
Why we ran this
Semantic search is good at finding content with similar meaning. That does not guarantee the content is the right version, or that it includes a prerequisite or exception stored somewhere else.
We wanted to test whether two kinds of structure could help:
- Metadata that says which version a source applies to.
- Knowledge graph relationships that connect prerequisites, exceptions, and replacement policies.
Before the run, we fixed the questions, scoring, and decision rules in a registered experiment.
What we tested
We compared four retrieval approaches using the same 24 questions and the same fictional supplier handbook. The handbook contained 36 pieces of content. Each approach could send up to 1,400 tokens of source material to the model.
Semantic search
Ranks sources by vector similarity to the question.
Metadata-filtered
Detects the requested version, then removes semantic results that do not apply.
Knowledge graph
Starts with filtered semantic results, then follows authored prerequisite, exception, and replacement policies.
Graph + related
Also follows looser “see also” links to test whether broader expansion helps or just adds noise.
We generated three answers for each approach. When Graph + related retrieved exactly the same evidence as Knowledge graph, we did not generate a duplicate answer set.
What we found
The version filter worked
Plain semantic search retrieved an outdated policy on 11 of 24 questions.
The metadata-filtered approach removed the outdated results from all 11 questions and did not lose any required evidence. Its evidence recall matched semantic search on all 24 questions.
That is the clearest result in the run: when the system knew which version applied and used that metadata as a filter, it stopped sending outdated policy to the model.
This does not mean metadata automatically improves vector search. The retrieval system has to be designed to use it.
The graph fixed one miss that semantic search could not
One question asked how to set up a rush order. A correct answer also required an artwork-proof approval step stored elsewhere in the handbook.
Semantic search ranked that prerequisite below the evidence cutoff. The version filter did not change that.
The knowledge graph followed the prerequisite relationship and retrieved the missing evidence. It answered the question correctly in all three repetitions.
This is the clearest case where graph traversal helped: plain semantic search and the metadata-filtered approach both missed the prerequisite.
Our graph strategy also created misses
The graph approach started with four semantic results instead of seven so there would be room for graph expansion.
On the broken-mugs question, those four results focused on mugs and fee policies. The needed damage-claim evidence was farther down the search results, and none of the starting sources connected to it.
The graph never reached the evidence it needed. This is seed starvation.
Loose related links added noise
When Graph + related changed the evidence package, it added irrelevant sources and never improved recall.
The typed relationships were useful because they meant something specific: prerequisite, exception, or replacement. The looser related links consumed evidence budget without improving recall.
The model invented a policy the handbook never stated
The broken-mugs question also asked whether a replacement would incur another setup fee.
The handbook does not answer that.
In 11 of 12 generated answers across the four retrieval approaches, the model made a plausible policy inference and stated it as fact anyway.
Changing retrieval architecture did not cause the behavior, and it did not fix it.
General-purpose evaluation also missed the failure. A check written specifically for the unanswered part of the question caught all 11 unsupported claims.
The run exposed a separate failure mode: a partially answerable question can produce an answer that appears grounded while still filling an evidence gap with an unsupported claim.
Fully unanswerable questions behaved better
We also included three questions the handbook could not answer at all.
Every generated answer abstained on all three fully unanswerable questions.
On those questions, the model abstained rather than supplying unsupported policy.
Research outcome
Keep the version filter. Rethink the graph approach.
The version filter delivered a clear gain without reducing evidence recall. The graph recovered one required prerequisite, but the four-seed design lost required evidence elsewhere.
This outcome applies to this experiment, not to knowledge graphs in general.
The next test is whether graph expansion can preserve that recovery without shrinking the semantic starting set.
Where this result is weak
This is a small experiment and the limits matter.
The corpus is small.
We tested 36 well-structured content units, not thousands of messy enterprise documents.
The metadata is unusually clean.
Every applicability tag is trustworthy. Real metadata can be missing, stale, or wrong. This experiment does not tell us how much bad metadata a filter can tolerate.
We tested one graph strategy.
The graph started from four semantic results and expanded under a fixed evidence limit. A different seeding or budgeting strategy could perform differently.
The questions are cleaner than real search logs.
They passed an independent naturalness review, but real users often type things like “installation” or omit product and version entirely. This experiment does not test how well a system can recover missing context.
Some question types have very small samples.
Those results are useful observations, not broad findings.
Some scoring uses an LLM judge.
The judge followed explicit rules and was blinded to the retrieval condition, but it is still a model making a judgment.
We do not claim that metadata beats knowledge graphs.
We claim that, in this small controlled test, a version filter captured most of the measurable gain. The graph exposed both a real advantage and a real design problem.
What we would test next
Give the graph more starting points.
The graph started with four search results; the filter had seven. Next, we’d keep all seven candidates available and let graph-connected evidence compete for the limited space. The question: can we keep the rush-order win without causing the broken-mugs miss?
Test messier searches.
Real users often type things like “installation” and leave out the product, version, or even what they are trying to do. We’d test what happens when that missing context comes from the query, the surrounding application, metadata, or nowhere at all.
Break the metadata on purpose.
Our metadata was unusually clean. We’d remove tags, make some stale, and make some wrong to see when version filtering stops helping and starts hurting.
Test a larger, messier corpus.
Repeat the comparison with more content, more overlap, more versions, and more near-duplicate sources. We want to know whether the same retrieval patterns hold when finding the right source is genuinely harder.
Test a guardrail for partial answers.
When the sources answer only part of a question, tell the model to say what the evidence does not establish. Then test whether that reduces unsupported claims without making good answers worse.
Explore the experiment
The interactive comparison shows the four retrieval approaches on the examples discussed here. You can inspect the evidence each approach selected, see why a source was included, compare the registered results, and try your own question.
The technical record contains the registered findings in full, including the decision rule, prediction scorecard, population tables, per-category results, and caveats.
Terms used here are defined in the RAGLens glossary.