Retrieval design · Supplier handbook · Experiment

Before adding another model call, try the obvious retrieval lever

I came across an idea called evidence-guided query reformulation. The approach, described in EviReform, is pretty intuitive: retrieve an initial set of evidence, let an LLM identify what the evidence still does not answer, generate a narrower query for that missing information, and search again.

That caught my attention because the knowledge graph in my first study had recovered useful evidence in some cases but missed it in others. A second, evidence-guided search seemed like a plausible way to recover what the first pass had missed, so I built it.

While I was working through that comparison, I ran into a much more basic question. The graph was starting from only four semantic search results. What happened if I simply gave it more? I added another condition with seven starting points and compared the two approaches.

The simple change won.

Giving the graph seven seeds recovered as much or more required evidence than the LLM reformulation step, without another model call. That does not mean seven is the right number or that reformulation is a bad technique. It means I had reached for an interesting new retrieval mechanism before checking whether the existing one was simply under-tuned. That was useful to find out.

Reformulation did recover missing evidence

I don’t want to flatten this into “the LLM approach failed,” because it didn’t. The reformulator looked at the question and the evidence already retrieved. If it thought something was unresolved, it generated one more search query. When it did that, the second search worked.

On one question about filing a damage claim, for example, the reformulator recognized that the inspection procedure was missing, searched for it, and recovered the required evidence. So the mechanism did what I hoped it would do. It just wasn’t better than the cheaper alternative in this experiment.

The extra model call also created another way to fail

The reformulator had to return a strict structured response. On one question, it returned explanatory text before the JSON. The system correctly treated that as a mechanism failure and fell back to the original retrieval result. That produced a 4.2% mechanism-failure rate in the Reformulated series and made one of my predeclared test cases unusable.

An additional model call does not just add cost and latency. It adds another component that can behave unexpectedly. If it does not produce a clear retrieval gain, that extra complexity is hard to justify.

One of my original assumptions was too simple

I also tested what happened when I removed explicit prose pointers such as “see Inspect your delivery” while leaving the underlying dependency statement and graph relationship intact.

Removing those pointers changed a lot of rankings and evidence packages. It changed which required evidence was present on zero questions. So I can’t claim from this study that the graph becomes uniquely valuable when those prose pointers disappear.

There was one interesting case where removing a pointer changed the reformulator’s judgment about whether more evidence was needed. That’s worth noticing, but one case isn’t enough to turn into a general rule.

Perfect recall still didn't guarantee a good answer

One result surprised me more than the headline result. On a damage-claim question, Reformulated successfully recovered all of the evidence I had labeled as required. But its final evidence package pushed out several setup-fee and reorder-policy units that were useful for answering another part of the question. Retrieval recall said 1.00. The generated answer was still worse.

That exposed a weakness in my own evaluation. “Required evidence” is useful for measuring retrieval, but it does not necessarily capture every piece of context the generator needs.

Where this result is weak

There are several reasons not to generalize too far from this study.

The corpus has only 36 units. Going from four seeds to seven is a big proportional change.
Seven was a test value, not an optimized value. I do not know whether five, eight, ten, or some adaptive depth would work better.
The graph relationships are clean and authored. Real knowledge graphs are often incomplete or noisy.
The evidence for one of the question categories is especially thin. After a reformulator failure excluded one case, the comparison rests on a single question.
Reformulation only activated on a small number of questions, so this is not a broad test of everything reformulation could do.
The pointer-removal test kept the underlying dependency facts in the prose. It does not tell me what happens when the graph contains information the prose never states.

And all of this happened under a fixed 1,400-token evidence budget. Change the budget and some of the tradeoffs may change too. So the conclusion is intentionally narrow.

What I would test next

The obvious next test is seed depth. Four was too narrow in this experiment and seven worked better, but I have no reason to think seven is the right number.

I’d test a range and look for the point where adding more seeds stops recovering useful evidence and starts crowding the package with noise or graph expansions. That seems more useful right now than adding another retrieval mechanism.

Research outcome

Before making the pipeline smarter, I should make sure I have tuned the obvious parts first.

This experiment reused the corpus, questions, and evidence budget from the structure-aware retrieval experiment, where the four-seed graph first showed seed starvation.

Terms used here are defined in the RAGLens glossary.

All research →