RAGLens

Ask the Google Developer Style Guide corpus: “How should I write alt text for images in documentation?” It sounds like exactly the kind of question a style guide answers. In a live run, retrieval scored 0.15 and the answer scored 0.95 — a spread a single aggregate score would flatten into an unremarkable “borderline.”

The two numbers tell different stories, and both are correct. Retrieval scored low because the indexed corpus did not provide retrievable support: an exhaustive pass over every indexed chunk found nothing above the calibrated relevance threshold. The answer scored high because it did the right thing with that situation — it said the sources don’t cover alt text instead of inventing guidance.

That is likely a content gap — and “likely” is doing deliberate work. An exhaustive retrieval pass proves only that this retriever found nothing; the content may exist in a form this retriever cannot match. Confirming a gap is the corpus owner’s call, so the diagnosis routes the question there rather than declaring the corpus incomplete.

Language models are not deterministic, and the same question does not always get the same behavior. Sometimes the model declines cleanly — a correct refusal, scored as healthy behavior, not failure. Sometimes it answers anyway, producing plausible-sounding alt-text guidance from its training data with nothing retrievable behind it.

The evidence facts are identical in both runs; what changes is what the model did with them. The diagnosis tells the branches apart: the refusal is labeled a correct abstention, the confident answer is labeled answered without retrievable support. That variance isn’t a flaw in the demo — it is the failure mode RAGLens exists to catch, because the unsupported answer is the one that reads most convincingly.