Retrieval design · WixQA · Decision note
Filtering would not fix the failures left after broader retrieval
Would applicability filtering address the retrieval failures left after increasing retrieval depth? No. Wrong-scope competition explains too few of the remaining failures.
After the retrieval-breadth experiment, the obvious next lever was applicability filtering: restrict the candidate set to the right product before ranking, so wrong-scope articles stop competing with the right one. Before running it, I checked whether filtering was likely to improve retrieval enough to justify an experiment.
With whole articles and unchanged embeddings, filtering only removes candidates; it does not change similarity scores. For every gold article already present in the stored top-100 rankings, its filtered rank can therefore be calculated directly. No new retrieval was needed.
Gold-article recall across 258 gold articles for 200 expert-written questions. Changes from baseline are shown in points.
| Filter | recall@5 | recall@10 | recall@20 | gold removed |
|---|---|---|---|---|
| No filter (breadth experiment baseline) | 69.8% | 81.0% | 91.1% | 0 |
| Product filter, correct product known, unlabeled articles kept | 72.1% (+2.3) | 84.5% (+3.5) | 92.6% (+1.5) | 0 |
| Product filter, correct product known, unlabeled articles excluded | 73.6% (+3.8) | 82.6% (+1.6) | 90.7% (−0.4) | 10 |
| Product filter using the product the question names | 67.1% (−2.7) | 79.1% (−1.9) | 88.0% (−3.1) | 10 |
| Drop feature requests and known issues | 72.1% (+2.3) | 81.8% (+0.8) | 90.3% (−0.8) | 3 |
| Correct product known and feature requests dropped (ceiling) | 75.2% (+5.4) | 84.5% (+3.5) | 93.0% (+1.9) | 3 |
For comparison, increasing k from 5 to 10 in the breadth experiment added 11.3 points of recall.
Filtering buys little because most of the competition is already in scope. Of the documents ranked above a gold article, 347 share its product, 600 carry no product label, and only 286 belong to another product. And 133 of the 258 gold articles have no product label at all. A strict filter that excludes unlabeled content therefore loses recall at k=20 and removes 10 gold articles.
The product named in a question is not reliable enough to use as a hard filter either. Of the 76 questions that name a product, 8 name one that does not match the gold article’s product. Filtering on that signal removes 10 gold articles and lowers recall at every depth.
Among the gold articles ranked beyond 20 for questions that name a product, none reaches the top 10 under the oracle product filter. Filtering does not recover what broader retrieval misses.
The result argues against hard filtering on stated scope. Boosting is a possible alternative, but it was not tested here.
Decision: do not build a product filter for this corpus. Any benefit is capped by label coverage, and feature requests must remain retrievable because three gold answers are themselves feature requests.
After the oracle product filter, 38 gold articles still sit beyond rank 5. In 27 of those cases, the articles above them come from the same product: the right scope, but the wrong article, often a long omnibus article outranked by narrower siblings.
That leaves a within-scope discrimination problem rather than an applicability problem. The next investigation is structural chunking, which will test whether changing the representation improves those cases.
How this was checked
- stored top-100 rankings from the breadth experiment, run 2026-09-07T16-40-08-365Z; 200 expert-written questions, 258 gold articles
- product = the Help Center URL slug prefix; 60% of articles carry one, and 95 questions have no product on any gold article
- “correct product known” uses the gold article’s own product, so it is an upper bound on what a product filter could do
- editor family (Wix Editor, Studio, Editor X, ADI) was also checked; it changes nine ranks and no recall figure
- the best filter moves the mean gold rank from 6.75 to 5.61 and the median from 3 to 2
- seven gold articles sit outside the stored top-100; their filtered rank is unknown
- no retrieval, embedding, or generation was run for this note
The full audit and its scripts are committed in the RAGLens repository under experiments/studies/foundation-02-filtering-audit.
Terms used here are defined in the RAGLens glossary.