Retrieval design · WixQA · Experiment
Does chunking help when whole articles fail?
I wanted to know whether changing the shape of the source content would help with retrieval failures that remained after the breadth and filtering work. The content, questions, and embedding model stayed the same. This time, instead of retrieving whole articles, I retrieved sections.
Across the 200 questions, 258 source articles counted as correct, and 77 still fell below the first five results. Long articles showed up more often among those misses, and many contained the relevant answer in a much smaller section. So I split the 6,221 help-center articles at their headings into 19,706 sections and ranked each article by its best-matching section.
Sections with title and heading, compared with whole articles, across 258 correct sources.
Net: +5 correct sources in the top five, out of 258.
The change moved a lot of individual sources without moving the overall numbers very much. Twenty-four sources that whole-article retrieval had missed moved into the top five, while 19 previously successful ones dropped out. That left a net gain of five. The share of correct sources found in the top five rose from 72.4% to 74.0%, and in the top 10 from 83.2% to 86.0%. By the top 20, there was effectively no difference.
| Share of correct sources found | top 5 | top 10 | top 20 |
|---|---|---|---|
| Whole articles | 72.4% | 83.2% | 92.3% |
| Sections with title and heading | 74.0% | 86.0% | 92.3% |
| Change, in points | +1.6 | +2.8 | −0.1 |
| k | Whole articles | Sections with title and heading |
|---|---|---|
| 1 | 34.8% | 39.2% |
| 3 | 60.2% | 64.0% |
| 5 | 72.4% | 74.0% |
| 10 | 83.2% | 86.0% |
| 20 | 92.3% | 92.3% |
| 50 | 96.4% | 97.2% |
The individual cases make the trade-off easier to see. Wix Editor: Managing Advanced Menus moved from rank 10 to rank 1 for a question about renaming a menu item because its 40-word “To rename a menu item” section could now compete on its own. But for a different question about menu text not updating, the same article went from rank 33 to missing entirely. Its sections matched the first question well and the second poorly, while sections from shorter articles filled the available slots. Receiving Your Payouts from Wix Payments showed the same downside, falling from rank 5 to missing behind a 45-word “Still need help?” section from another article.
Preserving the article title and heading with each section helped somewhat. Removing that context cost one to four points at most depths and lost a few more previously found sources, but it did not change the overall pattern. The biggest change came from making sections independently retrievable.
Another assumption weakened too. I expected questions with little vocabulary overlap with the correct article to be harder to recover. They were, but only slightly: 23% recovered, compared with 28% for the long-article group. Word overlap helped describe the failures, but it was not very useful for predicting which ones a dense retriever could fix.
Structural chunking changed retrieval substantially without producing a clean improvement. It recovered many failures while creating nearly as many new ones. Smaller units can expose relevant content that gets diluted inside a long article, but they also create many more pieces of content that can outrank the right source.
For a product or content team, that is the useful part: smaller, more precise chunks are not automatically better. Measure what they recover and what they break. The next question is whether we can take advantage of a strong section match without letting one small or generic section determine how the whole article ranks.
Test details
- WixQA Help Center snapshot: 6,221 articles, retrieved whole in the baseline; 19,706 heading-level sections (floor 40 / ceiling 400 words, tab panels split, no boilerplate removal) in the chunked arms
- 200 expert-written questions, 258 gold articles (52 questions have more than one); the simulated 200-question set as a replication appendix: same shape at k ≤ 5, the k=10 gain did not replicate (−2.0 points)
- Voyage voyage-3 for articles and sections alike; query vectors embedded once and the identical vectors used against all three indexes. The whole-article index was re-queried, not re-embedded; against the breadth experiment’s rankings 240 of 258 gold kept their rank and none moved five or more
- chunked arms: article rank = rank of its first (highest-scoring) section; top-500 sections collapsed to the first 100 distinct articles, no shortfalls; k always counts articles
- bare-section arm: the same sections without title or heading; below the title-and-heading arm at five of six depths (81 gold ranked better with context, 74 without)
- pre-declared decision rule: positive meant roughly +3 points at k=5 with fewer than five regressions; null meant within ±1 point with noise-level movement; observed was +1.6 with 19 regressions, so the result met neither branch and is recorded as mixed
- known limitation: section text was derived from the articles’ HTML, while the whole-article baseline used the released stripped text; the two agree closely (median token overlap 0.95 on the 209 unique gold articles) and where they differ the stripped text is the degraded side
- recall = gold articles present in the top-k ÷ gold articles for the question; multi-article questions keep each gold as its own requirement
- 6.2M embedding tokens, about $0.37; no language model was called
Runs 2026-09-10T16-20-50-786Z (whole articles), 2026-09-10T16-30-16-022Z (sections), 2026-09-10T16-31-01-216Z (bare sections). The stored rankings, design note, run record, and movement analysis are committed in the RAGLens repository under experiments/studies/foundation-03-representation.
Terms used here are defined in the RAGLens glossary.