Consistency Filtering
A quality gate for generated training data: after an LLM invents a query for a document, run that query through a retrieval system and keep the pair only if the source document comes back at the top. If the query cannot find the document it was written from, the pair is discarded.
The test is cheap, requires no human judgment, and catches the dominant failure mode of Synthetic Query Generation — queries too generic, too broad, or simply not answered by their source.
Why It Works
The filter is a round trip. Generation goes document → query; retrieval goes query → document. If the round trip does not return to where it started, one of two things is true, and both mean the pair should go:
- The query is not specific to the document — it matches many documents equally well, so it carries no discriminative signal and would teach a ranker nothing. See Query Specificity.
- The query is not actually answered by the document — the generator hallucinated or drifted, making the pair an outright false positive.
What survives is, by construction, a query for which the source document is the best available answer in the corpus. That is precisely the relation a ranking model is being trained to learn.
The Retention Rate as a Signal
The filter’s pass rate is a free diagnostic on generation quality. In Improving Search Ranking with Few-Shot Prompting of LLMs, with the zero-shot hybrid model from part two as the filter and a rank-#1 requirement:
| Stage | Pairs |
|---|---|
| Generated by flan-t5-xl | 33,099 |
| Passed consistency check | 14,156 |
| Retention | 43% |
The article reads 43% as indicating reasonable generation quality. The interpretive rule is loose but useful: a very low rate suggests the prompt is producing generic or wrong queries; a very high rate suggests the filter is too permissive to be doing work, or that the generated queries are near-copies of their documents and so trivially retrievable.
Design Choices
How strict is the threshold? Rank-#1 is the strictest form. Looser variants — top-5, top-10 — retain more data at the cost of admitting weaker pairs. The strict setting is affordable when the generator can cover more documents than you need, which is usually the case since generation is offline.
Which retriever does the filtering? Note the bootstrap: the filter in the worked example is the existing zero-shot ranking model. A better baseline makes a better filter, which makes better training data, which makes a better model. The starting baseline therefore matters twice over — part of why that series spends a whole post tuning BM25 before touching neural methods.
What about the discards? Documents whose generated queries failed the check simply contribute no positive pair. They remain in the corpus and can still be sampled as negatives — see Hard Negative Mining.
Related Concepts
- Synthetic Query Generation — the pipeline this is a stage of
- Hard Negative Mining — the next stage; negatives drawn from the same retrieved list
- Query Specificity — the property the filter implicitly tests for
- Judgment Lists — what the filtered output stands in for
- LLM as Judge — the alternative validation route: ask a model to score relevance rather than test retrievability
- Zero-Shot Retrieval — the baseline model that does the filtering
- PROMPTAGATOR — prior work built on the same round-trip idea
Articles
- Improving Search Ranking with Few-Shot Prompting of LLMs — Jo Kristian Bergum (Vespa); 33,099 → 14,156 pairs at a rank-#1 threshold