Awesome Search KG

LTRR: Learning To Rank Retrievers for LLMs

3 min read

LTRR: Learning To Rank Retrievers for LLMs

Authors: To Eun Kim, Fernando Diaz (Carnegie Mellon University) Venue: SIGIR 2026 (also a SIGIR 2025 LiveRAG spotlight) · arXiv 2506.13743 · code

Summary

RAG systems usually hard-wire one retriever even though no retriever wins on every query. LTRR frames the choice as learning to rank retrievers: for each query, rank a pool of retrievers by how much each would improve the LLM’s answer, and send the query to the top one. The authors place this explicitly in the lineage of distributed IR and Federated Search — resource selection, with retrievers as the resources.

Two design choices distinguish it:

  • The target is downstream utility, not retrieval relevance. A retriever’s label is the gain in answer quality it produces over answering with no retrieval, min-max normalised per query.
  • “No retrieval” is a candidate route. The router can decide the LLM is better off without context — merging which retriever and whether to retrieve into one decision (compare Adaptive Retrieval).

Setup

  • Retriever pool: BM25 and E5-base over a sampled FineWeb corpus, each plain or followed by one of two rerankers — six retrievers, plus the no-retrieval option. The router sees retrievers only through their outputs, not the corpus.
  • Features: pre-retrieval (query embedding reduced to 32 dimensions by PCA, query length, query type) and post-retrieval per retriever — query-to-results similarity (overall, average, max, variance), a Moran coefficient of semantic autocorrelation among the results, and how similar a retriever’s results are to the other retrievers’.
  • Models: pointwise, pairwise and listwise LTR — XGBoost, SVMRank, a feedforward network, DeBERTa — plus five train-free heuristic routers.
  • Labels: utility measured by BEM or by Answer Correctness (AC).
  • Data: a synthetic QA set generated with DataMorgana for controlled variation in question types.

Findings

  • Routing beats the strongest single-retriever RAG baseline, with pairwise XGBoost the best router.
  • Gains are statistically significant only for routers trained on the Answer Correctness objective; BEM-trained routers show no significant gain, which the authors attribute to metric reliability — the choice of utility metric matters as much as the model.
  • The improvements hold in-distribution and in some of the four unseen-query-type splits (multi-aspect, comparison, complex, open-ended — each held out of training).

Caveat

Post-retrieval features require running every retriever before routing. This router optimises answer quality, not retrieval cost — unlike cost-motivated routers such as Adaptive-RAG or RAGRoute, which decide before retrieving.

Graph View

Backlinks