Awesome Search KG

Learning to Rank

6 min read

Learning to Rank

Definition

Learning to Rank (LTR) is the application of machine learning to the ranking problem in information retrieval — training models to produce optimal rankings given a query and a set of candidate documents, using labeled (query, document, relevance) training data.

Why LTR?

Hand-tuned ranking formulas (BM25 weights, field boosts) are:

  • Brittle: require manual adjustment for new query types
  • Suboptimal: can’t model complex non-linear feature interactions
  • Domain-specific: no transfer across product categories or query types

LTR learns the ranking function from data, capturing complex signals (TF-IDF × freshness × popularity × semantic similarity) that are impossible to tune manually.

LTR Formulations

Pointwise

Score each document independently. Treat as regression/classification.

  • Loss: MSE on relevance grade
  • Simple but ignores relative ordering

Pairwise

For each query, learn that doc A should rank above doc B.

  • Input: (query, doc_A, doc_B, which_is_better)
  • Loss: hinge loss / logistic loss on pairwise preferences
  • RankNet (2005): foundational pairwise approach

Listwise

Optimize a ranking metric directly (NDCG, MAP) over the full ranked list.

  • Loss: gradient of NDCG (tricky — NDCG is non-differentiable)
  • LambdaMART (2010): dominant practical algorithm
  • Uses NDCG gain as gradient weighting

LambdaMART

LambdaMART is the dominant LTR algorithm in production:

  1. MART (Multiple Additive Regression Trees) = gradient boosted decision trees (GBDTs)
  2. Lambda = gradient weighting by NDCG improvement of swapping two documents
from lightgbm import LGBMRanker
 
model = LGBMRanker(
    objective="lambdarank",
    metric="ndcg",
    ndcg_eval_at=[5, 10],
    num_leaves=31,
    n_estimators=100
)
model.fit(X_train, y_train, group=query_sizes)

Features used:

  • BM25 score (title, body, description)
  • Semantic similarity (embedding cosine)
  • Popularity (click rate, sales)
  • Freshness (recency)
  • Price, rating
  • User features (if personalized)

Two-Phase: Retrieval → Ranking

LTR is almost always the second stage in a Retrieval Pipeline:

Fast first-stage (BM25 or bi-encoder) → top-1000
    ↓
LTR reranker (LambdaMART or neural) → top-20

The first stage uses simple fast signals; LTR adds expensive-to-compute features.

Where LTR Runs: In-Engine vs External Re-ranker

Independent of the algorithm, a learned ranker can be served in two architectures:

  • In-engine LTR — the model runs inside the search engine as a rescore phase over the top-N, with features computed from the engine’s own queries and fields. Examples: Solr LTR, Elasticsearch Learning to Rank (both the OSC plugin and Elastic’s native module), OpenSearch, Vespa. Lighter integration, no extra service — but tied to a single engine’s index.
  • External secondary re-ranker — the model runs as a separate service downstream of the engine: the application retrieves candidates, then calls the re-ranker. Engine-agnostic, so it can fuse results from multiple independent retrievers and ingest behavioral events directly — at the cost of an extra network hop and a stateful service to operate. Canonical example: Metarank.
In-engine LTRExternal re-ranker
Where it runsInside the search cluster, as a rescore phaseSeparate service downstream of the engine
Feature computationThe engine’s own query/field signalsOwn feature pipeline + Feature Store
Retriever scopeTied to one engine’s indexRetrieval-agnostic — can fuse multiple retrievers (Hybrid Search)
Behavioral eventsYou build Judgment Lists / Implicit Judgments yourselfIngests clicks/impressions directly; synthesizes negatives
Network hopsNone — single search requestExtra hop; app multi-queries then calls the re-ranker
Ops costNo new serviceA stateful service to run and scale
ExamplesSolr LTR, Elasticsearch Learning to Rank, OpenSearch, VespaMetarank

Rule of thumb: in-engine LTR is the lighter integration when all candidates live in one engine’s index; an external re-ranker earns its keep for real-time personalization from behavioral events or fusing multiple independent retrievers. See Learn-to-Rank with OpenSearch and Metarank (the OpenSearch Remote Ranker Plugin RFC is a convergence point between the two) and Reranking.

Neural LTR

More recent: neural rerankers replace LambdaMART:

  • Cross-Encoder: joint query-document encoding, listwise softmax loss
  • MonoT5: T5-based pointwise ranker
  • RankLLaMA: LLaMA fine-tuned for ranking
  • RankGPT: listwise LLM permutation reranker

Training Data

LTR requires labeled data:

  • Explicit judgments: Judgment Lists with annotated (query, doc, grade) triples
  • Click data: position-debiased click labels from query logs (Click Signals)
  • Conversion data: purchase signals for e-commerce

What the Model Optimizes Is a Business Decision

The “perfect ranking” that NDCG (and hence LambdaMART’s lambdas) is computed against is constructed from interaction weights — and that construction encodes a business goal. Training on raw clicks optimizes for curiosity, not value: in Roman Grebennikov’s Findify war story, a giant bong shot to the top of a collection because everyone clicked it out of curiosity and nobody bought it. Weighting purchases far above clicks in the label construction sent it back down — and the same lever can optimize for margin instead of purchases. Decide the target metric before training; see Roman Grebennikov - Personalizing Search Results in Real-Time.

Position Bias in LTR Training

Click data has position bias — documents at rank 1 get many clicks regardless of quality. Must correct with:

  • IPS (Inverse Propensity Scoring): weight clicks by inverse position probability
  • Counterfactual evaluation: compare click rates controlling for position

Unbiased training data → significantly better LTR models.

Feedback Loops: Don’t Train on Data Your Model Produced

A model trained on clicks over its own rankings eats its own output: users click top positions regardless of quality, so the model reinforces whatever it already ranks highly. Symptom (Findify): A/B metrics improve after launch, then slowly degrade (+8% → +6%). Fix: an Exploration vs Exploitation traffic split — a small (~1%) segment serves the first page shuffled, and only that segment feeds training, so position cancels out over impressions. IPS is the statistical alternative when you can’t afford exploration traffic. See Roman Grebennikov - Personalizing Search Results in Real-Time.

Articles

People

Graph View

Backlinks