Andreas Wagner

Background

Andreas Wagner is a search expert at SearchHub who authors the “Three Pillars of Search Quality in E-commerce” series. His framework organizes e-commerce search quality into three independent but related pillars.

Key Contribution: Three Pillars Framework

Wagner’s framework for e-commerce search quality:

1. Findability — can users find what they’re looking for at all?

   • Zero results, spelling correction, synonym coverage
   • Measured by: zero results rate, task success rate

2. Ranking / Relevance — are the results in the right order?

   • NDCG, precision, conversion correlation
   • Measured by: NDCG@k, CTR by rank

3. Discovery & Inspiration — do users find things they didn’t know they wanted?

   • Diversity Metrics, serendipity, exploration
   • Measured by: discovery rate, basket size, session depth

Each pillar requires different optimization strategies and different metrics — a system can excel at findability but fail at discovery.

Second Line of Work: Evaluation at Scale

Distinct from the quality taxonomy above, Wagner also writes on the economics of relevance evaluation — how to run LLM judgments across billions of query-document pairs when judging everything with a frontier model is not affordable at any budget.

The architecture: prune the pair space using implicit feedback to separate easy positives from candidate hard negatives, cascade the survivors through cheap quantized embedding judges before escalating disagreements to an LLM (Staged Judging), then distill the LLM’s judgments into a small model with sub-20ms CPU inference.

Where the Three Pillars framework answers what quality means, this answers how you can afford to measure it.

Critique of nDCG for eCommerce

In a three-part LinkedIn series (November 2026) Wagner, citing about 20 years in eCommerce information retrieval, argues that NDCG should not be the default target for product search: its log discount rarely matches measured shopper attention, small cutoffs ignore how far shoppers scroll, product grids make exposure drop in steps at row and viewport boundaries that differ by device, and the metric correlates weakly with business KPIs. The third part questions per-item ranking objectives in Learning to Rank: products with high CTR and low CVR may make the products displayed near them more appealing (attention spillover, halo and decoy effects), so optimizing each product’s own conversion probability could produce a worse overall ranking, a hypothesis he says needs controlled experiments. See Why nDCG Misleads eCommerce Search.

Articles in Vault

• Three Pillars of Search Quality - Findability — Findability pillar

• Three Pillars of Search Quality - Discovery and Inspiration — Discovery pillar: diversity, product exposure, multi-objective ranking

• Common Pitfalls of Onsite Search Experimentation

• Towards Scalable Relevance Engineering — judge economics: 93% problem reduction, staged judging, distillation to CPU-servable models

• Why nDCG Misleads eCommerce Search — nDCG’s discount, cutoff and KPI correlation questioned for product grids

Topics

• Search Result Diversity — MICES talk on result positioning and basket composition; Discovery pillar framework

Affiliation

• searchHub

Related People

• Daniel Tunkelang — overlapping quality framework
• Doug Turnbull — shared e-commerce search focus

Related Concepts

• Staged Judging — cascade architecture for affordable relevance judging
• LLM as Judge — the expensive stage his architecture economizes
• Search Evaluation — three pillars provide structure
• Diversity Metrics — discovery pillar
• NDCG — ranking/relevance pillar
• Query Understanding — findability pillar

On Jev

Grants Jev a promising path toward computational efficiency while arguing it does not mitigate the problem that matters — models asserting incorrect outputs with high confidence. See Reception of Jev.