History — 2026 week 36 (Aug 31 – Sep 6, 2026)
Newest first.
2026-09-03 — The Collections LLM Judges Are Measured On (3 new, 5 updated)
The TREC Deep Learning Track ran 2019–2023 on the MS MARCO corpora — 43 deeply-judged test queries per task in 2019 against 3.2M documents and 8.8M passages, with NIST assessors applying multi-graded judgments by depth pooling and document labels inferred from passage ones. ANTIQUE supplies the non-factoid counterweight: 2,626 real Yahoo! Answers questions and 34,011 crowdsourced annotations on a four-point scale that grades whether an answer is convincing, not whether it is correct. LLMJudge, a SIGIR 2024 LLM4Eval challenge built on TREC DL 2023, benchmarks judges rather than rankers — 4,423 test pairs scored on Cohen’s κ against humans and Kendall τ against system orderings.
Datasets — TREC Deep Learning Track · ANTIQUE · LLMJudge Updated — MS MARCO · LLM as Judge · Benchmarking LLM-based Relevance Judgment Methods · Criteria-Based LLM Relevance Judgments · Charles L. A. Clarke
2026-09-03 — There Is No Single LLM Judge (6 new, 1 updated)
Two 2025 papers sharpen what LLM-as-a-judge actually names. Negar Arabzadeh and Charles L. A. Clarke’s Benchmarking LLM-based Relevance Judgment Methods runs binary, graded, pairwise-preference and two flavours of nugget-based judging side by side over TREC Deep Learning 2019–2021 and ANTIQUE, separating two things usually reported as one: agreement with human labels, which pairwise preferences win, and agreement with human system rankings, which graded and binary pointwise labels win at Kendall τ up to 0.922. Neither implies the other. Naghmeh Farzi and Laura Dietz’s Criteria-Based LLM Relevance Judgments instead decomposes relevance into exactness, coverage, topicality and contextual fit, each graded 0–3, trading roughly 5.4× runtime for a grade a human can inspect and dispute.
Corrections
- LLM as Judge advised pairwise comparison over pointwise scoring as a flat rule; the paradigm depends on whether the output is a document grade or a system ranking, and the recommendation now says so.
Articles — Benchmarking LLM-based Relevance Judgment Methods · Criteria-Based LLM Relevance Judgments People — Negar Arabzadeh · Charles L. A. Clarke · Naghmeh Farzi · Laura Dietz Updated — LLM as Judge
2026-09-02 — The Generalization Cliff: Why Leaderboard Winners Fail On Your Data (1 new, 8 updated)
General-purpose embedding models that top public leaderboards collapse once asked to retrieve on a specific domain, sometimes catastrophically. BRIGHT anchors the sharpest evidence — a model scoring 59.0 nDCG@10 on MTEB scores 18.3 on BRIGHT’s reasoning-intensive queries — while Model Selection and Fine-Tuning Evaluation’s “don’t select on public benchmarks” rule and RTEB’s open-vs-private diagnostic explain why: leaderboard position reflects benchmark fit and contamination as much as real capability. Domain fine-tuning closes most of the gap — Thierry Damiba’s SPLADE series took e-commerce nDCG@10 gains from +7.2% to +27.5% over BM25 — but has its own cliff: the fine-tuned model lost to the off-the-shelf baseline on a neighboring catalog and forgot general web queries entirely.
Topics — The Generalization Cliff (moved out of the graph to drafts/the-generalization-cliff.md on 2026-09-03)
Updated — BRIGHT · RTEB · BEIR · Model Selection and Fine-Tuning Evaluation · Retrieval Benchmarks and Leaderboards · Embedding Models Compared · Fine-Tuning Text Embeddings For Domain-Specific Search · Fine-Tuning Sparse Embeddings for E-Commerce Search
2026-09-01 — Algebraic Rewrite Speeds Up Asymmetric Re-ranking in Vespa (2 new, 4 updated)
Dainius Jocas shows how algebraic rewriting speeds up Asymmetric Re-ranking in Vespa — scoring a full-precision query embedding against binary-quantized document embeddings to recover recall that quantization loses, without changing the underlying math. Rewriting the naive per-document expression (q·(2·doc − 1) + 1) / 2 as q·doc + (1 − Σq)/2 isolates a query-only term that Vespa’s constant-folding computes once per query instead of once per document, collapsing the per-document work to a single dot product against the unpacked bits. At 1M documents this cut end-to-end latency from 267ms to 193ms, a 28% improvement. Sourced from Jocas’s personal blog, jocas.lt.
Articles — Optimize Asymmetric Re-ranking with Algebra Concepts — Asymmetric Re-ranking Updated — Binary Quantization · Reranking · Vespa · Dainius Jocas
2026-08-31 — Etsy Adds LLM-Scored Semantic Relevance to Search Ranking (5 new, 5 updated)
Etsy’s search models historically ranked on engagement signals (clicks, add-to-carts, purchases) that skew toward already-popular listings regardless of query fit. How Etsy Uses LLMs to Improve Search Relevance adds Semantic Relevance as a complementary signal: an LLM annotator anchored to human golden labels scores query-listing pairs, then a three-tier distillation cascade — an o3 annotator, a Qwen 3 VL 4B teacher, a BERT two-tower student — compresses that judgment into a model cheap enough for real-time filtering, feature enrichment, loss weighting, and boosting. Fully-relevant listing share rose from 58% to 62% between August and October 2025. Sourced from Etsy’s Code as Craft blog, by Yuqing Zhang, Congzhe Su, and Susan Liu.
Articles — How Etsy Uses LLMs to Improve Search Relevance Concepts — Semantic Relevance People — Yuqing Zhang · Congzhe Su · Susan Liu Updated — Etsy · Etsy - Search Quality and Query Understanding · E-commerce Search · Knowledge Distillation · LLM as Judge
2026-08-31 — Choosing Between RRF and DBSF Fusion in Qdrant (4 new, 7 updated)
RRF discards score magnitude entirely — a document leading by a wide margin scores the same as one leading narrowly, as long as both land at rank 1. Qdrant’s Distribution-Based Score Fusion keeps the raw scores instead, rescaling each retriever’s returned set by its own mean and 3-sigma spread before summing. How to Tune Hybrid Search in Qdrant measures both across five datasets: DBSF beat default RRF by up to 0.0383 nDCG@10 on WANDS, while RRF’s own k parameter shifted results just as much — k=2 and k=61 picked different top results for 42% of WANDS queries. Sourced from Dylan Couzon’s Qdrant tuning guide and Qdrant’s hybrid-queries reference documentation.
Articles — How to Tune Hybrid Search in Qdrant · Hybrid Queries - Qdrant Concepts — Distribution-Based Score Fusion People — Dylan Couzon Updated — Reciprocal Rank Fusion · Score Normalization · Hybrid Search · Qdrant Vector DB · Qdrant · WANDS Dataset · BEIR