History — 2026 week 46 (Nov 9 – Nov 15, 2026)
Newest first.
2026-11-14 — llm-search-eval (1 new, 1 updated)
Gold labels run out when a new retriever surfaces products nobody judged, and a small metric gain over a few dozen queries can be chance. llm-search-eval, an MIT-licensed Python tool first released in October 2026, fills unlabeled pairs with LLM judges while keeping gold labels authoritative, measures agreement within judge panels and against gold, and attaches a paired randomization test and bootstrap intervals to every system comparison, with a CI gate on top. Offline Search Evaluation Tools now covers it. Described from the project’s own documentation; single author, no independent adoption yet.
Tools — llm-search-eval (LLM judge panels, paired statistics, CI gate)
Updated — Offline Search Evaluation Tools
2026-11-14 — Judge calibration in Quepid (0 new, 2 updated)
Quepid has generated relevance judgments with an LLM since v8, and a proposed judge calibration tool now checks whether those judgments can be trusted. An AI judge re-rates a random sample of pairs a reference has already rated, and agreement is reported at two levels: per pair, with ordinal Krippendorff’s α and a confusion table, and per ranking, with Kendall’s τ over per-query nDCG@10. Prompt changes are re-run on the same frozen pairs, and ratings and case scores are never touched. Offline Search Evaluation Tools now describes it; the case for it is in Calibrate Before You Trust. Still an open pull request.
Updated — Quepid · Offline Search Evaluation Tools
2026-11-14 — The offline search evaluation tooling landscape (1 new, 3 updated)
Offline evaluation is a pipeline: sample queries, collect judgments, run systems, compute metrics, test the difference. No single tool covers it end to end. Offline Search Evaluation Tools maps six layers, from reference metric engines such as trec_eval and ranx and the PyTerrier experiment framework, through the Elasticsearch ranking evaluation API and workbenches like Quepid and Rated Ranking Evaluator, to LLM judgment generators such as UMBRELA and RAG evaluators like RAGAS. The thin spots are clear: significance testing is found in research libraries but not in the workbenches, judge validation trails LLM judging, and stratified Query Sampling is barely tooled. Built from project repositories and documentation.
Topics — Offline Search Evaluation Tools
Updated — Relevance Evaluation Tools Compared · Retrieval Benchmarks and Leaderboards · Query Sampling
2026-11-14 — Alternative queries for zero results at Wallapop (3 new, 5 updated)
When a marketplace search returns nothing, you can drop one query term or swap a model number for its sibling. Rene Kriegler & Jean Silva - Alternative Queries for Zero Results follows up Rene Kriegler’s 2019 talk Rene Kriegler - Query Relaxation on Spanish queries. A fine-tuned RoBERTa classifier matches his old neural network at 0.98 accuracy once a token-order bug is fixed. A fill-mask model proposes replacement tokens, with mixed results. Wallapop shipped the simplest option instead: drop the rarest term. It scored about 80% offline against log-mined query pairs and was heading into an A/B test. Query Relaxation now covers token replacement and how to build that evaluation data. Haystack EU 2023 talk; speakers’ own figures.
Videos — Rene Kriegler & Jean Silva - Alternative Queries for Zero Results People — Jean Silva Companies — Wallapop Updated — Rene Kriegler - Query Relaxation (links to the follow-up) · Query Relaxation · Zero Results · Rene Kriegler · Haystack EU
2026-11-13 — Query relaxation judged by conversion, not empty pages (3 new, 5 updated)
Dropping words until a search returns something can cut the Zero Results rate while still losing the sale. Aron Negyesi’s post, added as Query relaxation - the forgotten hero of eCommerce search, recounts A/B tests at Rohlik where relaxed queries first converted at a fifth of the normal rate, then at over 60% after a better choice of which terms to drop. A spare-part search on a home improvement retailer’s site shows the failure from outside: the rare brand and model code appear to have been discarded. Query Relaxation and Search UX now cover measuring relaxed-query conversion and showing shoppers which words were dropped. Medium member-only post; figures are the author’s own.
Articles — Query relaxation - the forgotten hero of eCommerce search People — Aron Negyesi Companies — Rohlik Updated — Query Relaxation · Zero Results · Rene Kriegler - Query Relaxation · E-commerce Search · Search UX
2026-11-11 — Kendall, Spearman and Pearson side by side (0 new, 2 updated)
Judge papers quote several correlation coefficients for the same comparison, and the numbers are not interchangeable. Kendall Rank Correlation now sets Kendall’s τ beside Spearman’s ρ and Pearson’s r: what each compares, what each is sensitive to, and why τ normally comes out lower than the other two on the same data. The PoLL jury reached Pearson 0.917 but τ 0.778 against the same Chatbot Arena leaderboard. Only τ should be held to the τ ≥ 0.9 equivalence bar. Replacing Judges with Juries now links its correlation figures to the comparison.
Updated — Kendall Rank Correlation (Spearman and Pearson section, aliases) · Replacing Judges with Juries
2026-11-11 — LLM juries, judge cascades and claim-level fact-checking (12 new, 7 updated)
Asking one large model for every verdict repeats its systematic errors and favours its own outputs. LLM Jury covers Cohere’s PoLL paper, where three small judges from different model families agreed with human labels on KILT question answering better than GPT-4, at seven to eight times lower cost. Atomic Claim Verification checks long answers one fact at a time, following FActScore and Google DeepMind’s search agent SAFE. Staged Judging now covers Trust or Escalate, in which cheap judges pass unsure cases upward under a calibrated, provable human-agreement target. All figures are the papers’ own.
Corrections
- LLM as Judge advised using a capable model (GPT-4, Claude) because cheaper ones calibrate worse → it now notes that GPT-4 underperformed small judges on reference-based QA, and that small judges can estimate their own reliability about as well as GPT-4.
Articles — Replacing Judges with Juries · FActScore - Fine-grained Atomic Evaluation of Factual Precision · Long-form Factuality in Large Language Models · Trust or Escalate - LLM Judges with Provable Guarantees for Human Agreement Concepts — LLM Jury (panel of diverse judges, PoLL) · Atomic Claim Verification (FActScore, SAFE, F1@K) People — Pat Verga · Sewon Min · Jerry Wei · Jaehun Jung · Yejin Choi Datasets — LongFact
Updated — LLM as Judge · Staged Judging (cascaded selective evaluation) · Hallucination Detection (claim-level family) · Cohere · Google · Meta · Natural Questions
2026-11-10 — LLM as Judges: A Vocabulary, published (1 new, 7 updated)
Agreement, self-consistency and calibration each make a separate, narrower claim about an AI relevance judge, and none of them means the judge is right. LLM as Judges - A Vocabulary, Andrew Kornilov’s published glossary written from AI-as-judge work in Quepid, defines nine such terms: the four levels of agreement, inter-judge and ordinal-versus-binary agreement, self-consistency, qrels, adjudication, distilled students, two senses of calibration, and juries. It opens with a short primer on the statistics it quotes, from Cohen’s κ to macro F1. Much of its evidence comes from two 2026 arXiv papers, Better Retrievers, Worse Labels and CADRE.
Articles — LLM as Judges - A Vocabulary
Updated — LLM as Judge · Judge Calibration · Levels of Judge Agreement · Macro and Micro Averaging · Search Quality Assurance · Andrew Kornilov · Quepid
2026-11-10 — Behavioural effects and biases in search (1 new, 5 updated)
Shoppers do not judge results one at a time. A pricey iPhone they click but never buy can make the cheaper phones beside it look better, Autocomplete suggestions anchor what people end up typing, and reviews pull clicks regardless of fit. Psychological and Behavioural Effects and Biases in Search gathers these choice effects with the biases in how results are examined (Position Bias, trust bias, grid viewports) and in what gets logged (Presentation Bias, Impression Bias). It also covers novelty effects in B tests and order and verbosity bias in LLM judges. It draws on Daniel Tunkelang, on Roman Grebennikov’s curiosity clicks, and on Andreas Wagner, who reads the same click-heavy, never-bought pattern as a possible decoy.
Topics — Psychological and Behavioural Effects and Biases in Search
Updated — Position Bias · Presentation Bias · Impression Bias · Search UX · Why nDCG Misleads eCommerce Search
2026-11-10 — Products that sell their neighbours (0 new, 3 updated)
Some products draw many clicks and almost no sales, yet the products shown beside them seem to sell more. The latest iPhone gets the clicks, and the cheaper phones next to it start to look appealing. Andreas Wagner’s third post, now added to Why nDCG Misleads eCommerce Search, links this to attention spillover, halo and decoy effects. It argues that Learning to Rank objectives which score each product on its own can pick a worse overall ranking. Two shoe grids with the same expert rating differ in one slot, and one is marked at 8% less GMV. Wagner notes the pattern is correlational and would need controlled experiments.
Updated — Why nDCG Misleads eCommerce Search · Learning to Rank · Andreas Wagner
2026-11-09 — F1 in search, and macro vs micro averaging (2 new, 10 updated)
F1 scores a set, and search is judged by rankings, so most F1 figures in search measure a judge, classifier or router rather than the results page. Macro and Micro Averaging covers where that misleads. One overall F1 hides the hard boundaries: Allegro’s judge scored per-class F1 of 0.94 on exact matches but 0.33 on telling degrees of substitutability apart. On a literature-screening set, a GEPA-tuned prompt raised F1 ten points while deferring more true positives. Progressive scroll is the exception, where Doug Turnbull argues F1 over the whole result set fits better than NDCG. Macro, micro and weighted averaging follow the scikit-learn user guide; micro-F1 equals accuracy in single-label tasks.
Articles — scikit-learn - Metrics and Scoring Concepts — Macro and Micro Averaging (F1 averaging and its limits in search)
Updated — Precision and Recall · How Etsy Uses LLMs to Improve Search Relevance · Fine-Tuning Qwen3 Embeddings for Product Category Classification · Lightweight Query Routing for Adaptive RAG - A Baseline Study on RAGRouter-Bench · Knowledge Distillation · LoRA · Embedding Fine-tuning · Query Classification · LLM as Judge Vocabulary · Outcome-Based vs Semantic Relevance
2026-11-09 — Why nDCG misleads eCommerce search (1 new, 9 updated)
Is nDCG the right number to optimise for product search? Andreas Wagner of searchHub argues not, in two LinkedIn posts now combined as Why nDCG Misleads eCommerce Search. Its log discount is assumed rather than measured, nDCG@5 or @10 ignores how far shoppers scroll, and product grids make attention drop in steps at row and viewport boundaries that differ between desktop and mobile, which a smooth curve cannot model. Its tie to revenue also weakens once images, prices, brands and promotions drive clicks. NDCG gains an eCommerce-limitations section and Position Bias now covers grid-shaped examination. Andrew Kornilov’s reply is recorded: teams use nDCG because nothing clearer is implementable.
Articles — Why nDCG Misleads eCommerce Search
Updated — NDCG · Andreas Wagner · Position Bias · Presentation Bias · Click Models · searchHub · Andrew Kornilov · Search Quality Assurance · E-commerce Search