eBay
Global online marketplace where third-party sellers list and describe their own inventory. That structure shapes its search problems: the catalogue text is written by the people who benefit from ranking well, and “relevance” is entangled with seller economics rather than being a purely textual property.
LLM Judgment Bound to Business Metrics
eBay’s published work on keyphrase relevance judgments is one of the few documented industrial deployments of an LLM as Judge in e-commerce search (arXiv:2505.04209). Its contribution to the wider debate is less a modelling result than a framing one: an LLM judge works as a proxy for seller judgment provided it is bound to “a meticulous evaluation framework grounded in business metrics.”
That condition is the substance. Rather than validating the judge only against human agreement statistics, eBay ties its output back to revenue, so the judge is answerable to something the marketplace can measure independently of the text being judged. It places eBay in the same pattern as Etsy and Allegro — humans define the standard, the LLM scales it, business metrics keep it honest — a more modest claim than “LLM judges match humans,” and, in the documented deployments, the one that holds.
The revenue anchoring is also a structural defence: seller-authored listing text can move a text-similarity judgment far more easily than it can move a business metric. See Adversarial Relevance Judgment.
Related Concepts
- LLM as Judge — deployed as a scaled proxy for seller relevance judgment
- Adversarial Relevance Judgment — the marketplace threat model, where sellers write the judged text
- Semantic Relevance — the signal the judge is scaling
- Search Evaluation — the practice the business-metric framework governs
Related Topics
- E-commerce Search — product relevance as substitutability rather than topicality
- Two-Sided Marketplace Ranking — buyer and seller interests in the same ranking
- Search Quality Assurance — evaluation frameworks in production
Related Articles
- Do LLM Judges Actually Agree With Us — Andrew Kornilov; places eBay’s business-metric framing alongside Etsy’s and Allegro’s deployments