Allegro

Allegro is an e-commerce platform operating a product catalog spanning 13 departments across four languages — Polish, Czech, Slovak, and Hungarian. Its engineering blog (Allegro Tech) has documented the team’s Relevance Assessment Tool (RAT), an LLM-as-a-judge framework for evaluating search relevance at catalog scale.

People

• Joanna Marhula — RAT framework co-author
• Mateusz Sidor — RAT framework co-author

Published Articles

• Automating Search Relevance Assessment at Scale with LLM-as-a-Judge — RAT framework; multilingual judgment dataset, prompt design findings, local model migration for cost

Referenced In

• Do LLM Judges Actually Agree With Us — Andrew Kornilov; uses RAT’s overall quadratic-weighted kappa of 0.69 against per-class F1 of 0.51 and 0.33 on the substitutable/highly-substitutable boundary as the central argument for reporting agreement per class rather than in aggregate

Related Concepts

• LLM as Judge · Inter-Annotator Agreement · Prompt Sensitivity · Levels of Judge Agreement