Awesome Search KG

Creating Judgement Lists with Quepid

3 min read

Creating Judgement Lists with Quepid

A practical guide to establishing a collaborative system for creating relevance Judgment Lists with Quepid, the open-source evaluation tool from OpenSource Connections. The emphasis is on the human side: getting a team to produce consistent, well-defined judgements.

The Core Idea

A judgement list pairs queries with relevance ratings for their results, acting as a test collection for benchmarking search quality. Quepid supports four functions around it:

  1. Building query sets
  2. Creating judgement lists
  3. Computing search quality metrics
  4. Comparing different search algorithms via those metrics

Information Needs

The article’s central best practice: define an explicit information need behind each query. “best action movie” is ambiguous — recent? highly-rated? all action movies? Writing the information need down is what lets human raters judge consistently.

Setup Workflow

  1. Prerequisites — an Elasticsearch instance with indexed data (the tutorial uses TMDB via the es-tmdb repo).
  2. Case — stores queries, tuning settings, and the engine connection. Name it descriptively (“Movie Search Baseline”), configure the connection (API keys for Cloud), pick display fields (title, overview, cast, poster), add representative queries.
  3. Teams — share cases and rate collectively to improve assessment quality.
  4. Book of Judgements — structures the evaluation: naming convention, team assignment, scorer (e.g. DCG@10 → NDCG), and a selection strategy (single vs. multiple raters).

Human Rating Process

The rating interface shows the query, the information need, scoring guidelines, document metadata, and rating buttons (with keyboard shortcuts).

Example for the query “harrison ford”:

  • Heroes → Perfect (3) — Ford is in the cast
  • Ford v Ferrari → Poor (0) — “Ford” is the automaker, not the actor
  • Action Jackson for “best action movie” → Fair (1) — matches genre but rated 5.4/10

Rater challenges & remedies

Challenges: fatigue, personal bias, uneven domain expertise, competing priorities, subjectivity. Remedies: review query + information need + metadata systematically, consult guidelines with concrete examples, use “I Will Judge Later” / “I Can’t Tell” when unsure, and take breaks (Quepid gives visual rewards per batch).

Advanced Features

  • Query Sandbox — iterate on field weights, boosts, and query logic; metrics update immediately without touching production.
  • Inter-rater reliability — a Fleiss’ Kappa notebook surfaces ambiguous queries and data-quality issues.
  • AI-generated judgements — as of Quepid v8, judgements can be generated by an LLM to scale the process (LLM as Judge).

Follow-Up Workflow

Baseline metrics → hypotheses → experiments in the Sandbox → validate against the metric → iterate, without risking production.

People

Tools

Graph View

Backlinks