LocalNews
Overview
LocalNews is a paragraph-level benchmark for diverse news retrieval, built by the authors of NEWSCOPE to evaluate whether a retrieval system surfaces complementary perspectives on an event rather than repeating the dominant narrative. It is not a relevance benchmark with a diversity extension — the retrieval unit and the annotation are both designed around aspect coverage from the start.
Its distinguishing feature is provenance: events are assembled from Google News’ “Full Coverage” feature, which groups multi-source reports on the same story. That gives each event a naturally multi-outlet document set, which is the precondition for measuring perspective diversity at all.
Statistics
| Property | Value |
|---|---|
| Events | 103 |
| Paragraphs | 5,296 |
| Avg. sentences per paragraph | 7.1 |
| Avg. words per paragraph | 124.9 |
Articles are segmented into paragraphs capped at 512 tokens, preserving semantic boundaries.
Queries and Annotation
The query for each event is a one-sentence abstractive summary generated by GPT-4o-mini, standing in for a user’s information need. Summaries were regenerated up to five times where the resulting relevance labels had low positive rates.
Paragraphs were labeled for relevance against that summary across three annotation rounds resolved by majority voting, with 98.7% agreement. Paragraphs from outside the event’s Full Coverage set were auto-labeled irrelevant. A human review of 100 sampled items reported a 99% acceptance rate.
The LLM-generated query is the honest caveat here: intent is simulated, not observed. Nothing in the collection comes from a real user’s search.
What It Measures
Systems are scored on relevance (precision, recall, F1) and on three diversity measures introduced with the dataset — Average Pairwise Distance, Positive Cluster Coverage, and Information Density Ratio — at retrieval depths of 5, 10, 20 and 50. The two coverage-denominated metrics are where diversity-aware methods separate from relevance-only ones; average pairwise distance separates them much less cleanly.
Availability
Distributed as LocalNews.zip in the NEWSCOPE repository. The repository carries no license file, so redistribution terms are unstated.
Related Datasets
- DSGlobal — the companion global-news benchmark released alongside it
Related Concepts
- APD — one of the three diversity metrics reported on it
- Diversity Metrics — the broader family
- Search Evaluation
Related Topics
Related Articles
- Uncovering the Bigger Picture - Comprehensive Event Understanding via Diverse News Retrieval — the paper that constructed it