LongFact
A prompt set for benchmarking the factuality of long, multi-paragraph LLM answers, introduced with SAFE in Long-form Factuality in Large Language Models by Google DeepMind (Google). Repo: github.com/google-deepmind/long-form-factuality.
Contents
- 2,280 fact-seeking prompts across 38 manually selected topics, grouped into four supercategories: STEM, social sciences, humanities and other.
- Two tasks, LongFact-Concepts and LongFact-Objects, depending on whether a prompt asks about a concept or an object. Each task has 1,140 prompts, 30 per topic.
- The prompts were generated by GPT-4 and ask for long answers containing many specific facts.
Why It Exists
Earlier factuality benchmarks such as TruthfulQA, HaluEval, FreshQA, HalluQA and FELM mostly test one short fact per question. FActScore needs long-form answers but covers a single domain, people biographies. LongFact covers long-form answers across many topics.
How It Is Scored
LongFact has no gold answers. Each response is scored with SAFE, which checks every atomic claim against Google Search, and the results are combined with F1@K. See Atomic Claim Verification. In the paper’s benchmark of 13 models on 250 LongFact-Objects prompts, GPT-4-Turbo, Gemini-Ultra and PaLM-2-L-IT-RLHF scored highest.