NVIDIA Research: RAG with Long Context LLMs
Source: https://www.llamaindex.ai/blog/nvidia-research-rag-with-long-context-llms-7d94d40090c4 Publisher: LlamaIndex · Published: 22 October 2023 · Author: Ravi Theja
Summary
Does a long context window make retrieval unnecessary? NVIDIA Research’s answer, covered here, is no — retrieval helps both short- and long-context models.
Setup
- Models: Nemo GPT-43B (extended to 16K), LLaMA2-70B (extended to 16K and 32K)
- Benchmarks: QMSum, Qasper, NarrativeQA, QuALITY, MuSiQue, HotpotQA, MultiFieldQA-en
Findings
- Retrieval “significantly enhances the performance of both shorter 4K context language models and their longer 16K/32K context counterparts.”
- LLaMA2-70B-32K with retrieval surpassed GPT-3.5-turbo-16K and matched davinci-003.
- Optimal retrieval is 5–10 chunks. Retrieving 20 degraded performance, attributed to the “lost in the middle” phenomenon.
- A 4K model with retrieval matches much longer models while inferring faster.
Why It Matters
Point 3 is the durable one: more context is not monotonically better, so ordering and budgeting remain ranking problems even at 32K. See Long-Context RAG.
Related Concepts
- Long-Context RAG — primary topic
- RAG · Needle in a Haystack Test · Reranking · Prompt Compression