Retrieval-augmented generation: How chatbots look things up before they answer

Retrieval-augmented generation, usually shortened to RAG, is a way to make a language model look things up before it answers. The system first searches a collection of documents, picks the most relevant passages, and hands them to the model along with the question. The model then writes its answer using that material.

The idea sits behind many chatbots that cite sources, answer questions about company files, or report on breaking news.

A student taking an open-book exam

A model without retrieval is like a student taking a closed-book exam. Everything it knows was absorbed during training, and it stops learning on the day training ends. Ask about last week’s news and it either admits ignorance or, worse, produces a confident guess.

RAG turns that into an open-book exam. Before answering, the system flips through a shelf of reference material, finds the right pages, and lets the model read them. The model still does the writing and reasoning. The shelf supplies the facts.

That shelf can be anything: a help center, a legal archive, or a feed of news articles. If a publisher licenses its wire stories to an AI company, retrieval is one plausible way those stories could reach an answer. Whether any given deal works that way is a separate question, as the next section on our news story explains.

Search first, then write: The three steps

Most RAG systems follow the same loop.

1. Prepare the documents. Long texts are cut into chunks of a few paragraphs. Each chunk is converted into an embedding, a list of numbers that captures its meaning, so that chunks about similar topics end up with similar numbers. These are stored in a searchable index.
2. Retrieve. When a user asks a question, the question is converted into numbers the same way. The system finds the chunks whose numbers are closest, and returns the top handful.
3. Generate. The retrieved chunks are pasted into the prompt, next to the question. The model reads them and writes an answer, ideally pointing to which passage supports which claim.

Many systems also run a classic keyword search alongside the meaning-based one, because exact terms such as error codes or names are easy for keyword matching and sometimes missed by embeddings. A final reranking step can reorder the candidates so the best ones come first.

Where the idea came from

The name comes from a 2020 paper, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, by Patrick Lewis, Ethan Perez and colleagues, including Douwe Kiela, Sebastian Riedel and Wen-tau Yih. It was published on arXiv in May 2020 and accepted at the NeurIPS conference that year.

The authors described two kinds of memory. The model’s own weights are parametric memory. A dense index of Wikipedia, searched by a neural retriever, is non-parametric memory. Combining them, they reported state-of-the-art results on three open-domain question-answering tasks, and text that was more specific and factual than a model using its weights alone. They also argued that the approach offered provenance, meaning a trail to the source, and an easier way to update knowledge.

That second point is the practical appeal. Retraining a large model to add last week’s facts is slow and costly. Adding documents to an index is not.

Who uses it today

RAG is now a standard building block for enterprise assistants, customer-support bots and search products. AI labs also build tooling around it. Anthropic’s documentation describes a citations feature that grounds Claude’s responses in documents the developer supplies, returning the passages that support each claim so users can verify answers.

Anthropic has also published work on improving the retrieval step itself. In a September 2024 post it described Contextual Retrieval, which prepends a short, chunk-specific explanation to each chunk before indexing, because ordinary chunking can strip away context. Using top-20 retrieval, the company reported that the share of failed retrievals fell from 5.7% to 3.7% with contextual embeddings, to 2.9% when combined with keyword matching, and to 1.9% with reranking. Those are Anthropic’s own measurements on its own test sets.

Our news story fits this picture only loosely. Adweek reportedly found that Anthropic struck a licensing deal with The Associated Press, but the public text does not say whether the content is for training, live answers, or both. A licensed wire service could in principle give a chatbot an authoritative source for breaking news, and retrieval is the usual way to do that. That is an inference, not something either company has said.

Where RAG goes wrong

RAG reduces the guessing, but it does not remove it.

• Retrieval can miss. If the right passage is not among the chunks returned, the model never sees it, and may fall back on its memory or on a weaker match.
• The model can misread what it retrieved. A correct source does not guarantee a correct summary. Citations help users check, but someone still has to check.
• Sources can be wrong or hostile. A retrieved web page may be inaccurate, or may contain instructions aimed at the model. The system is only as trustworthy as the shelf it reads from.
• Chunking loses context. A paragraph that says “the company raised its forecast” means little without knowing which company. This is the problem Contextual Retrieval targets.
• It costs time and money. Each answer adds a search step and a longer prompt.

For small collections there is a simpler route. Anthropic’s post suggests that for a knowledge base under about 200,000 tokens, the whole text can go straight into the prompt, with prompt caching making that faster and cheaper. RAG earns its place when the collection is too large for that.

What to watch next

Three things are worth following. First, whether content licensing deals specify retrieval, training, or both, since the technical use shapes what a publisher is actually selling. Second, whether chatbots show visible attribution when they answer from licensed material. Third, how much retrieval quality improves as labs refine chunking, reranking and citation features.

For readers, the practical takeaway is simple: when a chatbot shows its sources, the quality of the answer depends on two things, how well it found the material and how faithfully it read it.