What is retrieval-augmented generation?

The steps

1. Query. The user asks a question. The system often rewrites it into several searches; see query fan-out.
2. Retrieve. Those searches run against an index: a web search index for public assistants, or a private document store for a company chatbot.
3. Select. The retrieved pages are broken into passages, and the passages most relevant to the question are kept.
4. Augment. The chosen passages are placed in the model's prompt alongside the question.
5. Generate. The model writes an answer from that material, usually with links to the pages it drew on.

question → searches → pages → passages → prompt → answer + citations

Why systems are built this way

A language model on its own has two weaknesses that retrieval addresses. Its knowledge stops at its training cut-off, so it knows nothing of last week's price change. And when it lacks a fact it may produce a fluent guess. Supplying current, relevant text at answer time reduces both problems and makes the answer checkable, because the sources can be shown.

It does not make answers infallible. The model can still misread a passage or combine two sources wrongly, which is why answers about your brand are worth monitoring.

What it means for a website

Each step is a filter your content has to pass.

• Be retrievable. The page must be crawlable by the engine behind the assistant and present in its index. A page blocked in robots.txt, refused by a firewall or set to noindex never reaches step 2. For Google's AI features that means Googlebot access and indexing. For other assistants it means their search crawlers; see AI crawler.
• Rank for the sub-queries. Retrieval is search. Pages that rank for the narrower questions inside a broad one get retrieved.
• Survive being cut into passages. Selection works on excerpts, so a paragraph that only makes sense with the three before it is a poor candidate. Paragraphs that name their subject and state a complete fact travel well.
• Be specific. Exact figures, steps, definitions and comparisons give the model something to use. Vague copy offers nothing worth quoting.
• Be quotable under your snippet settings. In Google, a page with nosnippet is not used as direct input for AI Overviews and AI Mode.

Passage-friendly writing, concretely

Compare two versions of the same fact.

It usually takes around that long, though as mentioned above this depends on the factors we covered.

Googlebot follows up to 10 redirect hops. If a chain is longer than that, Search Console reports a redirect error for the URL.

The first is meaningless out of context. The second can be lifted as it stands. Practical habits that produce the second kind:

• open each section with a sentence that answers the heading;
• repeat the noun instead of relying on "it" and "this" across paragraph breaks;
• put definitions, limits and steps in plain sentences or simple lists and tables;
• keep one topic per section under a descriptive heading.

None of this is new. It is the same clarity that earns featured snippets, applied because a machine now does the excerpting.

RAG versus training data

The two are separate routes into an AI answer. Training shapes what a model knows without looking anything up, which changes only when a new model is trained. Retrieval decides what it reads at the moment of the question, which changes as fast as an index updates. Blocking a training crawler affects the first route and leaves the second open, which is why the choices are made per bot.

To act on this, our pages on how to get cited by AI and optimizing for AI Overviews turn the principles into steps. Hydrogen SEO's AEO analysis checks the retrievability side for a URL: crawler access, indexing and snippet eligibility; see answer engine optimization.