What is llms.txt?
llms.txt is a Markdown-formatted text file served at the root of a website (/llms.txt) that gives AI systems a short, curated description of the site and links to its most useful pages. It was proposed in September 2024 by Jeremy Howard of Answer.AI as a way to hand language models clean context instead of making them parse full HTML pages with navigation, scripts, and boilerplate.
Its status matters: llms.txt is a proposal, not a standard. Adoption is real among documentation sites and developer tools, but no major AI provider has publicly committed to fetching and honoring it as part of its standard pipeline. It is a cheap bet with no downside, not a control mechanism.
How it works
The proposed format:
- An H1 with the site or project name (the only required part).
- A blockquote with a short summary.
- Optional free-form notes.
- H2 sections containing lists of links, each with a short description.
- An optional section titled "Optional" for secondary links that can be skipped when context is limited.
Some sites also publish llms-full.txt, containing the full text of their documentation in one file. That is a related convention, not part of the core proposal.
llms.txt is not a permissions file. It does not allow or block anything. Crawler access for AI bots is still controlled through robots.txt user-agent rules, and whether a given bot reads llms.txt at all is up to its operator.
Example
# Kiln Coffee
> Kiln Coffee is a specialty roaster in Portland. This site covers our
> coffees, brewing guides, and wholesale program.
## Guides
- [Pour-over guide](https://example.com/pour-over-guide/): Ratio, grind, and a 3-minute pour schedule.
- [Cold brew guide](https://example.com/cold-brew/): Steep times and concentrate ratios.
## Company
- [Wholesale](https://example.com/wholesale/): Pricing tiers and minimum orders for cafes.
## Optional
- [Roasting blog](https://example.com/blog/): Notes on each season's coffees.
In WordPress
WordPress does not generate llms.txt. Hydrogen SEO includes an llms.txt editor under Tools with default content you can edit and a reset option. The file is served at /llms.txt as plain text with X-Robots-Tag: noindex, follow, so it does not appear as a search result. The complete llms.txt guide covers what to include.
Common mistakes
- Treating it as a ranking or citation lever with guaranteed effects.
- Listing every URL, turning it into a second sitemap.
- Letting it go stale with links to renamed or deleted pages.
Common questions
Does Google use llms.txt?
Google has not said it uses llms.txt for Search or AI Overviews, and its public guidance does not mention it. Treat it as a forward-looking extra, not a Google requirement.
Is llms.txt the same as robots.txt?
No. robots.txt tells crawlers which paths they may fetch. llms.txt describes your content and points to key pages; it grants or blocks nothing.