What is an AI crawler?
An AI crawler is a bot that requests web pages on behalf of an AI product. The label covers three jobs: gathering training text, indexing pages for an assistant's search, and fetching one page at a user's request. Each usually has its own bot name.
That distinction decides what a robots.txt rule actually costs or protects.
The three kinds
| Kind | What it does | Examples | Effect of blocking |
|---|---|---|---|
| Training crawlers | Collect content used to train models | GPTBot, ClaudeBot, CCBot, Applebot-Extended | Content is not collected for future training. No effect on appearing in answers that come from search |
| AI search crawlers | Build the index an assistant searches | OAI-SearchBot, PerplexityBot, Claude-SearchBot | Pages cannot be found or cited in that assistant's search answers |
| User-triggered fetchers | Open a page when a person asks about it or pastes its URL | ChatGPT-User, Perplexity-User, Claude-User | The assistant cannot read the page for that user |
Google and Microsoft mostly reuse their search crawlers. AI Overviews and AI Mode draw on pages crawled by Googlebot, and Copilot on Bing's index. Google-Extended is a control token for Gemini, not a separate bot.
Controlling them with robots.txt
Each operator publishes the token its bot obeys. A site that wants to appear in AI search answers and stay out of training sets could use:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
Things to know before relying on it:
- It is voluntary. The major operators state that they honour robots.txt. Unknown scrapers do not, whatever name they send.
- User-triggered fetchers are a grey area. Some operators treat a request made for a user like a browser visit and may not apply robots.txt rules to it. Check each operator's current documentation.
- It is not retroactive. A new rule stops future collection. It does not remove what was gathered before.
- A rule under
User-agent: *applies to them too, unless a group names the bot.
Our robots.txt generator writes these groups, and the guide to blocking AI crawlers in robots.txt explains each choice.
Blocks you did not write
Robots.txt is often not what stops an AI crawler. CDNs, hosts and security plugins increasingly refuse or challenge AI bots by default, at the firewall. Your robots.txt says "allow" and the bot gets a 403. If AI visibility matters to you, look in your CDN's bot settings and in server logs for the status codes these user agents receive.
The reverse also happens: a site means to block training and the rule is misspelled, or sits in a group the bot never reads. Our robots.txt tester shows which rule applies to a given token.
How they read pages
Many AI crawlers download the HTML and do not run JavaScript. Content that only appears after scripts execute may not exist for them. Server-rendered HTML, a clear heading structure and text that makes sense when lifted out of its page are what they handle best. An llms.txt file is a proposed way to point AI systems at your key pages. Adoption is uneven, so treat it as optional guidance.
Deciding what to allow
There is no universally right answer, only a trade:
- Allow AI search crawlers and fetchers if you want to be cited and receive referral visits from assistants. For most businesses that publish to be found, this is the default.
- Block training crawlers if your content is your product, such as paid journalism, licensed data or original research, and you do not want it absorbed without terms. It costs nothing in current search visibility.
- Block everything only if you have a specific legal or commercial reason, and accept that assistants will describe your field without you.
Hydrogen SEO's AEO analysis reads your robots.txt for 13 AI crawlers in these groups and also tests whether a firewall turns the search crawlers away; see answer engine optimization. You can edit the file itself in the robots.txt editor.
Common questions
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot collects training data. ChatGPT's search feature uses OAI-SearchBot, and pages opened at a user's request use ChatGPT-User. Blocking GPTBot alone leaves search answers unaffected.
Do AI crawlers obey robots.txt?
The crawlers run by the major AI companies say they do, each under its published token. Compliance is voluntary, and unidentified scrapers ignore it, so robots.txt is a preference signal and not access control.
How do I see which AI crawlers visit my site?
Search your server access logs for their user agent names, such as GPTBot, ClaudeBot or PerplexityBot. Many CDNs also provide a bot report that lists AI crawlers separately.