robots.txt generator
Pick the rules you want and copy a ready robots.txt. The defaults suit most WordPress sites: keep crawlers out of /wp-admin/ except the AJAX endpoint front-end features need, skip internal search result pages, and point to your sitemap. AI training crawlers are off by default, and you opt out of them one at a time.
- robots.txt stops crawling, not indexing. Use a noindex tag for pages that must stay out of results.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/
Disallow: /cart/
Disallow: /checkout/
Sitemap: https://example.com/sitemap.xmlWhy these defaults
/wp-admin/with anadmin-ajax.phpallow. Admin screens are useless to searchers, but themes and plugins calladmin-ajax.phpfrom public pages, so blocking it can break how Google renders them.- Internal search (
/?s=). Search result pages are near-infinite and thin. Google recommends keeping them out of the index; blocking the crawl stops the waste before it starts. - The
Sitemap:line lets any crawler find your sitemap without Search Console.
What the file deliberately does not do: block /wp-content/, /wp-includes/, CSS or JavaScript. That advice circulated for years and now hurts, because Google renders pages like a browser.
The AI crawler choices
Each AI company publishes separate user-agents for training and for answering. The options here only cover training crawlers such as GPTBot, ClaudeBot and Google-Extended. Blocking them does not stop your pages being fetched when someone asks an assistant about them, and blocking Google-Extended has no effect on Google Search or AI Overviews. Whether to opt out is a business decision about licensing your content, not an SEO fix. The AI crawler guide explains the trade-offs.
Installing it on WordPress
WordPress serves a virtual robots.txt unless a real file exists in the site root. Either upload the output as robots.txt, or paste it into your SEO plugin's robots.txt editor so it stays editable from wp-admin. Then confirm the live file at yoursite.com/robots.txt and test individual URLs with the robots.txt tester.
Common questions
Does robots.txt hide a page from Google?
No. It only stops crawling. A blocked URL can still be indexed from links, without its content. Use a noindex tag, and leave the page crawlable, to keep it out of results.
Should I block /wp-content/uploads/?
No. That is where your images live, and blocking it removes them from image search and can hurt how pages render.
How long until Google sees a new robots.txt?
Google usually caches the file for up to about a day, so changes take effect on its next fetch rather than instantly.