robots.txt unreachable: when a failing robots.txt pauses crawling of the whole site

How Google treats each response

Google's documentation on robots.txt describes the behaviour by status:

• 2xx: the file is read and its rules applied.
• 3xx: redirects are followed for a limited number of hops. After that, Google treats the file as not found.
• 4xx, except 429: treated as if no robots.txt exists. Everything may be crawled.
• 5xx and 429: treated as a temporary failure. Google first behaves as though the whole site is disallowed and keeps retrying. If the error persists, it falls back on the last copy it cached. If the file is still failing after about 30 days, Google decides based on whether the rest of the site is reachable.
• DNS and network errors, timeouts: handled like server errors.

The practical reading: an outage of an hour changes nothing, since Google generally caches the file for up to 24 hours. A robots.txt that fails for days suppresses crawling.

Where to see it

• Settings → robots.txt in Search Console lists each robots.txt file Google found for the property, when it was last fetched and its status: fetched, not fetched because it was not found, or not fetched for another reason. The last of these is the one to worry about.
• URL Inspection → Test live URL on any page fails with a message that robots.txt is unreachable.
• Settings → Crawl stats → Host status has a robots.txt fetch check showing the failure rate over the past 90 days.

From your side:

curl -s -o /dev/null -w "%{http_code} %{time_total}s\n" https://example.com/robots.txt

Run it a few times, and once with -A "Googlebot", to catch intermittent and user-agent-specific failures.

Why WordPress robots.txt fails when pages do not

On most WordPress sites there is no robots.txt file on disk. The request goes through index.php, WordPress boots, and the output is generated. Cached pages keep loading when PHP is struggling, because the cache serves them without PHP. robots.txt is often not cached, so it is the first URL to return 500, 502 or 503 when PHP workers or the database are exhausted.

Other causes:

1. A plugin error on the robots.txt request only. Code hooked to the robots_txt filter or the do_robotstxt action that throws a fatal error. Check wp-content/debug.log.
2. Firewall or CDN rules challenging bots on .txt requests, or rate limits that robots.txt requests trip because crawlers fetch it often.
3. A redirect chain. http:// to https:// to www to a trailing-slash variant. Serve robots.txt directly on every host and protocol you use.
4. A subdomain without its own file. Rules are per host. shop.example.com needs its own robots.txt response; if that host errors for the path, the report shows it separately.
5. Maintenance mode returning 503 for every URL, robots.txt included, for longer than a brief update.

Fixes

• Repair the underlying error as you would any server error (5xx): read the PHP log, fix or remove the failing code, add capacity or caching.
• Exempt /robots.txt from bot challenges and rate limits at the firewall. It is meant to be fetched by machines.
• Let your page cache or CDN cache the robots.txt response for a short time, so it survives a brief PHP outage.
• If reliability is still a concern, a physical robots.txt file in the site root is served by the web server without PHP. The trade-off is that plugin editors can no longer manage it: Hydrogen SEO locks its robots.txt editor when a physical file exists, and you must maintain the sitemap line by hand.

Confirm recovery

The curl check should return 200 quickly and consistently. In Settings → robots.txt, open the file's menu and request a recrawl; the status should change to fetched with a current timestamp. Then run a live URL test on a normal page to confirm crawling is permitted again. Our free SEO audit requests robots.txt as part of every run and tells you when no file is found, and you can paste the live file into the robots.txt tester to check its rules once it is reachable.