No robots.txt found: why WordPress may not be serving one and how to fix it

What it looks like in the HTML

Before

GET /robots.txt
HTTP/1.1 404 Not Found

After

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemap.xml

Why it matters

robots.txt controls crawling, not indexing. A good one is short: it keeps bots out of admin areas and crawl traps, and points them to your sitemap. Without one, crawlers simply fetch whatever they find, which on most small sites is fine.

The status code matters more than the file's existence. A 404 means "no rules". A 5xx server error is treated differently: Google may stop crawling the site until it can fetch robots.txt again, because it cannot tell what is allowed. A 403 from a firewall is also worth fixing, because it may mean other bots are being blocked from more than just this file.

Why a WordPress site may not serve one

WordPress generates a virtual robots.txt on request when no physical file exists, so on a standard install /robots.txt should always load. When it does not:

• Server rules intercept the request. Some Nginx configurations serve /robots.txt only as a static file and never pass the request to WordPress, so without a physical file they return a 404.
• Plain permalinks. The virtual file relies on WordPress's rewrite rules. Sites set to Plain under Settings → Permalinks may not serve it.
• A firewall or CDN blocks the request. Security plugins and bot-protection services sometimes block unknown crawlers, returning a 403 or a challenge page. This audit identifies itself as HydrogenSEO-Bot, so a block aimed at unfamiliar bots can cause this result even if Googlebot gets through.
• WordPress is installed in a subdirectory, so its robots.txt lives at /blog/robots.txt instead of the domain root, where crawlers look.

How to fix it in WordPress

1. Open https://yoursite.com/robots.txt in a browser. Note what happens: a 404, an error, a challenge page, or a file. curl -I https://yoursite.com/robots.txt shows the status code.
2. Use Hydrogen SEO's robots.txt editor. Under Hydrogen SEO → Robots.txt Editor, you can view and edit the file from wp-admin. While it is empty, Hydrogen SEO serves its built-in default, and it adds a Sitemap: line pointing at your sitemap automatically. After saving, click View Robots.txt to read the live file; in the current version, saving can remove line breaks between rules, so always check the result.
3. If the editor is locked, it tells you why: either a physical robots.txt exists in the site root (the server serves that file directly; edit or delete it), or Discourage search engines is on under Settings → Reading.
4. Fix server rules. On Nginx, make sure the /robots.txt location falls back to WordPress (try_files $uri $uri/ /index.php?$args;), or upload a physical file. Your host can help.
5. Allow crawlers through your firewall. Check security plugin or CDN logs for blocked requests to /robots.txt.
6. For subdirectory installs, place a robots.txt at the domain root.

Keep the file short. A stray Disallow: / blocks your whole site.

How to check the fix

Load /robots.txt and confirm a 200 status and plain-text rules, each on its own line. Use Search Console's robots.txt report (under Settings) to see the version Google last fetched and any errors. Rerun the free SEO audit, which also checks whether your sitemap is listed in robots.txt.