How to fix "Blocked by robots.txt" in Search Console
"Blocked by robots.txt" means a rule in your robots.txt file tells Googlebot not to crawl this URL, so Google has not read the page and does not index it. To fix it, find the Disallow line that matches the URL and remove or narrow it, then click Validate fix. If you meant to block the URL, there is nothing to fix.
On WordPress, the most common surprise is the Discourage search engines setting, which noindexes every page and can make the served robots.txt block the entire site. Staging settings carried over to a live site cause the same thing.
Step by step
1. Decide whether the block is intended
Open Indexing → Pages → Blocked by robots.txt and read the examples. Blocking is normal for /wp-admin/, internal search (/?s=), cart and checkout URLs, and filter parameters. If the list contains only URLs like those, leave it. If it contains posts, pages or products you want in Google, carry on.
2. Check Discourage search engines first
Go to Settings → Reading and look at Search engine visibility. If Discourage search engines from indexing this site is checked, every page gets noindex, nofollow, and the robots.txt served can switch to User-agent: * / Disallow: / for the whole site. Hydrogen SEO's robots.txt editor locks itself with a warning while the box is checked. Uncheck it and save. This single checkbox is the cause behind many "my whole site disappeared" reports after a launch or a migration from staging.
3. Read the live robots.txt
Open yoursite.com/robots.txt and find the rule that matches the blocked URL. Rules match from the start of the path, and * matches any sequence of characters:
| Rule | Blocks |
|---|---|
Disallow: / | The entire site |
Disallow: /blog | /blog/, /blog-news/ and /blogroll/ |
Disallow: /*? | Every URL with a query string |
Disallow: /wp-content/ | Theme CSS, JS and images, which can break rendering |
Test the exact URL against Googlebot in the robots.txt tester; it shows which rule decided the result.
4. Find where the rule comes from
Your robots.txt can come from several places, and editing the wrong one does nothing:
- A physical
robots.txtfile in your web root always wins. Check with SFTP or your host's file manager. - The Hydrogen SEO robots.txt editor, when there is no physical file.
- Your host or CDN, which some platforms inject for staging domains.
The Hydrogen SEO editor shows a warning and locks itself when a physical file exists or when the site is set to discourage search engines, which tells you where to look.
5. Remove or narrow the rule
Open Hydrogen SEO → Tools → Robots.txt Editor, delete or narrow the offending line, and click Save Changes. For example, replace Disallow: /blog with Disallow: /blog/drafts/ if you only meant to block one folder. Then click View Robots.txt. In the current plugin version, saving can strip the line breaks between rules; if the live file shows rules on one line, fix and save again, because crawlers need one directive per line.
6. Test with URL Inspection and validate
Inspect a previously blocked URL in Search Console and click Test live URL. Crawl allowed? should now say Yes. Google caches robots.txt, usually for up to a day, so a live test right after the change may still show the old result. When it passes, click Validate fix on the report.
What the status means
robots.txt controls crawling, not indexing. This status means Google obeyed a rule and did not fetch the page. A related status, "Indexed, though blocked by robots.txt", means Google indexed the URL anyway from links elsewhere, without reading it, and may show it with no description.
That difference matters when you want a page out of Google. Blocking it in robots.txt stops Google from ever seeing a noindex tag on it. To remove a page, leave it crawlable and noindex it; see noindexing a page in WordPress.
When to leave it alone
Leave blocks on admin paths, cart and checkout, internal search and faceted filters. These save crawling for pages that matter. Remove blocks on anything you want in results, and on /wp-content/ or /wp-includes/ assets that pages need to render. Google must fetch CSS and JavaScript to see a page the way visitors do.
Common questions
Does Blocked by robots.txt mean my page is penalized?
No. Google is following your own instruction not to crawl the URL. Remove the rule if you want the page crawled.
How long after changing robots.txt will Google recrawl?
Google usually refreshes its copy of robots.txt within a day, then recrawls the URLs over the following days or weeks.
Should I block pages in robots.txt to remove them from Google?
No. Use a noindex tag and keep the page crawlable, so Google can see the tag.