Blocked due to access forbidden (403): when your firewall turns Googlebot away

Rule out the harmless cases first

Open the example URLs in the report. Plenty of 403s are correct:

• /wp-admin/ paths and wp-login.php variants behind a login-protection rule.
• Directory URLs such as /wp-content/uploads/2026/ where directory listing is disabled.
• xmlrpc.php when you have blocked it deliberately.
• Files in plugin folders that are denied direct access.

None of these should be indexed. If the examples are posts, pages, products or images you want in search, carry on.

Identify the layer that said no

The response headers usually name it.

curl -sI -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://example.com/pricing/

• A server: header naming your CDN, plus a challenge or "attention required" body, points at the CDN's bot management or firewall rules.
• A 403 page branded by a security plugin points at WordPress.
• A plain "Forbidden" page from Apache or nginx points at server configuration or file permissions.

This test has a limit worth knowing. It imitates Googlebot's user agent from your own IP address. Many firewalls block exactly that combination as a fake Googlebot, so a 403 here does not prove real Googlebot is blocked, and a 200 does not prove it is allowed if the block is IP-based. The reliable test is URL Inspection → Test live URL, which fetches from Google's network.

Common causes and what to change

1. Bot protection set too aggressively. "Bot fight" modes, JavaScript challenges and "block all automated traffic" presets catch crawlers. Use the provider's verified-bots allow option so known search engine crawlers skip challenges.
2. Country or ASN blocks. Googlebot crawls mostly from United States addresses. A rule that blocks the US, or blocks data-centre networks wholesale, blocks Google.
3. Rate limiting. A crawl burst trips a requests-per-minute rule and the rest of the burst gets 403 (or 429). Raise the threshold for verified crawlers.
4. User-agent blocklists in .htaccess copied from an old "bad bots" list that includes a pattern matching Google.
5. File permissions. Directories should normally be 755 and files 644. A folder set to 700 or owned by the wrong user returns 403 for everything inside it.
6. Hotlink protection on images that only allows your own referrer. Googlebot sends no referrer; make sure empty referrers are allowed, or images disappear from Google Images.

Allowing Googlebot without opening the door to impostors

Never allowlist on the user-agent string alone, since anyone can send it. Google documents two verification methods: a reverse DNS lookup on the requesting IP that resolves to a googlebot.com or google.com hostname and forward-resolves back to the same IP, or matching the IP against the address ranges Google publishes as JSON. Most CDNs and firewalls maintain a "verified bots" category built on those lists; use it instead of hand-written rules where you can.

Check the result

Run the live test again; Page fetch should read "Successful". In Settings → Crawl stats, the share of responses under Other client error (4xx) should fall over the next days. Then use Validate fix on the 403 row in the Page indexing report. If you audit the page with our free SEO audit and the audit itself fails to load the page, that is a hint the firewall is rejecting automated visitors in general, not only ours.