robots.txt is blocking CSS and JavaScript: why Google renders your page wrong

What it looks like

Before

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/themes/

After

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemap.xml

What Google loses when resources are blocked

• Layout. Without the theme stylesheet, Google's rendered view is an unstyled column. It cannot tell that the page adapts to phone screens, and Google indexes the mobile version of pages.
• Content. Anything inserted by a blocked script is missing from the rendered HTML: sliders, tabs, product data loaded by JavaScript, some page-builder widgets.
• Images. A blanket block on /wp-content/ also blocks /wp-content/uploads/, which removes your images from Google Images.

The page still gets indexed from its raw HTML, which is why this goes unnoticed for years.

Confirm it in Search Console

1. Inspect an important URL and click Test live URL.
2. Click View tested page, then the More info tab.
3. Open Page resources. Blocked files are listed as not loaded, with the reason "Googlebot blocked by robots.txt".
4. Open the Screenshot tab. An unstyled or partly empty page confirms the effect.

Only resources on your own host count here. Third-party scripts blocked by another site's robots.txt, such as some ad or analytics endpoints, are outside your control and rarely matter for rendering.

To find the rule responsible, paste your robots.txt and one blocked file URL into our robots.txt tester. It names the matching line.

Fix the file

The WordPress default is the right baseline, and it is all most sites need:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Delete rules that block /wp-includes/, /wp-content/, /wp-content/plugins/, /wp-content/themes/, /wp-content/cache/ and patterns like /*.js$ or /*.css$. Minified and combined assets from caching plugins live under /wp-content/cache/, so that folder in particular must stay crawlable. Keep the Allow line for admin-ajax.php: some themes and plugins load front-end content through it.

In Hydrogen SEO, make the change under Hydrogen SEO → Robots.txt Editor and use View Robots.txt to read the live result. If the editor is locked, a physical robots.txt file exists in the site root and must be edited or removed there. See the robots.txt editor documentation.

If you must keep a directory blocked

Occasionally a folder holds files that should not be fetched. Add more specific Allow rules for assets. Google applies the most specific matching rule, measured by path length, so the longer Allow patterns win:

User-agent: *
Disallow: /wp-content/plugins/
Allow: /wp-content/plugins/*.css
Allow: /wp-content/plugins/*.js
Allow: /wp-content/plugins/*.woff2

Check each exception with a real file URL in the tester, because wildcard rules are easy to get subtly wrong. Remember too that robots.txt is not a security measure. Blocking /wp-content/plugins/ hides nothing from an attacker, who does not consult robots.txt.

A group for Googlebot overrides the general group

If the file contains a User-agent: Googlebot group, Google follows that group only and ignores User-agent: * entirely. A file whose general group is clean can still block resources through an old Googlebot-specific group further down. Read the whole file.

Verify the render

Run the live test again after saving. Page resources should no longer list your own CSS and JavaScript as blocked, and the screenshot should match what a visitor sees on a phone. Google caches robots.txt for up to 24 hours, so scheduled crawling may take a day to catch up. In Settings → robots.txt, you can request a recrawl of the file to shorten that.