How to edit robots.txt in WordPress

Step by step

1. Look at the file you serve now

Open https://yoursite.com/robots.txt. A stock WordPress install serves something like this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

followed by a Sitemap: line. If you see Disallow: / on its own, the whole site is blocked, usually because Settings → Reading → Discourage search engines is ticked. Untick it on a live site.

2. Check for a physical robots.txt file

Look in the site root, next to wp-config.php, over SFTP or your host's file manager. A real robots.txt file there overrides anything WordPress generates. Hydrogen SEO's editor is locked with a warning in this case, because edits would have no effect. Either edit the file directly, or delete or rename it so the editor takes over.

3. Edit the rules in Hydrogen SEO

1. Open Hydrogen SEO → Robots.txt Editor (under Tools).
2. Edit the content. While the editor is empty, the site serves Hydrogen SEO's built-in default.
3. Click Save Changes. The new rules are live immediately.
4. Click View Robots.txt and read the live file.

In the current version, saving can remove the line breaks between rules, which leaves them on one line where crawlers may misread them. A fix is being worked on, so always read the live file after saving. You do not need to add a Sitemap: line; Hydrogen SEO adds one pointing at its sitemap automatically.

4. Write only the rules you need

Each group starts with a User-agent line and lists Disallow or Allow paths. Rules match from the start of the path, and Google supports * as a wildcard and $ for the end of a URL. A typical addition:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /internal-reports/
Disallow: /*?add-to-cart=

To address one crawler, give it its own group, such as User-agent: GPTBot. Blocking AI crawlers has its own guide: block AI crawlers in robots.txt.

5. Do not block CSS, JavaScript or uploads

Google renders pages to understand them. Rules like Disallow: /wp-content/ or Disallow: /wp-includes/ stop it loading your stylesheets, scripts and images, and pages can then look broken to Google. Old tutorials recommended these lines. Remove them if you have them.

6. Test the paths that matter

Click Validate in the editor to open an external robots.txt checker, or paste your file into the robots.txt tester and try your homepage, a post, a product and an image URL. In Search Console, Settings → robots.txt shows the version Google last fetched and any parsing problems.

7. Or change it with code

Without an SEO plugin, append rules to WordPress's virtual file with the robots_txt filter in a child theme or snippets plugin:

add_filter( 'robots_txt', function ( $output, $public ) {
    $output .= "Disallow: /internal-reports/\n";
    return $output;
}, 10, 2 );

This only works while no physical file exists. With Hydrogen SEO active, use its editor or its own hydrogen_seo_robots_txt filter so there is one source of truth.

What robots.txt will not do

• Remove pages from Google. A blocked URL can still be indexed from links elsewhere, shown without a description. To keep a page out, use noindex and leave it crawlable; see how to noindex a page.
• Hide private content. The file is public, and badly behaved bots ignore it. Private content needs a login.
• Fix duplicate content. Blocking a duplicate stops Google seeing its canonical tag. Use canonicals or redirects.

Common mistakes

• A leftover Disallow: / from a staging site, copied to production during a migration.
• A physical file uploaded years ago that nobody remembers, silently overriding every change made in the dashboard.
• Blocking a page and noindexing it at the same time. Search Console then reports it as blocked by robots.txt or "indexed, though blocked".
• Blocking URL parameters used by important pages, such as pagination or product filters people search for.

If something goes wrong, the editor's reset option returns the file to Hydrogen SEO's default, with the sitemap line intact.