Noindex in robots.txt: a rule Google does not support, and what to use instead

What it looks like

Before

User-agent: *
Disallow: /wp-admin/
Noindex: /thank-you/
Noindex: /tag/

After

User-agent: *
Disallow: /wp-admin/

# and on the pages themselves:
<meta name="robots" content="noindex, follow">

Check whether you are affected

Open your live /robots.txt and search for noindex. Then look at the URLs those lines name:

• Inspect one in Search Console. If it reports "URL is on Google", the rule is not protecting it.
• Search site:example.com/tag/ (or the relevant path) for a rough view of what is indexed.

Google's robots.txt report under Settings → robots.txt shows the file as Google last fetched it, with any problems it found. An unsupported line is skipped; it does not invalidate the rest of the file.

The three working replacements

1. A robots meta tag. For HTML pages. In WordPress with Hydrogen SEO:

• One page: open the SEO Settings sheet (Ctrl+E or Cmd+E in the editor), open Robots Meta, switch on No Index.
• A whole type, such as tags: Hydrogen SEO → Robots, then the Noindex switch in the Taxonomies or Post Types table.
• Search results and the 404 page are already noindex, follow without any setting.

The options are described in robots meta settings, and our guide to noindexing a page in WordPress walks through the single-page case.

2. An X-Robots-Tag header. For PDFs, images and other files that have no <head>. On Apache:

<FilesMatch "\.pdf$">
  Header set X-Robots-Tag "noindex"
</FilesMatch>

On nginx:

location ~* \.pdf$ { add_header X-Robots-Tag "noindex"; }

3. Removing or protecting the content. A deleted page returning 404 or 410 drops out. Content behind a login is never indexed.

The mistake that follows

Having replaced Noindex: with a meta tag, many people also add Disallow: for the same path "to be safe". That undoes the fix. A disallowed URL is not fetched, so the meta tag is never seen, and the URL can remain indexed from links, shown without a description. The status for that outcome is indexed, though blocked by robots.txt.

The rule of thumb: Disallow controls crawling, noindex controls indexing, and noindex only works on URLs that are not disallowed.

If a path was both disallowed and listed under Noindex:, remove both lines, add the meta tag, and wait for Google to recrawl. Once the URLs have dropped out you may reinstate the Disallow if saving crawl requests matters, accepting that Google will no longer see the tag.

Other lines Google skips

While you are editing, look for other directives that Google's parser does not act on:

• Crawl-delay: is ignored by Google. Some other crawlers respect it.
• Host: was specific to another search engine and has no effect in Google.
• Rules placed before any User-agent: line belong to no group and are ignored.

Google's parser supports User-agent, Allow, Disallow and Sitemap. Our robots.txt generator writes a WordPress-safe file using only supported lines.

Verify the replacement

1. View source on an affected page and confirm <meta name="robots" content="noindex, follow">, or run curl -sI on a file and confirm the X-Robots-Tag header.
2. Paste the URL and your robots.txt into the robots.txt tester; the result must be "allowed" for Googlebot.
3. In URL Inspection, run a live test. It should report that crawling is allowed and indexing is not, because of noindex.
4. Over the following weeks the URLs move to "Excluded by 'noindex' tag" in the Page indexing report. For anything urgent, a temporary request in the Removals tool hides the URL while you wait.