Noindex in the X-Robots-Tag header: the directive view-source does not show

What it looks like

Before

$ curl -sI https://example.com/pricing/
HTTP/2 200
content-type: text/html; charset=UTF-8
x-robots-tag: noindex, nofollow

After

$ curl -sI https://example.com/pricing/
HTTP/2 200
content-type: text/html; charset=UTF-8

Check the headers

curl -sI https://example.com/pricing/ | grep -i x-robots-tag
curl -sI -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://example.com/pricing/ | grep -i x-robots-tag

No output means the header is absent. Run the second form because some rules apply only to crawler user agents. In a browser, the same information is under DevTools → Network → the document request → Response Headers.

Test several kinds of URL: a page, a post, an image, a PDF and the homepage. A header on everything points at the server or CDN. A header on one file type or directory points at a targeted rule.

A header may also name a specific crawler, as in X-Robots-Tag: googlebot: noindex, which affects only that bot.

Headers that are supposed to be there

Do not remove these:

• REST API and admin endpoints. WordPress sends X-Robots-Tag: noindex on /wp-json/ responses and admin-ajax.php.
• XML sitemaps. Hydrogen SEO sends X-Robots-Tag: noindex, follow with its sitemap files and with llms.txt, so the files themselves do not appear as results. The URLs listed inside are unaffected.
• Files you chose to keep out of search, such as PDFs of internal documents, where a header is the only way to apply noindex.

The problem is the header on pages you want to rank.

Where an unwanted header comes from

Apache configuration. Search .htaccess in the site root and parent directories, and the virtual host file:

grep -rn "X-Robots-Tag" .htaccess /etc/apache2/ 2>/dev/null

Typical lines are Header set X-Robots-Tag "noindex, nofollow", sometimes inside a <FilesMatch> block that is broader than intended.

nginx configuration. Look for add_header X-Robots-Tag in the server block and included files.

The hosting platform. Staging environments, temporary domains and "development mode" switches on managed hosts commonly add the header to every response. If the live domain was attached to an environment still flagged as staging, the header follows. Check the host's environment settings and its documentation for the header.

The CDN. Response header rules, "transform rules" or edge functions can add it. Preview and deployment-protection features on some platforms add it automatically to non-production URLs.

A plugin or theme. Security, maintenance-mode and "hide my site" plugins send it with PHP's header() function. Search the code:

grep -rn "X-Robots-Tag" wp-content/plugins wp-content/themes wp-content/mu-plugins

WordPress itself does not add the header to normal front-end pages, so on a regular post it always has an outside source.

Remove it

1. Delete or narrow the rule at the source you identified. For a <FilesMatch> or location block meant for PDFs, tighten the pattern so it matches only those files.
2. On a managed host, switch the environment to production or remove the staging flag, following the host's instructions.
3. Reload the web server if you changed its configuration, and purge the CDN and page cache. Cached responses keep their old headers.
4. Run the curl check again.

If you cannot find the source, bisect: bypass the CDN by requesting the origin directly, then deactivate plugins in halves, then switch theme, checking headers after each step.

Two directives are stricter than one

When a page has both a robots meta tag and an X-Robots-Tag header, Google combines them and applies the most restrictive result. A page whose meta tag says index, follow is still noindexed by a noindex header. That is why fixing the SEO plugin's setting has no effect when the header is the cause; the related case of two tags in the HTML is covered under multiple robots meta tags.

Like the meta tag, the header only works if the URL can be crawled. A URL disallowed in robots.txt never has its headers read.

Get the pages back

After the header is gone:

1. In Search Console, inspect an affected URL and run Test live URL. "Indexing allowed?" should now read yes.
2. Click Request indexing for your most important pages.
3. Open Excluded by 'noindex' tag in the Page indexing report and click Validate fix so Google rechecks the rest.
4. Resubmit the sitemap if the header was site-wide.

Pages return as they are recrawled. Our free SEO audit checks the robots meta tag in the HTML and does not read response headers, so a clean audit result does not rule this issue out; keep the curl check in your launch routine. The meta-tag causes are listed under page set to noindex.