What is the X-Robots-Tag?
The X-Robots-Tag is an HTTP response header that carries the same directives as the meta robots tag, such as noindex, nofollow, and nosnippet. Because it travels in the response headers instead of the HTML, it works for files that have no <head>: PDFs, images, videos, text files, and feeds.
It is also useful when you want to apply a rule to many URLs at once from server configuration, for example noindexing every PDF in a downloads folder, without editing the files themselves.
How it works
When a crawler requests a URL, the server replies with status, headers, and body. If the headers include X-Robots-Tag: noindex, Google drops that URL from the index after crawling it, exactly as if the page had a meta noindex. Multiple directives are comma-separated, and you can send the header more than once. You can scope a header to one crawler by prefixing its name, as in X-Robots-Tag: googlebot: nofollow.
If both a header and a meta tag are present, Google combines them and applies the most restrictive instruction. As with the meta tag, the URL must not be blocked in robots.txt, or the crawler never sees the header.
Example
Checking a response with curl:
$ curl -I https://example.com/files/price-list.pdf
HTTP/2 200
content-type: application/pdf
x-robots-tag: noindex, nofollow
Setting it for all PDFs on Apache, in .htaccess:
<FilesMatch "\.pdf$">
Header set X-Robots-Tag "noindex"
</FilesMatch>
On Nginx, the equivalent is an add_header X-Robots-Tag "noindex"; line inside a location block matching .pdf files.
In WordPress
WordPress media files are served directly by the web server, not by PHP, so plugins cannot add headers to them without server rules. The header is set in .htaccess, Nginx config, or a CDN rule. Plugins can send it on responses WordPress itself generates. Hydrogen SEO, for example, serves the /llms.txt file with X-Robots-Tag: noindex, follow so the file is readable by AI systems but does not show up as a search result; see the llms.txt editor.
Common mistakes
- A server-wide noindex header left from staging. It is invisible in page source, so it is easy to miss. Check headers with
curl -Ior the URL Inspection tool. - CDN or caching layers stripping or adding headers so what the origin sends is not what Google receives.
- Blocking the file in robots.txt and expecting the header to work.
Common questions
How do I check if a page has an X-Robots-Tag?
Run curl -I on the URL, open the Network tab in browser developer tools and inspect the response headers, or use URL Inspection in Search Console, which reports whether indexing is blocked by a header.
Is X-Robots-Tag better than the meta robots tag?
Neither is better. Use the meta tag for normal HTML pages and the header for non-HTML files or rules applied at the server level.