robots.txt syntax not understood: the lines Google skips and why

Where Google shows parsing problems

Open Settings → robots.txt in Search Console and click the file. The report displays the version Google last fetched and flags lines with problems. This matters for two reasons: it is the copy Google is using, which may be up to a day older than what your server sends now, and it shows the file as fetched by Google, which reveals cases where Google receives something different from your browser.

For rule-by-rule testing, paste the file into our robots.txt tester, enter a URL and a crawler name, and it reports allowed or blocked together with the deciding line.

Mistakes that make a line meaningless

Rules with no group. Allow and Disallow lines only count inside a group that starts with User-agent:. Lines placed above the first User-agent: apply to nobody.

Disallow: /private/        # ignored: no user-agent yet
User-agent: *
Disallow: /tmp/

A missing colon or a misspelt field. Disallow /tmp/ and Dissalow: /tmp/ are skipped. Field names are not case-sensitive, so disallow: is fine.

Full URLs in rules. Disallow: https://example.com/tmp/ does not match anything useful. Paths start with /. Only the Sitemap: line takes a full URL, and it must be absolute.

Regular expressions. The format supports two special characters: * for any sequence of characters and $ for the end of the URL. Disallow: /page-[0-9]+/ is treated as literal text.

Several paths on one line. Disallow: /a/ /b/ is one odd path, not two rules. Write one path per line.

Everything on one line. A file whose line breaks were stripped is read as a single broken rule. Hydrogen SEO versions before 0.0.27 could save the file this way; 0.0.27 repairs it once on update. If you ever pasted the file through a field that removes newlines, look at the live output.

Mistakes that parse but do the wrong thing

• Path case. Values are case-sensitive. Disallow: /Private/ does not block /private/.
• Prefix matching. Disallow: /blog blocks /blog/, /blog-news/ and /blogroll.html. Add the trailing slash when you mean the directory.
• An empty Disallow. Disallow: with no value allows everything. Disallow: / blocks everything. One character separates the two.
• Split groups for a specific bot. A crawler uses the most specific group that names it and ignores the rest. If you add User-agent: Googlebot with a single rule, Googlebot no longer follows your User-agent: * rules; repeat them in its group.
• Trailing comments with # inside a path. Everything after # is a comment, so a rule for a URL fragment is cut short.

File-level problems

• Wrong location. The file must be at the root of the host: https://example.com/robots.txt. A file in a subdirectory is never read. Each subdomain and each protocol has its own.
• Not plain text. If the URL returns an HTML page, for instance a themed 404 with a 200 status or a login page, the "rules" are HTML tags. The parser finds no valid lines.
• Encoding. The file should be UTF-8 plain text. Google ignores a byte-order mark at the start, but characters outside UTF-8 may be dropped, which can corrupt a rule.
• Size. Google enforces a limit of 500 KiB and ignores content beyond it. Files only approach that when rules are generated per URL, which is a sign that noindex or a pattern would serve better.

Rebuild it cleanly

For most WordPress sites the right file is short enough to retype:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemap.xml

Our robots.txt generator produces this baseline with optional AI crawler groups. In Hydrogen SEO, paste the result into Hydrogen SEO → Robots.txt Editor, save, and click View Robots.txt; the editor keeps your line breaks and blank lines as typed, and its reset option restores the default with the sitemap line. Details are in the robots.txt editor guide and in our walkthrough on editing robots.txt in WordPress.

Confirm the file is read as intended

Fetch the live file with curl -s https://example.com/robots.txt and compare it with what you saved. Test three or four real URLs in the tester: one that must be crawlable, one that must be blocked, a CSS file, and the sitemap. Then request a recrawl of the file in Search Console's robots.txt report and check that no lines are flagged in the fetched version.