Hydrogen SEO
Log inStart free
Technical

What is a user agent?

A user agent is the software making an HTTP request, and in practice the term means the string that software sends in the User-Agent request header to identify itself. Browsers send one, and so does every crawler, which is how servers, logs and robots.txt rules tell Googlebot from Chrome or from an AI bot.

In SEO the term shows up in two places with slightly different meanings: the full string in your server logs, and the short product token you write after User-agent: in robots.txt.

Updated By the Hydrogen SEO team

The full string versus the robots.txt token

This is what Google's smartphone crawler sends:

code
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

W.X.Y.Z stands for the current Chrome version, which changes as Google updates its renderer. In robots.txt you never write that whole line. You write the token, and matching ignores case:

code
User-agent: Googlebot
Disallow: /cart/

One crawler can answer to several tokens. Google's image crawler obeys a Googlebot-Image group if one exists and falls back to Googlebot, then to *.

Why the string cannot be trusted alone

Anyone can send any user agent. Scrapers routinely call themselves Googlebot to get past rate limits, so a log line is a claim, not proof. To confirm a real search crawler:

  1. Run a reverse DNS lookup on the requesting IP. Genuine Googlebot resolves to a host under googlebot.com or google.com, and genuine Bingbot to search.msn.com.
  2. Run a forward lookup on that hostname and check that it returns the same IP.

Google and Bing also publish their crawler IP ranges, which is the easier route for firewall rules.

Where it affects a site

  • Robots rules. Groups in robots.txt and crawler-specific meta robots tags such as <meta name="googlebot" content="noindex"> are keyed on the token.
  • Firewalls and CDNs. Bot protection that blocks by user agent is a common reason a page works in a browser and fails for a crawler. Blocking by string alone also blocks nothing that lies.
  • Serving different content. Showing crawlers one page and people another, based on user agent, is cloaking when the content differs in substance.
  • Log analysis. Grouping requests by user agent is the first step of any log file analysis.

Testing as a crawler

You can request a page with a crawler's string to see what it is served:

code
curl -I -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://example.com/

A 403 here while a normal request gets 200 points to a firewall rule. It is only a rough test, because your request does not come from Google's IP addresses. Our robots.txt tester shows which rule group a given token falls under.

On WordPress the question usually comes up when a security plugin or host firewall is involved. If Search Console reports pages it cannot fetch while they load normally for you, compare the two requests above before changing anything in robots.txt.

Common questions

What is Googlebot's user agent?

The robots.txt token is Googlebot. The full header value for the smartphone crawler starts with Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) and ends with (compatible; Googlebot/2.1; +http://www.google.com/bot.html), with the current Chrome version in between.

Is User-agent: * the same as all bots?

It is the fallback group. A crawler uses it only when no group names that crawler specifically, so a bot with its own group ignores the rules under the asterisk.

Can I block a bot by its user agent?

You can ask it to stay away in robots.txt, which well-behaved crawlers respect, or refuse it at the firewall. Neither stops a bot that sends a false string.

Related terms