What is crawl-delay?
Crawl-delay is a line in robots.txt that asks a crawler to pause for a number of seconds between requests to your server. It was never part of the robots.txt standard, so support depends on the crawler: Bing's crawler follows it, and Google's does not read it at all.
Site owners reach for it when bots are eating server resources. It can help with some crawlers, but it is a blunt tool, and for Google you need a different approach.
Syntax and what the number means
The directive sits inside a user-agent group, with a value in seconds:
User-agent: bingbot
Crawl-delay: 5
User-agent: *
Crawl-delay: 10
A value of 5 asks for at most one request every five seconds, which works out to roughly 17,000 URLs a day at best. That arithmetic matters: a delay of 10 on a site with 200,000 URLs means a crawler would need more than three weeks to fetch each page once, so new and updated pages get noticed late.
Crawlers pick the most specific group that names them, so a bingbot group overrides the * group for Bing.
Which crawlers respect it
- Googlebot: no. Google does not support the rule and skips the line. Google sets its own pace from how quickly and reliably your server answers.
- Bingbot: yes. Bing reads the value, and Bing Webmaster Tools also has a crawl control setting for choosing quieter hours.
- Other bots: it varies. Several search and SEO crawlers honour it. Scrapers and badly behaved bots ignore robots.txt altogether, so a delay line does nothing against them.
Slowing Google down without crawl-delay
Google slows down by itself when a server struggles: slow responses and server errors make it back off. If you need an emergency brake, Google's documented method is to answer crawler requests with 500, 503 or 429 for a short period. Keep that to a day or two, because URLs that return errors for longer can be dropped from the index. The old crawl rate limiter setting in Search Console has been retired.
For a lasting fix, cut the number of URLs worth crawling instead of the speed: block endless filter and search URLs, fix redirect chains, and cache pages so each request costs less. Our note on crawl budget covers where crawl activity usually leaks.
On a WordPress site
The virtual robots.txt that WordPress generates contains no crawl-delay line. You can add one by editing the file, for example with the robots.txt editor in Hydrogen SEO, and check the result with our robots.txt tester. Before you do, look at your access logs to see which bot is really causing load. It is often not a search engine, in which case a firewall or rate-limit rule at the server is the tool that works.
Common questions
Does Google respect crawl-delay?
No. Googlebot ignores the crawl-delay line. Google adjusts its crawl rate automatically from your server's response times and errors.
What is a good crawl-delay value?
If you need one at all, keep it low, between 1 and 5 seconds, and apply it only to the crawler that is causing load. High values stop crawlers from reaching new and updated pages in reasonable time.
Can crawl-delay hurt SEO?
It can in Bing and other engines that obey it. A long delay limits how many pages they fetch per day, so large sites end up with stale or missing pages in those indexes.