What is the Crawl stats report?
The Crawl stats report shows what Googlebot has been doing on your site for the past 90 days: how many requests it made, how much it downloaded, how fast your server answered, and which status codes it received. It lives in Google Search Console under Settings → Crawl stats.
It is the nearest thing to server logs for people who cannot get at their logs, with the advantage that the requests are confirmed as Google's.
Where to find it and who can see it
Open Settings, then Open report next to Crawl stats. It is available for Domain properties and for URL-prefix properties at the root of a host, such as https://example.com/. A property for a subfolder like https://example.com/blog/ does not get one.
The three headline charts
- Total crawl requests. Every request counts, including ones for CSS, JavaScript and images, and repeated requests for the same URL.
- Total download size. Bytes fetched during crawling.
- Average response time. How long your server took to deliver a response, not counting rendering.
Watch for changes, not absolute values. A sudden fall in requests alongside a rise in response time says the server slowed down and Google backed off. A spike in requests after a migration or a new sitemap is normal.
Host status
This panel reports whether Google could reach the site at all, in three checks:
- robots.txt fetch: if the file returns server errors, Google pauses crawling, so a broken robots.txt is serious.
- DNS resolution.
- Server connectivity.
A green tick means no significant problem in the period. A red mark means a recent failure rate high enough to affect crawling, and it deserves immediate attention from your host.
The breakdown tables
| Grouping | Useful for |
|---|---|
| By response | Share of 200, 301, 404, 5xx and others. A growing share of redirects or errors is crawl capacity spent on nothing |
| By file type | How much goes to HTML versus scripts, images and JSON |
| By purpose | Discovery, URLs never crawled before, versus Refresh, recrawls of known pages |
| By Googlebot type | Smartphone, desktop, image, page resource load and others |
Each row opens a list of example URLs with the time of the request, which is how you find out what kind of URL is producing the 404s or soaking up the requests.
Using it to answer real questions
- "Is Google wasting crawl on junk?" Open By response and By purpose, then scan the examples for parameters, internal search results and feeds. This is the practical side of crawl budget.
- "Did my host have an outage?" Host status and the response-time chart show it, with dates.
- "Is my firewall blocking Google?" Look for
403or429rows under By response. - "Why are new pages discovered slowly?" A tiny Discovery share on a site that publishes daily points at weak internal linking or a stale sitemap.
What it does not tell you
The examples are a sample, not a full log. It covers Google only, so other search engines and AI crawlers are invisible here. And it shows crawling, not indexing: a URL fetched a hundred times may still be left out of the index. For complete data, including other bots, you need log file analysis.
On WordPress, two patterns stand out in this report: a large share of requests to /wp-json/ or admin-ajax.php, and comment-reply or feed URLs. Hydrogen SEO's crawl cleanup switches remove several of those addresses from your pages.
Common questions
Where is the Crawl stats report in Search Console?
Under Settings, in the Crawling section. Click "Open report" beside Crawl stats. It is not in the main left-hand menu.
Why can't I see the Crawl stats report?
It is only shown for Domain properties and for URL-prefix properties that cover a whole host. Properties limited to a subfolder do not have it.
What is a good average response time?
There is no official threshold. Lower and stable is better, and a few hundred milliseconds is typical for cached sites. A sustained rise is the signal to investigate, because Google reduces crawling when a server slows down.