To judge crawl efficiency, check Search Console's Crawl Stats report: whether total crawling has plateaued over time, and whether the average response time shows the server is fast enough or is limiting Googlebot.
Thing · Report
Crawl stats
The Search Console report on Google's crawling of a site: requests, response codes, file types, purpose and host status.
- Claims
- 13
- In Google’s docs
- 0
- Said at the event
- 11
- Not in docs
- 0
- Kit items
- 5
Glossary · Crawl Stats report
A Search Console report, under Settings, that shows how much Google crawls a site, the responses and errors it received, the file types fetched by its crawlers and the split between discovery of new URLs and refresh of known ones.
Google’s documentation 0
Documented in
Said at the event 11
Slide and stage claims that name it, the ones Google’s documentation does not cover first.
Label
Grade
Consistent with docs 5
Google said it knew of no plans to add crawl stats to the Search Console API, because that would be a completely new part of the API.
A Google panelist said adding a report that is a variation of the Performance report to the Search Console API would be easier than adding crawl stats, because the Performance report is already in the API.
John Mueller said the Crawl Stats report's split of crawling by purpose, discovery versus refresh, gives a rough view of whether discovery crawling is in a reasonable range, though measuring publish-to-first-crawl time is more accurate.
John Mueller · Day 1 · Q&A
John Mueller said that if the Crawl Stats report shows about half of a site's crawling going to discovering new pages, that is a lot of discovery crawling, which suggests that crawling of new pages is not the problem.
John Mueller · Day 1 · Q&A
Confirmed by docs 4
To debug crawl issues, Google points site owners to the Crawl Stats report in Search Console, found in the Settings page under Crawl stats.
Gary Illyes · Day 1 · How crawling works
Search Console's Crawl Stats report shows how much Google crawls from a site, the errors Google received, and the content types fetched by the different crawlers.
Gary Illyes · Day 1 · How crawling works
Google said soft 404s, like some other problem categories, are looked up in Search Console's page indexing report, while other crawl issues are debugged in the Crawl Stats report.
Gary Illyes · Day 1 · How crawling errors affect Search
To check whether a site has a crawl budget problem, use Search Console's crawl report (Crawl Stats), which breaks crawl requests down by response and by file type and shows crawl problems Google finds.
Nothing to verify 2
StageD1-C341
Gary Illyes called the Crawl Stats report the most valuable tool for debugging crawl errors, together with server logs, which he called the real thing for those who know how to read them.
Gary Illyes · Day 1 · How crawling works
StageD1-C461
An audience member asked whether and when crawl stats will be added to the Search Console API.
From the audience · Day 1 · Q&A
Press and analysis 2
AnalysisD1-C382
The stage rule of thumb (under a few thousand URLs crawl budget is unlikely to be a problem) and Google's crawl budget guide (D1-C097: sites with over a million pages that change about weekly, or over 10,000 pages that change daily) leave a middle range where a site should check Search Console's Crawl Stats report before blaming crawl budget for slow indexing.
Ibrahim Anjro · Day 1 · How Google thinks about crawl budget
AnalysisD3-C609
With a typical refresh of about 30 days and deep or soft-404-like pages recrawled only every few months, link important deep pages from strong hub pages and fix pages that look like errors; then check recrawl intervals by site depth in server logs or the Crawl Stats report.
Ibrahim Anjro · Day 3 · How long does it take to..?
Built on these claims 5
Kit items about Crawl stats: their own words name it, or several of the claims they rest on do.
Developer requirements 4
Monitor Crawl stats and server logs for verified Googlebot traffic
Rests on 13 claims, 4 of them naming Crawl stats; its own words name Crawl stats
Measure the time from publishing a URL to Googlebot's first crawl of it
Rests on 7 claims, 2 of them naming Crawl stats; its own words name Crawl stats
Watch the Page indexing report and treat its not-indexed reasons as a work list
Rests on 10 claims, 1 of them naming Crawl stats; its own words name Crawl stats
Verify every production host in Search Console before launch
Rests on 1 claim, 0 of them naming Crawl stats; its own words name Crawl stats
Also inglossary term Crawl Stats report
Connected things 9
Relations
- Part of Search Console structure, no claim needed
- Measures Crawl budget
1 claim, 1 documented
To check whether a site has a crawl budget problem, use Search Console's crawl report (Crawl Stats), which breaks crawl requests down by response and by file type and shows crawl problems Google finds.
Most often named with it
Things named in the same claim, with the number of claims they share.
- Search Console 6
- Search Console API 3
- Crawl budget 2
- Googlebot 1
- Page indexing report 1
- Performance report 1
- Soft 404 1