Google Answers A Crawl Budget Issue Question

February 28, 2024

Someone on Reddit posted a question about their “crawl budget” issue and asked if a large number of 301 redirects to 410 error responses were causing Googlebot to exhaust their crawl budget. Matt answered a question about a Google crawl budget by first explaining that there was no crawl budget in the way that SEOs conceive of it:“The first thing is that there isn’t really such thing as an indexation cap. There is also not a hard limit on our crawl.”In 2017 Google published a crawl budget explainer that brought together numerous crawling-related facts that together resemble what the SEO community was calling a crawl budget. This new explanation is more precise than the vague catch-all phrase “crawl budget” ever was (Google crawl budget document summarized here by Search Engine Journal). Reddit Question About Crawl RateThe person on Reddit wanted to know if the perceived low value pages they were creating was influencing Google’s crawl budget.

The source of this news is from Search Engine Journal