Google: Don’t Use 403/404 Error Responses For Rate Limiting Googlebot

Rate limiting Googlebot means slowing down how fast Google crawls a website. Google recommends several ways to limit Googlebot’s crawl rate, chief among them is through the use of the Google Search Console. Rate limiting through search console will slow down the crawl rate for a period of 90 days. Server error response code 429 means “too many requests” and that’s a valid error response. Google recommends consulting their Developer Page about rate limiting Googlebot.