Google Adds New Documentation For Mystery Crawler

Although Google added official documentation for this crawler the information provided seems to encourage more clarification. The new crawler documentation is for the Google-Safety user agent. Google-Safety CrawlerThe documentation in the Special-case Crawlers Google-Safety crawler is one that’s used by Google’s processes for finding malware. The new documentation for the Google-Safety Crawler:“The Google-Safety user agent handles abuse-specific crawling, such as malware discovery for publicly posted links on Google properties. Overview of Google crawlers and fetchers (user agents) – Special-case crawlers