Google Reveals Two New Web Crawlers

May 18, 2024

Google revealed details of two new crawlers that are optimized for scraping image and video content for “research and development” purposes. GoogleOther CrawlersThe two new crawlers are versions of Google’s GoogleOther crawler that was launched in April 2023. Google lists user agent tokens for both of the new crawlers which can be used in a robots.txt for blocking the new crawlers. GoogleOther-VideoUser agent tokens:GoogleOther-VideoGoogleOtherFull user agent string:GoogleOther-Video/1.0Newly Updated GoogleOther User Agent StringsGoogle also updated the GoogleOther user agent strings for the regular GoogleOther crawler. Read the updated Google crawler documentationGoogleOther-ImageGoogleOther-VideoFeatured Image by Shutterstock/ColorMaker

The source of this news is from Search Engine Journal