Search engines like Google use website crawlers to read and understand webpages. To help you navigate the world of website crawlers, we’ll walk you through what crawlers are, how they work, and how you can safely use the right tools to your advantage. Publicly available crawlers are slightly different from search engine crawlers like Googlebot or Bingbot (the unique web crawlers that Google and Bing use). But crawlers access and index website content. Site AuditSite Audit uses crawlers to access your website and analyze its overall health.