Crawlers, search engines and the sleaze of generative AI companies

I used to work at Google many years ago, mostly in web search. These search engines voluntarily obey the Robots Exclusion Protocol, taking a website’s implementation of the Protocol as a directive, an absolute command, not just a mere hint. We also respect robots.txt too, so if you don’t want Brave Search crawling your site, it won’t. Visiting the Brave Search API homepage shows several price tiers, including some called “Data for AI”. Google’s Publisher Controls initiativeThere may be a new type of web crawler coming soon, one specifically for generative AI.