GPTBot: OpenAI releases new web crawler

OpenAI has published information about GPTBot, its new web crawler. OpenAI uses it to crawl the web, consume knowledge for its AI features (e.g., ChatGPT) and provide AI-generated answers to questions (or prompts). To disallow GPTBot to access your site you can add GPTBot to your site’s robots.txt:User-agent: GPTBot Disallow: /To allow GPTBot to access only parts of your site, you can add the GPTBot token to your site’s robots.txt like this:User-agent: GPTBot Allow: /directory-1/ Disallow: /directory-2/GPTBot documentation. OpenAI also published the IP ranges that GPTBot uses. You can disallow GPTBot from crawling your site if you don’t want OpenAI using your content in any way.