OpenAI has published information about GPTBot, its new web crawler. OpenAI uses it to crawl the web, consume knowledge for its AI features (e.g., ChatGPT) and provide AI-generated answers to questions (or prompts). To disallow GPTBot to access your site you can add GPTBot to your site’s robots.txt:User-agent: GPTBot Disallow: /To allow GPTBot to access only parts of your site, you can add the GPTBot token to your site’s robots.txt like this:User-agent: GPTBot Allow: /directory-1/ Disallow: /directory-2/GPTBot documentation. OpenAI also published the IP ranges that GPTBot uses. You can disallow GPTBot from crawling your site if you don’t want OpenAI using your content in any way.