AI bots · GPTBot
GPTBot
GPTBot is OpenAI's crawler. It powers ChatGPT browsing, the "Atlas" agent, and the Read tool. It is not used to train new models. That's a separate, opt-out crawler (see Google-Extended-style flags).
Allow GPTBot
Paste this into /robots.txt. GPTBot will respect the directive on its next refresh (typically within 24 hours).
User-agent: GPTBot
Allow: /Block GPTBot from one path
You can scope the deny narrowly. Most teams keep /private/ and /admin/ closed.
User-agent: GPTBot
Allow: /
Disallow: /private/
Disallow: /admin/Blocking won't block training
GPTBot is the browsing crawler. Blocking it does not stop your content from being used to train models. That's a separate decision and a different mechanism.
Edge cases
- GPTBot-Image follows the same directives, but you can also reference it explicitly.
- Cloudflare's "Block AI bots" toggle adds a UA-matching layer on top of robots.txt; it'll override your Allow.
- If you serve different robots.txt per geography, GPTBot crawls from US-East. Confirm what it actually sees with our AI crawler access checker.