The complete list of AI crawlers in 2026

There are 14 named AI crawlers worth knowing about as of May 2026, operated by 9 vendors (OpenAI, Anthropic, Google, Apple, Perplexity, Meta, Common Crawl, Bytedance, Cohere). This is the canonical reference: for each bot, the user-agent string, the operator, what it fetches the web for, whether it honors robots.txt, and the policy Crawlmind recommends. Use it as a checklist when you write your robots.txt.