The seven AI crawlers you should know about in 2026

If you've looked at your access logs lately you've probably seen a zoo of new user-agents. Most of the "block all AI" guides treat them as one undifferentiated thing. They're not. Each bot has a specific job, and the right posture depends on what your site is trying to accomplish.

Here's the short field guide to the seven that matter in 2026, in roughly decreasing order of how much they affect your AI visibility.

#1. GPTBot (OpenAI)

What it does: fetches pages for ChatGPT's browsing tool. When a ChatGPT user asks a question and the model decides to look at the live web, GPTBot does the fetch. The cited URLs in the answer come from these crawls.

Right posture for most SaaS: allow. Blocking GPTBot is one of the most common AI visibility self-owns. Many guides conflate it with OpenAI's training crawler, which is a separate flag.

Common misconfiguration: blocking GPTBot to "stop training" while leaving the OpenAI training opt-out unset. The actual effect is to lose ChatGPT citations while NOT actually opting out of training.

#2. PerplexityBot (Perplexity)

What it does: fetches pages for Perplexity's Sonar answer engine. Cites the source URLs in the structured response.

Right posture: allow. Perplexity's citation surface is one of the cleanest and highest-quality AI traffic sources right now. Users who click through from a Perplexity citation are pre-qualified.

Common misconfiguration: same as GPTBot. Blanket block "because AI."

#3. ClaudeBot (Anthropic)

What it does: fetches pages for Claude's web search and citations.

Right posture: allow. Same logic as GPTBot and PerplexityBot. Claude's citation share is smaller than OpenAI's or Perplexity's but growing.

Common misconfiguration: forgetting it exists. ClaudeBot is the newest of the major three and we still see customer robots-txt files that don't list it.

#4. Google-Extended (Google)

What it does: this is the OPT-OUT flag for using your content in Google's Bard / Gemini training and AI Overview generation. It is NOT a fetch crawler. Listing it doesn't change Google's regular indexing.

Right posture: allow unless you have a specific reason. The common misconfiguration here is to disallow Google-Extended thinking it'll keep your content out of Google search. It won't, it only opts you out of AI features. Disallowing it removes you from AI Overviews and Gemini.

#5. Applebot-Extended (Apple)

What it does: same as Google-Extended but for Apple's AI features (Apple Intelligence, Siri's AI improvements).

Right posture: allow unless you have a specific reason. Apple's AI feature set is still emerging but has long-term potential. The opt-out is permanent until you change it.

#6. CCBot (Common Crawl)

What it does: Common Crawl's crawler. Common Crawl is the open-corpus dataset that many AI models (including most open-source ones) train on.

Right posture: judgment call. If your content is paywalled, proprietary research, or otherwise sensitive, blocking CCBot is defensible. If your content is marketing material designed to be seen, allowing CCBot helps you appear in future open-source models which themselves power downstream tools.

Common misconfiguration: blocking CCBot under the impression it's related to ChatGPT or Perplexity. It isn't, directly.

#7. Bytespider (ByteDance / Doubao)

What it does: ByteDance's crawler. Feeds the Doubao AI assistant (ByteDance's ChatGPT competitor, large in the Chinese market) and TikTok's content discovery.

Right posture: block unless you have a specific business interest in the Chinese market or Doubao surfaces. Bytespider is notorious for aggressive crawling, ignoring Crawl-delay, and spiking traffic costs on small sites. For most Western SaaS the strategic value is negligible.

#A reasonable default robots.txt for SaaS

User-agent: GPTBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: CCBot
Allow: /

User-agent: Bytespider
Disallow: /

User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xml

Adjust for your specific case, but this is a defensible starting point that gets you cited by the engines that matter without opening you up to the aggressive ones.

#How to verify the bots are following your rules

Bots are honor-system. Crawlmind logs which user-agents have hit each customer's site over the last 90 days and flags discrepancies (bot identified itself as PerplexityBot but didn't appear to respect the listed rules, for example). For a manual version, parse your access logs and group by user-agent. Anomalies stand out fast.