Learn
AI discoverability, in depth.
Practical references on llms.txt , GEO , AI crawler access, schema for AI search, and the technical-SEO checks that move AI citation. Written by the team building Crawlmind.
Looking for a specific rule? See the full rule index.
Foundations
AI crawlers
GPTBot: how to allow OpenAI to crawl your site
What GPTBot is, what it's used for, and the exact robots.txt to allow or block it.
Read →
ClaudeBot: how to allow Anthropic to crawl your site
What ClaudeBot is, the difference between ClaudeBot, anthropic-ai, and Claude-Web, and the exact robots.txt to allow or block each.
Read →
The complete list of AI crawlers in 2026
Every AI crawler operating against the public web as of May 2026: user-agent, operator, what it fetches, whether it honors robots.txt, and the recommended policy.
Read →
GEO tactics
Schema markup for AI search: a 2026 reference
Which schema types matter for ChatGPT, Perplexity, Claude, and Gemini: and which Google-era schemas are now noise.
Read →
AI citation readiness: write pages AI engines cite
The structural and editorial signals that determine whether AI engines cite your page in their answers.
Read →
Perplexity SEO: how to be cited by Perplexity
Perplexity is the most citation-friendly AI engine in 2026. Here's how its index works and what to optimize.
Read →