We respect your privacy.

We use strictly necessary cookies to keep you signed in and to protect against CSRF. With your permission we also use a small amount of first-party analytics to improve the product. We do not sell your data and we do not use third-party advertising trackers. See our cookie policy and privacy policy .

Crawlmind
Home/Glossary/ClaudeBot

AI crawlers

ClaudeBot

Updated 2026-05-17

ClaudeBot is Anthropic's web crawler, used for training Claude and for live retrieval in Claude's web-search feature. It replaced the older anthropic-ai user-agent, which is still respected for backwards compatibility.

Allow ClaudeBot

User-agent: ClaudeBot
Allow: /

Block ClaudeBot

User-agent: ClaudeBot
Disallow: /

Most sites should also include the legacy directive:

User-agent: anthropic-ai
Disallow: /

Related

Where this comes up

See how your site scores on ClaudeBot + every other AI-discoverability signal.

Free audit