We respect your privacy.

We use strictly necessary cookies to keep you signed in and to protect against CSRF. With your permission we also use a small amount of first-party analytics to improve the product. We do not sell your data and we do not use third-party advertising trackers. See our cookie policy and privacy policy .

Crawlmind
← All learn topics

Rule site.robots-txt.blocks-ai-bots

Allow AI crawlers in robots.txt

site.robots-txt.blocks-ai-bots is a check in Crawlmind's site audit that grades high-impact issues of this kind. This page explains why the rule matters and the exact fix.

Impact: highEffort: lowFixable: 1-click

Why it matters

Your robots.txt currently blocks (or partially blocks) one or more AI bots. Blocking is a valid choice if you don't want your content cited by AI assistants: but it's usually accidental: an over-broad `User-agent: *` Disallow inherits to specific bots, or a CDN's default rules block AI agents without you knowing. The fix is to grant explicit Allow rules per bot.

The fix

```
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /
```

Per RFC 9309, the longest-matching user-agent group wins, so an explicit per-bot Allow overrides a `User-agent: *` Disallow.

References