We respect your privacy.

We use strictly necessary cookies to keep you signed in and to protect against CSRF. With your permission we also use a small amount of first-party analytics to improve the product. We do not sell your data and we do not use third-party advertising trackers. See our cookie policy and privacy policy .

Crawlmind
← All learn topics

Rule site.robots-txt.missing

Add a robots.txt

site.robots-txt.missing is a check in Crawlmind's site audit that grades high-impact issues of this kind. This page explains why the rule matters and the exact fix.

Impact: highEffort: lowFixable: 1-click

Why it matters

Without /robots.txt at the apex domain, every crawler: Google, AI assistants, screen-scrapers: has to guess your intent from your sitemap (if any) and your response codes. A minimal robots.txt costs nothing to publish and gives you explicit allow/disallow for AI bots that would otherwise be ambiguous.

The fix

```
User-agent: *
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

Sitemap: https://example.com/sitemap.xml
```

References