We respect your privacy.

We use strictly necessary cookies to keep you signed in and to protect against CSRF. With your permission we also use a small amount of first-party analytics to improve the product. We do not sell your data and we do not use third-party advertising trackers. See our cookie policy and privacy policy .

Crawlmind
← All learn topics

Rule site.robots-txt.blocks-all

robots.txt blocks every crawler from the entire site

site.robots-txt.blocks-all is a check in Crawlmind's site audit that grades high-impact issues of this kind. This page explains why the rule matters and the exact fix.

Impact: highEffort: lowFixable: 1-click

Why it matters

A `Disallow: /` under `User-agent: *` tells every crawler that the entire site is off-limits. This is almost always a mistake: usually a staging-config file that got copied into production, or a deploy that forgot to switch off the "block all" rule used during the build. The site is invisible to Google, Bing, ChatGPT, Perplexity, Claude, and every other crawler that respects robots-txt (which is essentially all of them).

The fix

Check what your live site is serving:

```sh
curl -s https://example.com/robots.txt
```

A reasonable default robots.txt for a public site:

```
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml
```

If you genuinely want to block some paths, be specific:

```
User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /internal/
Allow: /

Sitemap: https://example.com/sitemap.xml
```

If this is a deliberate staging-environment block (you don't want Google indexing your staging URL), use HTTP basic auth or a CIDR firewall instead. `Disallow: /` is unreliable as a security mechanism: crawlers respect it, but the URLs are still indexable if someone links to them externally.

References