What "partial" AI-bot access in your robots.txt actually means

"Partial" AI-bot access is a robots.txt verdict that means a crawler is allowed to fetch your site but is disallowed from at least one path, and for most well-configured SaaS sites it is the correct and desirable state, not a warning. If you have ever run an AI-crawler checker and seen GPTBot, ClaudeBot, PerplexityBot, and a dozen others all flagged "partial" in amber, the instinct is to assume something is misconfigured. Usually nothing is.

This post explains how the three verdicts (allowed, partial, blocked) are derived from a robots.txt file, why "partial" is the default outcome for a healthy site, and the one case where partial genuinely signals a problem.

#How the verdict is computed

Per RFC 9309, the standard that governs robots.txt, each crawler obeys the most specific User-agent group that matches its name, falling back to the User-agent: * group if it has no group of its own. Almost no site writes a dedicated block for every AI bot, so the vast majority of crawlers fall through to the * group.

Given the rules in the group that applies to a bot, the verdict is straightforward:

• Blocked: a blanket Disallow: / with no Allow: exceptions. The bot is shut out entirely.
• Allowed: no non-empty Disallow: rules at all. The bot can crawl everything.
• Partial: there is at least one Disallow: path that is not a blanket /. The bot can crawl the site, but some paths are off limits.

That last branch is the one that surprises people. The moment your applicable group contains a single Disallow: /admin line, every bot subject to that group is, by this definition, "partial." It is not a softer kind of blocked. It means "crawls the site, minus a few paths."

#Why a healthy SaaS site shows "partial" everywhere

Here is a robots.txt that is doing exactly the right thing:

User-agent: *
Allow: /
Disallow: /me/
Disallow: /orgs/
Disallow: /admin/
Disallow: /auth/
Disallow: /api/

Public marketing pages, docs, and the blog are open. Private dashboards, the auth flow, and the API are closed. No AI bot has its own group, so they all fall through to *, and * has five non-blanket Disallow: paths. Result: every tracked bot reports "partial."

That is precisely what you want. You do not want GPTBot indexing a logged-in dashboard at /orgs/, and you certainly do not want it crawling /api/. So 100 percent of the bots showing "partial" here is the healthy outcome, even though an amber chip makes it look like a caution. The signal that matters is not the verdict word, it is which paths are disallowed.

#When partial is actually a problem

Partial is only a red flag when the disallowed paths include content you want cited. Two patterns to watch for:

#1. A content directory is blocked

If the disallow list contains something like Disallow: /blog/ or Disallow: /docs/, you are hiding the exact pages AI engines should be reading. This is the failure mode worth catching, and it is why a good checker shows the actual paths, not just the word "partial." A blocked /admin/ is fine; a blocked /guides/ is lost visibility.

#2. A wildcard is broader than intended

A rule like Disallow: /*? (meant to block tracking-parameter URLs) can accidentally match canonical pages that legitimately use query strings, such as paginated archives or filtered category pages. Wildcards are powerful and easy to over-apply.

#How to read your own report

When you see a wall of "partial," do this:

• Look at the disallowed paths, not the verdict. If they are all private or utility routes (/admin, /cart, /api, /search, /wp-admin/), you are done. That is a correct configuration.
• Confirm your content paths are absent from the list. Your blog, docs, product, and pricing pages should never appear in a Disallow: line.
• Only act if a content path is blocked. Then remove or narrow that specific rule. Do not try to turn every "partial" into "allowed," because that would mean exposing your private routes.

#The summary

"Partial" AI-bot access means a crawler can reach your site but is disallowed from one or more paths, and it is the normal, correct state for any site that keeps private routes out of the index. Every bot showing partial is not a problem in itself. The only thing worth checking is the path list: private and utility routes being blocked is exactly right, and a content directory being blocked is the real issue. Judge the paths, not the color of the chip.