Agents quit where users would retry
Crawlmind Engineering··6 min read
An agentic visit is a live, in-session fetch: an AI assistant loads your pages while a person waits, under a fixed step budget, to answer a question or finish a task. It is not a crawl, and the difference matters more than the label suggests. A crawler that meets a slow response comes back tomorrow. An agent that meets a slow response fails the task now, and reports back that your site did not work.
Most GEO work is aimed at the retrieval layer: what the engine indexed, what it chunked, what it cited. The agentic layer sits underneath and runs on different rules. It is also the layer where a condition your monitoring would file as a blip turns into a total loss.
#A ten second delay is not a minor degradation
WAREX, a reliability harness published on arXiv in October 2025, takes three existing web-agent benchmarks and injects the faults real websites actually produce: client-side network delays, server errors returned as 408, 429, 502 and 503, and slow JavaScript endpoints that leave images and buttons broken on an otherwise loaded page.
The injected network fault is a 10 second delay. Not an outage, not a 500. A delay.
On WebVoyager, task success fell from 42.0% with no fault injected to 2.0% under the network delay. On WebArena the same fault took the baseline of 12.4% down to 3.7%, and on REAL it took 17.0% down to 4.5%.
Server errors hurt less but still hurt a lot. Under injected 4xx and 5xx responses, WebArena dropped to 5.3%, REAL to 7.1%, and WebVoyager to 30.0%, against those same no-fault baselines.
The absolute numbers are low because these benchmarks are hard, and they were run with an older model backbone. Read the ratios instead. A transient fault removed roughly 70% to 95% of remaining agent capability depending on the benchmark. Nothing about the page content changed. Nothing about the task changed. The agent simply ran out of room.
#Your visitors retry without noticing. The agent spends a step.
Humans absorb a startling amount of site instability. A slow page gets a reload. A 503 gets a back button and a second attempt thirty seconds later. A broken button gets tried twice, then routed around. None of this shows up in your funnel as a failure, because the visitor repaired it for you.
Compare the crawler contract, which is documented and forgiving. Google states that 5xx and 429 server errors prompt its crawlers to temporarily slow down, that already indexed URLs are preserved in the index and only eventually dropped, and that crawl rate increases again once the server returns 2xx. A crawler has a long memory and unlimited patience, so a bad hour costs you almost nothing.
An agent has neither. It has a step budget, a user waiting, and no concept of coming back later. Every retry it does spend competes with the steps it needs to finish the job. This is why the fault classes that barely register in your SEO reporting are the ones that dominate agentic outcomes.
#Reading is close to solved. Doing is not.
Web Bench covers 2,454 open-sourced tasks across 452 websites in 17 categories, and splits them by what the agent has to do rather than by which site it is on. Read tasks are the majority at 1,580, or 64.4% of the set, with create, update, delete and file manipulation making up the rest.
The split in results is wide. Most agents clear 75% on read tasks, while the best fully automated agent completed 46.6% of non-read tasks. The strongest overall result reported was 66.0% of all tasks.
The benchmark authors also note that a meaningful share of failures happen before the agent gets to act at all, and they name the causes: proxy blocking of datacenter IP ranges, CAPTCHA walls, and login and authentication checks that flag the session as a bot. Those are infrastructure decisions, not content decisions, and they are usually owned by a team that has never been asked about AI visibility.
If your commercial value depends on an agent reading a spec, a price, or a comparison, the odds are decent. If it depends on an agent submitting a form, configuring a quote, or completing a signup, the odds are close to a coin flip on a good site and worse on a defended one.
#Interstitials read as content
WAREX also ran a deceptive popup against the REAL benchmark's 112 tasks. The agents clicked it at 97.3% with GPT-4o, 86.6% with Qwen2.5-VL, and 98.2% with GPT-OSS.
The security reading of that result is its own topic. The GEO reading is simpler and more immediately useful: agents do not have a mental category for "thing the site put in front of the thing I want." Your consent banner, newsletter modal, app-install prompt, geographic redirect gate and age check are all, to an agent, page content that appears to require interaction. Each one costs steps. Some of them are terminal.
Every interstitial you stack is a tax on a budget you cannot see and did not set.
#The blocking layer sits on top of all of this
The hostile-response rate is climbing independently. Analysis of Cloudflare Radar's crawler response-status series found that 35.8% of all crawler requests returned a 4xx client error in Q2 2026, up from 10.2% a year earlier.
Much of that is deliberate, and blocking training crawlers is a legitimate business decision. The problem is that intent does not survive the boundary. Agent fetches frequently arrive from datacenter IPs with non-browser fingerprints, which is exactly the signature a rule written against scrapers is designed to catch. A policy aimed at bulk harvesting will quietly catch the assistant acting on behalf of your prospect, and you will never see the session it killed.
#What to check
Five things, in rough order of payoff:
- Measure the tail, not the median, on agent-visible paths. Your p50 is fine. The question is what your p95 and p99 look like on the pages an assistant would load to answer a question about your product.
- Make degradation partial instead of total. If your answer lives in the initial HTML response, a slow analytics or widget endpoint costs the agent nothing. If the answer waits on client-side hydration, that same slow endpoint is a dead page.
- Treat 429 as a product decision. Rate limiting is reasonable. Rate limiting the class of traffic that is increasingly how buyers reach you is a tradeoff someone should be making on purpose.
- Audit the interstitial stack on your highest-intent pages. Count the things that must be dismissed before the content is reachable. That count is a direct subtraction from the agent's step budget.
- Separate the policy question from the accident question. Deciding to block training crawlers is policy. Discovering that the same rule blocks user-triggered agent fetches is an accident, and it is worth auditing separately.
#None of this shows up in your reporting
A failed agent session leaves almost nothing behind. No bounce, because there was no session in your analytics. No error your support team hears about, because the user asked an assistant and got a shrug. No missing citation to investigate, because the page may well still be cited elsewhere.
You find it by looking at what you serve rather than at what you publish: status-code distribution by user agent, tail latency on the paths that matter, and the interstitials sitting between a fetch and an answer. That is unglamorous infrastructure work. Right now it is also the cheapest AI visibility available, because almost nobody is doing it.
Related field notes
September 24, 2026 · 6 min
Google pays for grounding, not for links
Google's AI contribution pilot pays when a page shapes an answer, not when it is linked afterward. That rule says a citation count measures the wrong thing.
September 23, 2026 · 5 min
A browser agent is not a crawler
Agentic browsing runs inside the user's own session, so robots.txt, bot allowlists and crawler analytics all miss it entirely.
September 22, 2026 · 4 min
An MCP endpoint is not a discovery channel
NLWeb and MCP make your site answerable by agents that already found you. Nothing on the open web is hunting for a /mcp route yet.
Share or discuss
New posts, no spam. Roughly monthly. Unsubscribe with one click.