AI crawlers fetch your site from the US

Locale-adaptive delivery is the practice of changing what a single URL returns based on the visitor's IP country or Accept-Language header, and it decides which version of your site AI engines read, because the major AI crawlers arrive from a small set of US datacenter addresses and send no language preference at all.

Most international sites were built for a search engine that solved this problem a decade ago. The AI crawlers did not inherit the fix.

#The crawlers live in a few US datacenters

Every serious AI company now publishes the IP ranges its crawlers fetch from, so you can check this rather than guess. OpenAI lists separate ranges for GPTBot, OAI-SearchBot and ChatGPT-User at gptbot.json, searchbot.json and chatgpt-user.json. Perplexity publishes perplexitybot.json and perplexity-user.json. Anthropic publishes a single list covering ClaudeBot, Claude-User and Claude-SearchBot at claude.com/crawling/bots.json.

Those files are meant for allowlisting, but they also answer a question nobody thought to ask: where is the crawler standing when it asks for your page?

A February 2026 analysis geolocated 484 published subnets, roughly 19,116 addresses, across the major search and AI crawlers (EdgeComet). The split is stark. GPTBot ran from 20 subnets across three US cities, Phoenix, Atlanta and Des Moines, with what the analysis describes as zero international presence (EdgeComet). OAI-SearchBot covered about five US cities. PerplexityBot was smaller still: 12 subnets, all of them in Ashburn, Virginia (EdgeComet).

Compare that to the crawlers those teams are competing with. Googlebot spanned 165 subnets across 22 countries, and Bingbot reached 14 countries including Beijing and Dubai (EdgeComet).

One agent broke the pattern, and the exception is instructive. ChatGPT-User, the agent that fetches a page because a person asked for it in the moment, ran from 224 subnets across 15 countries, with 48% of its subnets outside the US (EdgeComet). Live user-initiated fetches follow the user. The bulk crawling that builds the index behind the answer does not.

Treat those figures as a snapshot, not a constant. Published ranges change when a vendor adds capacity or shifts cloud regions, which is exactly why the JSON files exist. The shape of the finding is the durable part: index-building AI crawlers are concentrated in one country, and search crawlers are not.

#Google fixed this for itself

Google documented the same failure mode years ago and built around it. Its guidance on locale-adaptive pages warns plainly that "Google might not crawl, index, or rank all your content for different locales," for two reasons: Googlebot's default addresses appear to be US-based, and Googlebot does not send an Accept-Language header (Google Search Central).

Google's answer was geo-distributed crawling. The same documentation now states that Googlebot crawls with IP addresses based outside the USA in addition to US ones, and tells site owners to treat Googlebot from a given country the way they would treat a real user from that country (Google Search Central).

No AI engine has published an equivalent. Until one does, the US fallback is the canonical version of your site as far as an AI index is concerned.

#The two signals your server never receives

When an AI crawler requests a page, your locale logic runs with both of its usual inputs missing or wrong.

The IP resolves to a US datacenter, so any country-based branch takes the American path. And no Accept-Language header arrives, so any content negotiation falls through to whatever your default is. If your default is a language picker, a redirect, or an interstitial, that is the page that gets read and stored.

The consequences are ordinary and easy to miss:

• Country redirects. A visitor from a US IP hitting example.com/product gets a 302 to /en-us/product. Your German, Japanese and Brazilian pages are then reachable only by users who were never being crawled from those countries in the first place.
• Regional availability gates. Pages that render "not available in your region" for US traffic hand the crawler a blank product.
• Currency and pricing. Prices shown in USD only, with no other variant at a stable URL, means the model has one price to cite for every market you sell in.
• Consent and region walls. A GDPR consent interstitial usually will not fire for a US-geolocated request, which is the lucky case. A wall keyed to any non-browser client is not.
• Country-level WAF blocks. These work in reverse. Blocking everything outside your served regions is safe for AI crawlers only as long as they stay in the US, which is a bet on someone else's infrastructure decisions.

We covered the adjacent case, where the same brand ranks differently depending on the language of the prompt, in AI visibility is per language. This is the supply-side half of that problem: the engine cannot answer in a language whose page it never retrieved.

#Serve one URL per variant

The fix has been the standard recommendation for international SEO for years, and it now matters more than it did. Google's own advice is to use separate locale URL configurations annotated with rel="alternate" hreflang instead of adapting one URL (Google Search Central). That advice survives the shift to AI retrieval intact, because a distinct URL is the only thing an AI crawler standing in Phoenix can actually fetch.

Four things to check this week:

1. Request your own site from a US datacenter with no Accept-Language header and read what comes back. That is the input to every AI index you care about. Do it for a product page, a pricing page and a localized landing page.
2. Replace IP-based redirects with a suggestion. Detect the visitor's country if you want, then offer a banner linking to the local URL rather than forcing a 302. The crawler keeps the page it asked for.
3. Make every locale reachable by a stable, linkable, crawlable URL, listed in your sitemap and cross-referenced with hreflang. See canonical and hreflang in an AI answer world for how those annotations behave once retrieval, not ranking, is the goal.
4. Audit country blocks and consent walls at the edge, not in your application. The block that returns a 403 to a crawler runs before robots.txt is ever consulted.

The measurement that closes the loop is simple. Pull your verified AI bot requests, filtered to the published IP lists above, and check which locale paths appear. If every AI crawler request in your logs resolves to /en-us/, your other markets are not underperforming in AI answers. They were never read.