We respect your privacy.

We use strictly necessary cookies to keep you signed in and to protect against CSRF. With your permission we also use a small amount of first-party analytics to improve the product. We do not sell your data and we do not use third-party advertising trackers. See our cookie policy and privacy policy .

← All posts

Terms of service now govern AI agents

Crawlmind Engineering··5 min read

An AI agent policy is the set of rules a site sets for software that acts on a person's behalf, and United Airlines just showed that those rules can live in a contract rather than in robots.txt.

According to Search Engine Journal's reading of the new terms, United updated its website terms and its contract of carriage on Nov. 11. The website terms require written permission before any AI agent searches, books or buys on united.com or the app. Search engines that index the public pages are exempt. As of Nov. 14, SEJ reports, United's robots.txt named no AI crawlers or agents at all.

That combination is the useful part for anyone who works on AI visibility. The airline kept the door open to search engines, closed it to agents, and did all of it in legal text that no crawler parses.

#What the terms actually separate

SEJ's summary describes three different kinds of access, each handled differently.

Indexing. Search engines crawling public pages are explicitly exempt. Standard browsers and assistive technology used by a person are exempt too.

AI reuse of the data. The terms bar using United's fares, schedules, availability and pricing to train, fine-tune, ground or evaluate an AI model, per SEJ.

Agent actions. Any AI assistant, bot, scraper, browser extension or built-in browser feature that acts for a traveler needs permission, even when a person supervises or approves each step. SEJ notes that agent features built into a browser are not exempt.

The contract of carriage backs this up. SEJ reports that Rule 6.J.6 adds using AI agents to make, modify or service a ticket to the list of prohibited practices, alongside hidden-city ticketing. Rule 5.H lists bookings made or serviced by AI agents among those United may cancel without notice. Bookings made by agents before Nov. 11 will be honored, according to Business Traveller.

Trip planning with a general AI tool is still allowed, as long as the tool does not touch united.com or the app.

#Why robots.txt could not carry this policy

Robots.txt answers one question: may this user agent fetch this path. It was never designed to say "you may fetch this page to index it, but not to act on it for a customer." The standard itself says so. RFC 9309 states that its rules are not a form of access authorization.

United's distinctions do not map to user agents. An AI-enabled browser fetching a fare page looks like a person's browser, because in a sense it is one. A training pipeline and an index can draw on the same crawl. The line United cares about is purpose: index, reuse, or transact. Purpose is something a contract can name and a path rule cannot.

Google's own controls show the same split from the other side. Google's crawler documentation describes Google-Extended as a robots.txt token with no user agent of its own. It controls whether content is used to train Gemini models and to ground Gemini Apps and Vertex AI answers with content from the Search index. Google says it does not affect inclusion in Google Search and is not a ranking signal. The same page says Google plans to support the AIPREF training signal in early 2027.

So even inside one company's crawler stack, "be in search" and "be used by the model" are separate switches. United simply moved the switch it cared most about out of robots.txt entirely.

#The gap nobody has closed

SEJ points out that United's terms do not address answers that AI systems generate from search engine indexes. That is the case that matters most for AI visibility.

Consider the sequence. A search engine indexes a United fare page under the indexing exemption. An AI answer surface built on that index then summarizes the page for a user. The terms bar grounding on United's data, but they also exempt the indexing that feeds the grounding. Google's documentation places Gemini grounding under Google-Extended, and United's robots.txt, per SEJ, did not name it. Which rule wins is not settled by any source we found.

For a brand, this ambiguity is a planning risk in both directions. A site that wants AI answers to describe it accurately could write terms broad enough to discourage the engines that would do that. A site that wants to keep its pricing out of AI answers could find that its terms say one thing and its robots.txt says another, and that crawlers only read the second.

#How to set your own policy

Treat access as three decisions and write each one down where the relevant party will actually see it.

  1. Indexing. Decide which crawlers may fetch which paths, and express it in robots.txt. This is the only layer that automated systems read before they fetch.
  2. AI reuse. Decide whether you want your content used for training and for grounding, and set the tokens that exist for it, such as Google-Extended. Check that the tokens match whatever your terms claim. A ban in legal text with an open robots.txt is the inconsistent state United is in, per SEJ.
  3. Agent actions. Decide what software may do on your site for a user: read, add to cart, check out, change an account. This needs terms, and it needs detection, because agents in a browser do not identify themselves the way crawlers do. SEJ notes United has not said how it detects agents.

Then test the result from the outside. Ask the major AI answer engines about your products and see what they say, which pages they cite, and whether the answers draw on data you meant to keep out. If an engine quotes your prices while your terms forbid grounding on them, the terms are not doing what you think, and robots.txt is where the fix starts.

Most sites will land somewhere less strict than United. A retailer probably wants agents to find products and may even want them to check out. A publisher may want indexing and grounding with attribution but not training. The point of the United case is not the specific answer. It is that the three questions now have different answers, and a single robots.txt file can no longer hold all of them.

Related field notes

Share or discuss

Field notes in your inbox

New posts, no spam. Roughly monthly. Unsubscribe with one click.