robots.txt now asks what for, not just who

A content signal is a line in robots.txt that declares what your content may be used for, separately from which crawler is allowed to fetch it. That distinction is new, and it matters more for AI visibility than most teams have noticed.

For thirty years robots.txt answered one question: who gets in. You named a user agent and gave it paths. The AI era strained that model, because the same company now runs several bots with different jobs. OpenAI documents GPTBot as the crawler for content that may be used in training its foundation models, and OAI-SearchBot as the one that surfaces sites in ChatGPT's search features (OpenAI crawler docs). Blocking one is a very different business decision from blocking the other. We covered that user-agent split in robots.txt for AI: partial-access patterns.

Two newer standards add a second axis on top of it: purpose.

#The three content signals

Cloudflare's Content Signals Policy extends robots.txt with a Content-Signal line carrying up to three named uses (Cloudflare):

• search: building a search index and returning links and short excerpts.
• ai-input: feeding content into a model at answer time, which the policy describes as retrieval augmented generation, grounding, or other real-time use for generative AI answers.
• ai-train: training or fine-tuning models.

The syntax sits inside a normal user-agent group:

User-Agent: *
Content-Signal: search=yes, ai-train=no
Allow: /

Cloudflare applied exactly that default to the 3.8 million domains already using its managed robots.txt feature (Cloudflare). Read the line again and notice what is missing. ai-input is not there. Cloudflare left it out deliberately, on the grounds that it does not know a given site owner's preference and does not want to guess.

#ai-input is the signal that decides citations

This is the part worth internalizing. Of the three signals, ai-input is the one that governs whether your page can be pulled into a generated answer at query time. That is the mechanism behind almost every citation you would want from ChatGPT, Perplexity, Gemini, or AI Overviews. Training is about future model weights. Retrieval is about today's answer, and today's link.

The Content Signals Policy states that when a signal is absent, the operator neither grants nor restricts permission for that use (Cloudflare). So a large population of sites currently ships a robots.txt that says yes to search, no to training, and nothing at all about the use that actually produces citations. That is not a bug in the policy. It is an unfilled field, and it is yours to fill.

If your business wants AI answer traffic, the honest expression of that is explicit:

User-Agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

If you sell the content itself and want no part of generative answers, the opposite is equally expressible. What you should not do is leave the field blank and then wonder why your stance is ambiguous to the systems reading it.

#RSL turns the signal into a licence

Really Simple Licensing takes the same idea further. Launched on 10 September 2025 by the RSL Collective, co-founded by RSS co-creator Eckart Walther and former Ask.com chief executive Doug Leeds, it lets publishers attach machine-readable licensing terms to their content (Wikipedia). Early backers included Reddit, Yahoo, Medium, Quora, O'Reilly Media and Ziff Davis (Search Engine Land).

The 1.0 specification, published on 10 December 2025, defines an XML document discoverable several ways, including a License: directive in robots.txt pointing at an absolute URL, a Link header with rel="license", or an embedded <script type="application/rsl+xml"> block (RSL 1.0 spec). Inside, a <payment type=""> attribute selects the commercial model: free, attribution, purchase, subscription, training, crawl, use, or contribution. The spec also carries its own usage vocabulary, with tokens including ai-train, ai-input, ai-index and search.

Note attribution in that list. RSL is usually discussed as a payment mechanism, and that framing undersells it for anyone whose goal is visibility rather than revenue. A licence that grants free use in exchange for visible credit and a working link is a licensing position, not a paywall. It is closer to what most marketing teams already want from AI assistants, written down in a form a machine can parse.

#What these signals cannot do

Neither standard is enforcement. The Robots Exclusion Protocol became a formal internet standard in 2022 as RFC 9309, and that document is explicit that compliance is voluntary. Content signals inherit the same property; Cloudflare's own guidance is to pair them with firewall and bot management rules if you want them enforced rather than merely stated. Search Engine Land made the same point about RSL, noting that it depends on AI companies choosing to participate and that model builders have a history of ignoring robots.txt (Search Engine Land).

Where a declared preference does help is legal and commercial. A signal is evidence of an expressed position, machine-readable and timestamped, which is a better starting point for a licensing conversation than nothing. It also costs one line of text.

#What to do this week

Open your robots.txt and check four things.

1. Whether a Content-Signal line exists at all. If your site sits behind a CDN with managed robots.txt, one may have been added for you, with ai-input unset.
2. Whether the signal you publish matches your actual commercial intent. A search=yes, ai-train=no default is a reasonable stance for most B2B software companies, but it is silent on the use you probably care about most.
3. Whether your user-agent rules contradict your signals. Declaring ai-input=yes while disallowing the retrieval crawlers that would act on it is a mixed message, and the user-agent block is the one that actually stops a fetch.
4. Whether you want a licence at all. If attribution is the outcome you want, RSL gives you a way to say so.

Our own crawler reads robots.txt on every audit and flags the contradiction in point three, because it is the one that quietly costs citations. The rest is a policy decision, and it belongs to you rather than to your CDN's default.