Independent AI intelligence Two editions daily · ET
Fervor AI
Menu

Analysis · November 2, 2026 · concept

Machine-Readable Training Preference Signalprivacyregulationagent-infrastructure

robots.txt Is Getting One AI Training Opt-Out. Your Old Bot Rules Can Overrule It.

A new shared token separates "don't train on this" from "don't crawl this." Here is how it works, where it breaks, and what to change in your file this week.

robots.txt has only ever known two words: come in, or stay out.

On October 30, a draft standard proposed a third: read this if you like, but don't train on it. One shared line now means "AI training" to every provider that agrees to read it. Google has said it will. OpenAI says it will too.

The catch is the part nobody puts in the announcement. The new line does not replace the old ones. When your old vendor lines say something different, they win.

That matters because of how we got here. The same crawl can now feed a search index, a chatbot's live answer, and the training set for the next model. Site owners who wanted to be found but not trained on had no clean way to say so, so they kept a growing list of vendor bot names (GPTBot, Google-Extended, ClaudeBot and friends), each with its own rules, each documented on a different page. Most of those lines are still in place.

So the useful question this week is not "should I add the opt-out?" It is "does the rest of my file agree with it?"

What actually shipped

The proposal is an IETF Internet-Draft called "Machine-Readable Training Preference Signal," written by Kevin Kelley of Anthropic and dated October 30, 2026. The IETF is the body that writes internet standards, and an Internet-Draft is a proposal on its way through that process. This one is an individual draft, which means it has no formal standing yet.

It defines a single policy token: __usage_ai-training. A policy token looks like a crawler name in robots.txt, but no crawler uses it. It exists only to carry a preference.

Two vendor commitments matter:

  • Google. Its crawler documentation says it plans to support the token in early 2027. Google's John Mueller, quoted by Search Engine Roundtable, put it this way: "We aim to support this new control along with other AI providers in 2027."
  • OpenAI. OpenAI published its own post the same day (summarized by Unite.AI): OpenAI aims to support the signal "in the coming months," calls it voluntary, and says it will not replace the robots.txt controls OpenAI already offers.

That last point is the whole article in one clause.

How the token works, in plain terms

Think of robots.txt as the sign on your front door. Until now, the sign could say "no entry" to specific visitors. The new token adds a second kind of sign: "you can come in and look, but don't copy the furniture."

Here is the entire syntax, straight from the draft:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Read it as: no AI training on anything, except the press kit, where training is welcome. It uses the same path matching robots.txt has always used (RFC 9309, the standard version of the format), so the longest matching path wins inside the group.

Three details in the draft change how you should write it.

It does not control crawling. The draft says plainly that the token "is not a directive for crawling behavior." A bot that honors it can still fetch your pages; it just should not train on them. Your crawl rules stay exactly where they are.

The wildcard does not count. Many sites already have a blanket rule:

User-agent: *
Disallow: /private/

The draft says wildcard groups "MUST NOT be interpreted as matching the policy token." So whatever your * group says, it says nothing about training under this scheme. If you want the preference, you have to write the token by name.

No group means no opinion. If your file never mentions the token, you have not opted in or out. You have just not said anything, and each provider falls back to its own rules.

The catch: specific rules beat the shared one

The draft has a precedence rule that sounds harmless: "Specific expressions SHOULD take precedence over general ones." Vendors can also document their own tokens as more specific than the shared one.

Google already has. Its own token, Google-Extended, controls whether Google uses your content to train Gemini. Per Mueller's note, Google will read a __usage_ai-training opt-out as if you had disallowed Google-Extended, but only when your file does not mention Google-Extended itself. Name Google-Extended explicitly, and that line is what Google follows.

So picture a file that says this:

User-Agent: __usage_ai-training
Disallow: /

User-agent: Google-Extended
Allow: /

To a human, the first group reads like a firm no. To Google, the second group is the more specific statement, and it says yes.

That file is not far-fetched. Plenty of sites added Allow lines for AI bots during the last two years, sometimes to stay visible in AI answers, sometimes because a plugin did it. Those lines are still sitting there.

OpenAI's case is murkier. Its crawler docs say that "disallowing GPTBot indicates a site's content should not be used in training." That means a GPTBot line is already a training statement. OpenAI has not published how it will resolve a GPTBot line that disagrees with the shared token, so the only safe file is one where they agree.

The insight here is simple. The shared token is a default, not an override. It speaks for you only where you have not spoken more specifically.

The trade hiding inside Google-Extended

There is a second wrinkle, and it is the one most likely to surprise people.

Google-Extended does not only cover training. Google's crawler documentation says it also covers grounding, which is when Google feeds Search content to Gemini at answer time so the reply can cite fresh pages. Search Engine Roundtable's report adds that sites allowing Google-Extended stay eligible to appear in Gemini grounding responses.

Put those together. If Google maps the shared opt-out to a Google-Extended disallow as described, an opt-out written with the shared token could also take your pages out of Gemini's grounded answers. That is my reading of two Google statements, not something Google has spelled out, so treat it as a question to watch rather than a settled fact.

It does tell you what decision you are really making with Google. Today, "don't train Gemini on me" and "don't cite me in Gemini answers" travel on the same token. The shared signal does not split them for you. Google Search itself is unaffected either way; Google's docs say Google-Extended is not a ranking signal and does not change whether you appear in Search.

What the token does not do

  • It is voluntary. The AIPREF vocabulary draft this builds on says it "does not ensure preferences are followed," per Unite.AI's summary. A scraper that ignores robots.txt today will ignore this too.
  • It is not retroactive. The draft ties a preference to the robots.txt in effect when a page was fetched, and it does not ask providers to revisit content they collected earlier.
  • Private deals win. The draft says "Private agreements MAY override preferences expressed via this mechanism." If your platform or publisher signed a licensing deal, that deal decides.
  • It is narrow. "Training" here means changing a generative model's learned weights (the numbers it adjusts as it learns). The draft does not address search indexing, retrieval or live fetching for answers.
  • User-triggered fetches may skip it. OpenAI's bot docs say robots.txt rules "may not apply" to ChatGPT-User, the agent that fetches a page when a person asks ChatGPT to.
  • Google and OpenAI don't honor it yet. Google says early 2027. OpenAI says the coming months. For those two, everything you write now is a statement for later.

Put this into practice

None of this takes long. Budget fifteen minutes.

1. Read what your file says today. Pull the live file and list every named agent:

curl -fsSL https://yoursite.com/robots.txt | grep -i '^[[:space:]]*user-agent[[:space:]]*:'

Look specifically for GPTBot, Google-Extended, ClaudeBot, anthropic-ai, CCBot, Applebot-Extended, and any line with Allow: / under an AI-related name. Plugins and CDN settings sometimes write these without telling you. The -L follows redirects, which robots.txt is allowed to use, and remember that each hostname (www, blog, docs) serves its own file.

2. Decide per vendor, then write it down once. For each AI company you care about, answer two questions separately: may it train on my content, and may it show my pages in its answers? The second answer usually maps to a search bot (OAI-SearchBot for ChatGPT search, regular Googlebot for Google Search).

3. Add the shared group, and make your named lines agree with it. A consistent "find me, don't train on me" file looks like this:

# Shared AI training preference (IETF draft, honored by providers on their own schedules)
User-Agent: __usage_ai-training
Disallow: /
Allow: /press/

# Vendor training tokens, kept consistent with the line above
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# Search crawlers you still want
User-agent: OAI-SearchBot
Allow: /

If you want Gemini to keep citing you, the Google-Extended line is where you will have to choose, and today that choice also covers training.

4. Leave your * group alone. It still governs crawling, and under the draft it never spoke for training anyway.

5. Put a date in your calendar for early 2027. That is when Google's support is due. Re-check what Google and OpenAI have actually shipped, and adjust.

Expect nothing visible to happen when you deploy this. No crawler will behave differently tomorrow. You are writing down a preference that providers have promised to read later, and the main thing you control right now is whether that preference contradicts itself.

Honest limitations

I want to be clear about what I could and could not check. OpenAI's announcement was unreachable from my end, so I worked from Unite.AI's summary, and an independent check against OpenAI's post confirmed those points. The grounding consequence for Google is my inference from two of Google's own statements, not a quote. The draft is a first version (-00) and will change. And none of this protects you from bots that ignore robots.txt, which is a different problem that firewalls and bot management solve, not text files.

A preference is a request. It works on companies that care about being seen to honor it, and that is a real but partial set.

Where this leaves you

For thirty years, robots.txt has been a polite agreement between people who wanted to be found and companies that wanted to find them. This draft is the first time the file has a word for "learn from me" as separate from "look at me." That is a genuine improvement, and it costs you two lines.

But a two-line opt-out sitting on top of a file full of old, contradictory bot rules says nothing clearly. Open your robots.txt today, read it the way a vendor will, and make every line tell the same story. Then decide for yourself which bargain with which company is worth it.

Sources: IETF draft-kelley-aipref-training-preference-signal-00 (October 30, 2026); Google common crawlers documentation, Google-Extended section; Search Engine Roundtable on Google's support; OpenAI announcement (not reachable during research; summarized via Unite.AI); OpenAI crawler documentation; RFC 9309.