Blocking AI training has so far meant writing a different name for each company in robots.txt, such as Google-Extended, GPTBot and ClaudeBot, and blocking them one by one.
On October 30, 2026, Google said it will support a shared AI training preference signal that several AI companies read, in 2027. This signal is only a statement of "please don't use this for training," though, so search visibility and citation in AI answers still have to be decided separately.
"I don't want our blog posts used for AI training, but what if blocking them drops us from search?" If you run a company blog, you've probably had this worry at some point. Each AI company uses different crawler names, and even within one company, training and search crawlers are split, so it's easy to get confused. In this post, we'll look at what Google's shared signal is, how the methods available today differ from each other, and what criteria marketers can use to review robots.txt.
What is the "shared AI training signal" Google plans to support?
According to Google's Search Central blog announcement, Google plans to support a shared "policy token" in robots.txt, based on work discussed in the AIPREF working group at the IETF (the international body that develops internet standards).
The goal is that when a site opts out of AI training with this token, it can send the same message to all participating AI companies at once.
Google said that if a site opts out with this token and doesn't address Google-Extended separately, Google will treat it as if Google-Extended were disallowed.
The token name doesn't appear in the blog post itself; it's written in the Google-Extended section of Google's crawler documentation and in an IETF draft.
The individual IETF draft 「Machine-Readable Training Preference Signal」, posted the same day, doesn't create any new directives; it proposes using a token called __usage_ai-training in the User-Agent position with the existing robots.txt syntax.
Here's the example from the draft.
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
It means the whole site should not be used for AI training, while the press kit folder (/press-kit/) may be used for training.
Here, AI training follows the definition in the IETF vocabulary draft: "using an asset to modify the learned parameters of a generative AI model," and models used only for classification or ranking are excluded.
As an analogy, the existing crawler block is like locking the front door of a building, while the new token is closer to leaving the door open and posting a usage notice on the wall inside that says "please don't take this material for training." The draft also makes clear that this token is not a crawling directive, and says to use groups named after crawlers if you want to reduce crawling itself.
Figure: Crawler-name groups block crawling itself, while the shared token only states a preference that crawled content not be used for training. (figure labels in Korean)
What methods can block AI training today?
The shared token is a 2027 story, so for now you have to use company-specific names. Even within one company, training and search bots are split, so the result depends on what you block. Here's what each company's documentation says, side by side using the same criteria.
| Name | Company | What it does | What happens if you block it |
|---|---|---|---|
| Google-Extended | A token that controls whether content is used for training Gemini models and for grounding (supplying search index content when answering) in Gemini Apps and Vertex AI | The documentation says it has no effect on inclusion or ranking in Google Search | |
| GPTBot | OpenAI | Crawls content that may be used to train generative AI foundation models | Signals that content should not be used for training |
| OAI-SearchBot | OpenAI | Used to show sites in ChatGPT's search features | The site is dropped from ChatGPT search answers (it can still appear as a navigational link) |
| ClaudeBot | Anthropic | Collects web content that may be used for training | Future site materials are excluded from training data |
| Claude-SearchBot | Anthropic | Improves search result quality for users | Visibility and accuracy in search results may drop |
__usage_ai-training | Shared (proposed) | States a preference about generative AI training | A signal hoping participating companies leave content out of training; Google aims to support it in 2027 |
Sources: Google-Extended documentation, OpenAI crawler documentation, Anthropic crawler help article.
There's one point worth stressing here.
Google-Extended is a "control name" with no separate HTTP user agent, so the crawling is done by existing Google crawlers, and the AI features documentation covering AI Overviews and AI Mode separately says to use nosnippet, data-nosnippet, max-snippet and noindex to limit the information shown in Search.
The same page introduces Google-Extended separately as a way to limit AI training and grounding in Google's "other systems," so it's hard to see it as a switch for reducing AI Overviews visibility. On the other hand, keep in mind that nosnippet also reduces regular search snippets.
There are also bots that visit when a user directly asks, such as ChatGPT-User and Claude-User, and OpenAI's documentation says robots.txt rules may not apply because these actions are initiated by a user.
Figure: Only the scope written in official documentation is shown. "No effect" means the documentation states so, "not mentioned" means the documentation has no information, and "not applicable" means it's another company's service and unrelated. (figure labels in Korean)
Block or allow? Decision criteria by type of site
So is blocking everything the safe choice? In my view, the first step is to separate "what exactly you don't want." That's because being used for training, being cited in AI answers and appearing in search results are now separate switches. The following are criteria I set based on the scope of the official documentation, so if your company has its own policy or legal guidance, that comes first.
| Site situation | Training bots (GPTBot, ClaudeBot, Google-Extended) | Search and answer bots (OAI-SearchBot, Claude-SearchBot) | Reason |
|---|---|---|---|
| Marketing blogs and product pages aimed at traffic | Consider allowing | Allow | The goal is wide reach, and Google-Extended is tied to Gemini Apps grounding |
| Paid reports and members-only material | Block | Depends | Apart from opting out of training, the basics are keeping logged-in content out of crawlers' reach in the first place |
| Sites whose assets are original photos, illustrations or writing | Block | Consider allowing | You can opt out of training while keeping search traffic |
| Press releases and notices you want spread widely | Allow | Allow | Like /press-kit/ in the draft example, you can allow just a folder |
You should also factor in that Google-Extended bundles training and grounding together.
Google's announcement explains that sites that explicitly allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps. That means blocking it just because you dislike training may also reduce your chances of being linked as a source in Gemini Apps answers, so you need to weigh the two.
So what should marketers change?
Let's walk through the steps assuming a marketer running a company blog has been given the policy "opt out of AI training, but keep search and AI answer visibility."
- Open your current robots.txt: Open
yoursite/robots.txtin a browser and check whether names likeGPTBot,ClaudeBotorGoogle-Extendedare already there, and whether there's a line that blocks everything, likeDisallow: /underUser-agent: *. - Split the policy by switch: As in the decision table above, write down training, AI search answers and Google Search separately, decide allow or block for each, and agree on it with your team.
- Block only each company's training names: As in the example below, block only the training names as groups and leave the search bots alone.
- Repeat for each subdomain: Anthropic's help article says you need a robots.txt for each subdomain you want to opt out, so if your blog lives on a separate host like
blog., add it there too. - Wait for it to take effect and keep a record: OpenAI's documentation says, for its search bot, that it can take about 24 hours for robots.txt changes to take effect. Recording the date and what you changed makes it easier to find the cause later if traffic shifts.
Below is an example for step 3. It's not the result of applying it to my site, but a template combining the names and syntax from the official documentation.
# Training: opt out
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# Shared signal (Google aims to support it in 2027, same existing syntax)
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
# Search bots (Googlebot, OAI-SearchBot, Claude-SearchBot) are not blocked
Whether to add the shared token now is up to you. Since Google's support target is 2027, what actually works today is the company-specific names.
Figure: The steps in order, from checking the current file to repeating for subdomains and recording changes. (figure labels in Korean)
※ Note: limits and caveats of the shared signal
- Still a proposal: The
__usage_ai-trainingdraft is an individual draft at version 00, and the document itself says it is not endorsed by the IETF. The vocabulary draft also says some sections do not yet reflect working group consensus, so details may change. - No enforcement: The AIPREF working group charter puts technical enforcement of preferences out of scope. Whether to honor the signal is up to each company, and Google's announcement doesn't name the other participating companies.
User-agent: *doesn't apply: The draft states that wildcard groups do not apply to the policy token. A single "block everything" line won't convey the training preference, so you have to write the token name directly.- Different wording on timing: The blog says "in 2027," while the Google-Extended documentation says "early 2027."
- Content already collected: The draft says the preferences in the robots.txt in effect when the asset was retrieved apply, and Anthropic's help article also says that after blocking, "future" materials are excluded. Blocking now doesn't undo past training.
- IP blocking isn't recommended: Anthropic's help article explains that blocking IPs can stop the bot from reading robots.txt, so your opt-out may not be conveyed properly.
Google said this approach is consistent with the EU AI Act's Code of Practice for general-purpose AI models, but the announcement says nothing about its legal meaning in Korea. If the material involves copyright, I recommend getting a legal review as well.
Wrapping up
Blocking AI training uses company-specific names today and is heading toward using a shared token as well from 2027, and either way, "training" and "search and answer visibility" have to be decided separately. I'll also split this blog's robots.txt by switch, and once support for the shared signal actually starts, I'll share what changed. For the detailed syntax, please refer to Google's announcement and the IETF draft.
FAQ
- Does blocking Google-Extended hurt my Google Search ranking?
- Google's crawler documentation says Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal. However, it applies not only to Gemini model training but also to grounding in Gemini Apps and Vertex AI, so blocking it may reduce your chances of appearing as a source in Gemini Apps answers.
- What's the difference between blocking GPTBot and blocking OAI-SearchBot?
- According to OpenAI's documentation, blocking GPTBot signals that your content should not be used to train generative AI foundation models, while blocking OAI-SearchBot drops your site from ChatGPT search answers. If you want to opt out of training but keep ChatGPT search visibility, block only GPTBot. It can take about 24 hours for robots.txt changes to take effect.
- Can I add __usage_ai-training to robots.txt now?
- It uses the existing robots.txt syntax in the User-Agent position, so adding it causes no syntax problem. However, Google aims to support it in 2027 and the draft is not yet endorsed by the IETF, so what actually works today is company-specific names like GPTBot, ClaudeBot and Google-Extended.
- If I block AI training in robots.txt, is content that was already trained on removed?
- Anthropic's help article says that blocking ClaudeBot excludes future site materials from training data, and the IETF draft says the robots.txt preferences in effect when the asset was retrieved apply. I couldn't find any official statement that content already collected gets removed.
Related posts
- AI Overviews and SEO: How Google's AI Search Changes Things - Conditions for appearing in AI Overviews and controls like nosnippet and Google-Extended
- SEO Basics: How Google Crawling, Indexing and Ranking Work - What robots.txt does at the crawling stage
- What Is Naver AI Briefing? How Content Gets Cited as a Source - Naver's crawler Yeti and the conditions for being cited in AI Briefing