Google, OpenAI, Microsoft and Anthropic have agreed to honour one shared signal that lets any website say "do not use my content to train AI." Instead of blocking each company's crawler by name, site owners will add a single rule to their robots.txt file. Anthropic already supports it, Microsoft starts on 1 February 2027, Google in 2027 and OpenAI "in the coming months."
READ ALSO: Mistral Releases Large 4 Weights: A Trillion-Parameter Open Model
READ ALSO: Google Launches Free Skilling Africa Hub With Offline AI Lessons
Opting out of AI training with the new signal does not remove your site from Google Search.
What changes
Today, a blogger in Lagos who does not want their articles used to train AI has to know the name of every AI crawler and block each one separately: Google-Extended for Gemini, GPTBot for OpenAI and so on. New crawlers appear all the time, so most site owners never keep up.
The new signal comes from a draft standard at the Internet Engineering Task Force, written by Anthropic's Kevin Kelley. It defines one shared "policy token" for AI training. One rule in robots.txt covers every company that honours it. An example from the proposal looks like this:
| robots.txt line | What it means |
|---|---|
| User-Agent: __usage_ai-training | The rule applies to AI training by any provider that supports the signal |
| Disallow: / | Do not use anything on this site for AI training |
| Allow: /press-kit/ | Except this folder, which may be used |
Who supports it and when
| Company | Support |
|---|---|
| Anthropic | Already supported |
| Microsoft | From 1 February 2027 |
| Planned for 2027 | |
| OpenAI | "In the coming months" |
Google explained how it will read the signal in a Search Central post by John Mueller on 30 October. If a site opts out of training with the shared token and does not mention Google-Extended separately, Google will treat it as if the site had blocked Google-Extended, the crawler that gathers content to train Gemini. Existing company-specific rules keep working alongside the new one.
What it does not do
This is the part site owners should read twice:
- It does not block Google Search. Your pages still appear in search results.
- It does not undo past training. It only applies to crawling from the moment each company starts honouring it.
- It only covers training. Other uses, such as AI answers that quote and link to your page, are still being negotiated in the same standards group.
- It works only for companies that honour it. Crawlers that ignore robots.txt are not stopped by it.
Google also notes a trade-off. Sites that specifically allow Google-Extended can appear in responses in Gemini apps and receive clicks from them. Blocking training may cost some of that exposure.
What Nigerian bloggers and publishers should do
- Decide what you want. If your income depends on search traffic and you are happy to be in AI answers, you may not need to block anything. If you sell original reporting, research or courses, opting out protects it.
- Find your robots.txt. It lives at yoursite.com/robots.txt. On WordPress, SEO plugins such as Yoast or Rank Math let you edit it from the dashboard.
- Keep your existing rules. Do not delete lines that block or allow Googlebot for search.
- Add the shared token when the companies switch it on, and keep any crawler-specific rules you already rely on until support is confirmed.
- Check your Search Console after any change to make sure Googlebot can still reach your pages.
Why it matters
For years, AI companies trained on the open web with little say for the people who wrote it. A single, shared opt-out that the biggest labs agree to honour gives small publishers the same control big media houses negotiate in licensing deals. It is a meaningful step for African bloggers and publishers, even though it only protects content from the point each company starts honouring it.
One robots.txt rule will soon tell the biggest AI companies to stay out of your content for training, without hurting your Google rankings.
