An AI Training Management Service That Preserves Search Visibility

Search and AI training can now be managed separately

On October 30, 2026, Google announced that it would support a common way for websites to signal to multiple AI companies that they do not permit training use. The approach uses a marker called __usage_ai-training in robots.txt, the public instruction file placed at a site’s root address. Google aims to support it in early 2027.

The marker uses existing robots.txt syntax. A site can, for example, opt out of training use across the whole site while allowing only its press-release path. This reduces the work of finding and entering the different crawler names used by each company.

Google plans to interpret the common marker as an opt-out from Google-Extended when the common signal is set to deny use and no separate Google-specific rule exists. Google-Extended is Google’s own marker, available since 2023. Google says it limits use for future Gemini model training and certain Search grounding features without affecting inclusion or ranking in Google Search.

However, the common approach is still an initial draft submitted to an international organization that discusses internet technical rules. It is not a finalized standard, so its name and order of application may change. You also need to check separately which companies beyond Google will support it, and when.

This file is closer to a usage notice posted at an entrance than a locked door. It does not stop crawlers that ignore the notice from accessing a site. Actual blocking requires separate measures, such as a security service in front of the site or a login requirement.

A new operating budget for content owners

Consider the content lead at a paid tax-information site with 12 employees. This person needs to decide how far years of explanatory articles can be used for AI training, while keeping search visitors.

Today, they ask a developer to update robots.txt, then repeat the same decision across four places: Google Search Console, the site’s front-end security service, and the terms-of-use page. When an AI company introduces a new marker, they must find where it applies and check again that it does not conflict with search-crawling rules.

The issue is not just one line of configuration. It is the management that follows. If there is no record of who selected a site-wide opt-out, when they did so, whether they allowed only specific materials, or whether last month’s setting remains unchanged, the reasoning disappears when the person in charge changes.

Once the common marker is supported, a workflow could let users choose the scope of training permission on one screen, then set separate exceptions only for companies that need them, as with Google-specific rules. If the service generates the final notice and compares it with the file actually on the site, it can immediately show incorrectly copied lines or conflicting settings.

Many decisions still remain with people. The content lead must decide whether free public articles and paid materials receive the same terms, how to weigh the benefit of appearing as a source in search answers against opting out of training, and whether to actually block crawlers that ignore the signal.

The person most likely to receive a new budget is not the head of development, but the person responsible for content operations or rights management. They need to explain results through whether search traffic was maintained, whether all opted-out paths were reflected, and whether configuration-change and response records can be submitted on time.

So the product to sell is not a simple configuration-file generator. A service has a clearer reason for approval when it checks status every month, stores before-and-after changes, and produces a one-page record for management or clients.

Abroad, the model is expanding from control to transactions

In the United States, Cloudflare began a private test of “Pay Per Crawl” in July 2025. The feature lets site operators allow or block AI crawlers and put a price on access. Crawlers that do not accept the price receive a “402” response indicating that payment is required.

As of July 2026, the feature remained a test available only to a limited set of customers. Cumulative transaction value and the number of participating companies have not been disclosed. Still, it points to a direction in which configuration-management budgets could later extend to budgets for content-access fees and settlement management.

The RSL standard, which began in the United States, aims to link a license document in robots.txt and specify separate terms for training, AI answer generation, and search indexing. It can include conditions such as free use, attribution, subscriptions, or payment per crawl.

RSL said that, after its September 2025 launch, more than 1,500 media, brand, and technology organizations worldwide had supported it by December of that year. This is a self-reported figure, and actual licensing revenue has not been disclosed. But it is an example of expanding a simple opt-out file into a document of commercial terms.

TollBit in the United States lets content sites set prices by page and purpose of use, then connects AI services to receive material through authorized paths. Operators can see which crawler requested which page and can require payment or a contract for unauthorized requests.

TollBit says its network handles more than 10,000 sites, but it has not disclosed the total amount actually paid out. The visible result so far is not large revenue, but an operating model that brings price lists, usage records, and a payment channel into one place.

Four things you can build now

1. An AI training opt-out configuration health check

This service lets an independent specialist publisher with a high share of search traffic enter a URL and find conflicts among Google Search rules, Google-specific AI training rules, and the common marker. It is needed now because existing and new settings need to be checked together before the common approach is introduced.

The first screen places a site-address field beside three statuses: “Keep search,” “AI training use,” and “Block actual access.” Each result shows the text to change and the affected paths.

2. A dashboard for managing multiple client sites

This service is for a web agency managing multiple hospital and shopping-site websites, showing each client’s permission policy and implementation status on one screen. Company-specific markers will remain alongside the common marker, so copying and checking work grows with the number of sites.

The first screen shows a list of client sites, the last-check time, whether settings conflict, and changes awaiting approval. The operator sends a proposed change to the client for approval, then downloads only the file to apply.

3. A vault for evidence of configuration changes

This service automatically preserves when and how notice files and terms of use changed for a company selling paid reports and educational materials. Because a voluntary signal is not enforced blocking, the first need is to preserve the terms displayed at the time and crawler records rather than assume a dispute.

The first screen shows a chronological change history, plus “who approved it,” “which paths changed,” and “whether an external crawler checked it.” Monthly records can be downloaded for internal reporting and contract discussions.

4. Content-use terms and an inquiry channel

This service is for an information site with hard-to-recreate material, such as industry pricing or legal explanations. It presents the permitted scope and a contract-inquiry method to AI companies. As per-request pricing and standard licenses are being tested abroad, there is a reason to add a negotiation channel after a simple opt-out.

The first screen includes options such as “Search allowed,” “Training prohibited,” and “Open to discussing answer citations,” along with a contract-inquiry button. Do not promise automatic payment. Instead, record the requesting company and the target pages so people can set the price and terms.

What to check in 30 minutes today

Choose ten sites in one industry you want as customers, then open each site by adding /robots.txt to its address. If three or more sites have AI-company-specific rules that vary widely or settings that are hard to distinguish from search rules, the first service—one that explains a site’s status from its URL alone—is worth testing.

Why this matters where you are

Check whether sites in your market already separate search crawling, AI training, and actual access blocking in their public rules. The specific markers and companies supporting them may differ from those described here. But the operational need to record decisions, compare live settings, and prepare evidence can be tested with sites in any market.