A site can now stay visible in Google while refusing AI training
On September 15, 2026 Cloudflare announced a setting called Disallow AI Training in an official blog post. It publishes the preference in robots.txt: Googlebot, Applebot and Bingbot keep crawling for search while AI training on the content is refused. Google and Apple have committed that this does not affect search ranking; Microsoft is building its robots.txt-based mechanism for early 2027. The controls are available on every plan, including free.
One setting separates two decisions: yes to search, no to training
According to Cloudflare's official blog post of September 15, 2026, titled "Have it both ways: stay discoverable in search while disallowing AI training", the new Disallow AI Training setting is built for mixed-use crawlers, bots that Google, Apple and Microsoft use for both search indexing and AI training. The setting publishes the preference in the site's robots.txt: Googlebot, Applebot and Bingbot keep accessing the site for search crawling while AI training is disallowed. In practice a Disallow rule is added for Google-Extended on Google's side and Applebot-Extended on Apple's side, and Cloudflare reports that both companies have committed that the preference does not affect search ranking. Search Engine Journal covered the news the same day under Matt G. Southern's byline and noted that the related Bot Preference Sync feature was introduced in August 2026. Microsoft is the exception for now: Bing takes the training preference via the NOARCHIVE meta tag and is building a robots.txt-based mechanism for early 2027.
Three controls, four options and a migration effective September 15
Cloudflare splits the controls into three: Search, Training and Agent. Options are Allow, Disallow AI Training, Block on pages with ads and Block; the second exists only under Training, the third blocks crawlers only where ads are detected. The controls sit at domain level under Security Settings, on every plan including free. The old Block AI Bots toggle and Managed Robots.txt are being retired; Managed Robots.txt users move to Bot Preference Sync, effective September 15. Migration rules: for Training, previous Block or Block on pages with ads selections become Disallow AI Training; domains with Block AI Bots off keep all three controls at Allow; domains with it on get Search Allow, Training Disallow AI Training and Agent Block on pages with ads. That trio is also the recommended default for new ad-monetized domains; sites without ads get Allow on all three with Preference Sync on. Google promises URL-level transparency tools for Google-Extended in the coming weeks; Apple plans a URL-level review tool in 2027.
Why it matters: search traffic and training data are now separate decisions
Cloudflare's own figures show less than 1 percent of its sites blocking search bots while 17 percent use a mechanism that blocks training. The same post says more than half of consumers read summaries in search, summary readers are over 40 percent more likely to end their search there, and consumers referred by AI search convert at three to five times or more the rate of traditional search. Cloudflare gives mixed-use crawler operators an Accountable label under four conditions: a robots.txt or similar training opt-out, a way to opt out of AI summaries (directly with the operator for now, through Cloudflare next year), URL-level visibility into which pages are open to training plus search metrics, and an assurance that refusing training will not affect traditional search results. Amazon, Anthropic, Meta and OpenAI crawlers also qualify because they run separate search and training bots, and Cloudflare Radar tracks all of them publicly. Cloudflare has talked to operators directly since July, expects open standards such as ai-prefs to matter, and targets content volume controls for AI summaries by early 2027. The sources name no external source for the statistics, no executives, and no technical detail on how robots.txt is edited.
What it means for sites in Türkiye: check your existing Cloudflare settings first
First the boundary: neither source contains anything specific to Türkiye or the region; the announcement applies to all Cloudflare customers. The practical meaning is clear, because the controls exist on every plan, including free. A Turkish SMB site, e-commerce store or ad-monetized publisher on Cloudflare can stay visible in Google and Apple search while refusing AI training with a single setting. The order we suggest: first, open the three controls under Security Settings and read the current state; the migration rules apply automatically based on the old Block AI Bots setting, so your site may already be on a new combination. Second, check your robots.txt, confirm the Google-Extended and Applebot-Extended lines read as you expect, and if you used Managed Robots.txt, verify the move to Bot Preference Sync. Third, if Bing traffic matters, Microsoft's robots.txt support is not expected before early 2027; until then the training preference depends on the NOARCHIVE meta tag. Fourth, if your site earns ad revenue, watch which pages the Block on pages with ads option in the Agent control covers.
The UNALSOFT view
Our reading: robots.txt is no longer a technical detail, it is the record of a site's commercial choice for the AI era. Cloudflare's data shows site owners want to close the door on training without leaving search, and the new setting separates the two. In our web design deliveries we treat robots.txt, crawler preferences and the three Cloudflare controls as part of the launch checklist, and decide with the client where content should appear and where it should not be used. The right combination depends on the revenue model: an ad-monetized publisher and an e-commerce store do not have to make the same choice.
Sources
Cloudflare Blog, official announcement · Search Engine Journal, news report
Want to review your site's AI crawler settings together?
Let's check your Cloudflare controls, your robots.txt and your search visibility in a single session. A short conversation is enough to start.