Your site, your rules: new AI traffic options for all customers
15:00 · July 1, 2026 · Cloudflare AI Blog
For our second Content Independence Day, we’re giving website owners finer options to manage AI traffic. Instead of a one-size-fits-all block, all customers can now easily distinguish and manage Search, Agent, and Training bots, alongside the new ability to protect ad-monetized pages.
Summary
Cloudflare has expanded its bot-management tools to give site operators finer control over automated traffic from AI systems. Rather than treating all “AI bots” as a single category, the company now classifies crawlers according to three primary behaviors: Search, which builds an index to answer later queries and is expected to drive referral traffic; Agent, which performs real-time actions on a user’s behalf, such as fetching content for chat interfaces or browser agents; and Training, which permanently incorporates scraped material into model parameters.
These distinctions are accompanied by updated default settings that take effect on 15 September 2026. On pages that display advertising, Training and Agent crawlers will be blocked by default while Search crawlers remain permitted. Multi-purpose bots that combine Search with Training will be governed by the most restrictive applicable rule, so operators that have chosen to block Training will also block the combined crawlers unless they explicitly opt out beforehand.
Enterprise customers receive an additional visibility layer called BotBase, a searchable directory that lists every known bot together with its assigned behaviors and content-use classification. The same taxonomy underpins a new robots.txt signal, “use,” which lets owners express preferences for how their content may be retained: immediate (no storage or reuse), reference (indexing and excerpting with links), or full (summarization and reproduction). Cloudflare-managed robots.txt files now include this parameter by default, and bots that disregard the declared preference risk losing Verified status.
Together, the classification scheme, default policies, BotBase directory, and content-use signal replace the earlier binary “Block AI bots” toggle with a set of granular levers that apply to all customers, including those on the free tier.
Why it matters
This article is highly relevant for Security and Privacy Professionals in the Netherlands as it provides actionable, technical controls to enforce data privacy and protect corporate content from unauthorized AI scraping. Implementing these Cloudflare features aligns perfectly with EU data protection standards and helps organizations mitigate risks associated with shadow AI data collection.
