Unmasking the crawls with Attribution Business Insights
08:00 · July 1, 2026 · Cloudflare AI Blog
Cloudflare’s new Attribution Business Insights dashboard helps website owners understand crawler behavior, appetite, and potential value, fueling business-level conversations around crawl compensation.
Summary
Cloudflare has introduced the Attribution Business Insights dashboard for customers of its Bot Management service. The tool supplies publishers and business teams with a consolidated view of automated traffic, distinguishing human visitors from different categories of bots and highlighting which operators extract content without returning meaningful referral traffic.
The dashboard addresses a shift in web economics that began with traditional search-engine crawlers. Those systems maintained relatively balanced crawl-to-referral ratios, typically scanning pages only a few times for each visitor sent back to the site. In contrast, contemporary AI crawlers exhibit ratios ranging from 118:1 to nearly 50,000:1, imposing infrastructure costs while bypassing the page views that sustain advertising, subscriptions, and direct audience relationships.
Key metrics presented include site-wide and per-operator crawl-to-referral ratios over 24-hour, seven-day, or 30-day windows; a breakdown of top bots by volume, origin country, and bandwidth consumption; and an updated taxonomy that labels crawlers according to purpose—Training for model development, Search for retrieval-augmented generation indexes, or Agent for direct user-facing responses. The interface also separates bot activity on content pages from overall traffic and shows whether each operator is currently allowed or blocked.
By surfacing these figures in a form accessible to non-security stakeholders, the dashboard supports evidence-based decisions on access policies and licensing negotiations. Action remains confined to existing security rules, keeping the analytics layer focused on investigation rather than enforcement.
Why it matters
This article is highly relevant for Dutch security and privacy professionals as it provides actionable tools to manage AI data scraping, a critical issue under EU copyright and data protection frameworks. It empowers organizations to protect proprietary content, manage infrastructure costs, and enforce data privacy against aggressive AI crawlers.
