Changelog

Subscribe to RSS

2025-07-01

Introducing Pay Per Crawl (private beta)

We are introducing a new feature of AI Audit — Pay Per Crawl. Pay Per Crawl enables site owners to require payment from AI crawlers every time the crawlers access their content, thereby fostering a fairer Internet by enabling site owners to control and monetize how their content gets used by AI.

For Site Owners:

• Set pricing and select which crawlers to charge for content access
• Manage payments via Stripe
• Monitor analytics on successful content deliveries

For AI Crawler Owners:

• Use HTTP headers to request and accept pricing
• Receive clear confirmations on charges for accessed content

Learn more in the Pay Per Crawl documentation.

2025-07-01

AI Audit refresh

We redesigned the AI Audit dashboard to provide more intuitive and granular control over AI crawlers.

• From the new AI Crawlers tab: block specific AI crawlers.
• From the new Metrics tab: view AI Audit metrics.

To get started, explore:

• Manage AI crawlers.
• Analyze AI crawlers.

2024-09-23

AI Audit

Every site on Cloudflare now has access to AI Audit, which summarizes the crawling behavior of popular and known AI services.

You can use this data to:

• Understand how and how often crawlers access your site (and which content is the most popular).
• Block specific AI bots accessing your site.
• Use Cloudflare to enforce your robots.txt policy via an automatic WAF rule.

To get started, explore AI audit.