Skip to content
Cloudflare Docs

What is Pay Per Crawl?

AI crawlers often consume vast amounts of web content. Some provide mutual benefit to content owners by indexing content for search engines, but others engage in activities such as content scraping without permission.

The resulting landscape leaves content owners with limited options for managing AI crawlers or receiving compensation for automated access to their intellectual property.

What is Pay Per Crawl?

Pay per crawl is a feature of AI Audit that enables site owners to control and monetize AI crawler access to content by setting a price per zone.

Each time an AI crawler requests content, they either present payment intent via request headers for successful HTTP 200 access, or receive an HTTP 402 Payment Required response with pricing. Cloudflare acts as the Merchant of Record for pay per crawl and also provides the underlying technical infrastructure.

Ultimately, pay per crawl enables:

Pay per crawl components

Additional resources

Refer to the following resources.