What is Pay Per Crawl?
AI crawlers often consume vast amounts of web content. Some provide mutual benefit to content owners by indexing content for search engines, but others engage in activities such as content scraping without permission.
The resulting landscape leaves content owners with limited options for managing AI crawlers or receiving compensation for automated access to their intellectual property.
Pay per crawl is a feature of AI Audit that enables site owners to control and monetize AI crawler access to content by setting a price per zone.
Each time an AI crawler requests content, they either present payment intent via request headers for successful HTTP 200 access, or receive an HTTP 402 Payment Required response with pricing. Cloudflare acts as the Merchant of Record for pay per crawl and also provides the underlying technical infrastructure.
Ultimately, pay per crawl enables:
- Site owners to take control of their content, and charge a fee every time an AI crawler accesses a page in their Cloudflare zone. For more details, refer to use pay per crawl as a site owner.
- AI crawler owners to pay to access content on sites protected by pay per crawl. For more details, refer to use pay per crawl as an AI owner.
Refer to the following resources.
Was this helpful?
- Resources
- API
- New to Cloudflare?
- Products
- Sponsorships
- Open Source
- Support
- Help Center
- System Status
- Compliance
- GDPR
- Company
- cloudflare.com
- Our team
- Careers
- 2025 Cloudflare, Inc.
- Privacy Policy
- Terms of Use
- Report Security Issues
- Trademark