Bot & Crawler Insights in Cloudflare Radar
Radar now offers expanded insights into web crawlers, giving you greater visibility into aggregated trends in crawl and refer activity.
We have introduced the following endpoints:
/bots/crawlers/summary: Returns an overview of crawler HTTP request distributions across key dimensions./bots/crawlers/timeseries_group: Provides time-series data on crawler request distributions across the same dimensions.
These endpoints allow analysis across the following dimensions:
user_agent: Parsed data from theUser-Agentheader.referer: Parsed data from theRefererheader.crawl_refer_ratio: Ratio of HTML page crawl requests to HTML page referrals by platform.
In addition to crawler-specific insights, Radar now provides a broader set of bot endpoints:
/bots/: Lists all bots./bots/:bot_slug: Returns detailed metadata for a specific bot./bots/timeseries: Time-series data for bot activity./bots/summary: Returns an overview of bot HTTP request distributions across key dimensions./bots/timeseries_groups: Provides time-series data on bot request distributions across the same dimensions.
These endpoints support filtering and breakdowns by:
bot: Bot name.bot_operator: The organization or entity operating the bot.bot_category: Classification of bot type.
The previously available verified_bots endpoints have now been deprecated in favor of this set of bot insights APIs.
While current data still focuses on verified bots, we plan to expand support for unverified bot traffic in the future.
Learn more about the new Radar bot and crawler insights in our blog post ↗.
Was this helpful?
- Resources
- API
- New to Cloudflare?
- Products
- Sponsorships
- Open Source
- Support
- Help Center
- System Status
- Compliance
- GDPR
- Company
- cloudflare.com
- Our team
- Careers
- 2025 Cloudflare, Inc.
- Privacy Policy
- Terms of Use
- Report Security Issues
- Trademark