Skip to content
Cloudflare Docs

Changelog

New updates and improvements at Cloudflare.

Subscribe to RSS
View all RSS feeds

hero image

Bot & Crawler Insights in Cloudflare Radar

Web crawlers insights

Radar now offers expanded insights into web crawlers, giving you greater visibility into aggregated trends in crawl and refer activity.

We have introduced the following endpoints:

These endpoints allow analysis across the following dimensions:

  • user_agent: Parsed data from the User-Agent header.
  • referer: Parsed data from the Referer header.
  • crawl_refer_ratio: Ratio of HTML page crawl requests to HTML page referrals by platform.

Broader bot insights

In addition to crawler-specific insights, Radar now provides a broader set of bot endpoints:

These endpoints support filtering and breakdowns by:

  • bot: Bot name.
  • bot_operator: The organization or entity operating the bot.
  • bot_category: Classification of bot type.

The previously available verified_bots endpoints have now been deprecated in favor of this set of bot insights APIs. While current data still focuses on verified bots, we plan to expand support for unverified bot traffic in the future.

Learn more about the new Radar bot and crawler insights in our blog post.