Cloudflare Docs
Cloudflare Fundamentals
Cloudflare Docs
Cloudflare Fundamentals
Edit this page
Give us feedback
Set theme to dark (⇧+D)
  1. Products
  2. Cloudflare Fundamentals
  3. Reference
  4. Cloudflare crawlers

Cloudflare crawlers

Cloudflare may crawl or make HTTP requests to your site to make sure its protected and performing properly.

​​ Crawling situations

​​ Specific products

Cloudflare will crawl your site when you have specific products enabled:

  • Always Online
    • User-Agent: Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +http://www.cloudflare.com/always-online)
  • Health checks
    • User-Agent: Mozilla/5.0 (compatible; Cloudflare-Healthchecks/1.0; +https://www.cloudflare.com/; healthcheck-id: <HEALTHCHECK_ID>)
    • HEALTHCHECK_ID is a 16-character string associated with the health check ID.
  • Load balancing monitors
    • User-Agent: Mozilla/5.0 (compatible; Cloudflare-Traffic-Manager/1.0; +https://www.cloudflare.com/traffic-manager/; pool-id: <POOL_ID>)
    • POOL_ID is a 16-character string associated with the load balancing pool ID being monitored.
  • Prefetch URLs
    • User-Agent: Mozilla/5.0 (compatible; CloudFlare-Prefetch/0.1; +http://www.cloudflare.com/)
  • SSL/TLS recommender
    • User-Agent: Cloudflare-SSLDetector
    • This crawler ignores your robots.txt file unless there are rules explicitly targeting the user agent.
  • Security Insights
    • User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36 (compatible; +https://developers.cloudflare.com/security-center/)

​​ Other situations

Cloudflare will also crawl your site in other, specific situations:

  • Speed tests
    • User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36 PTST/190628.140653
    • Triggered when: You launch a speed test from within the Cloudflare dashboard.
  • Support diagnostics:
    • User-Agent: Cloudflare-diagnostics
    • Triggered when: Cloudflare Support Engineers perform error checks and by continuous monitoring used to raise intelligent alerts in the Cloudflare dashboard.
  • Custom Hostname validation:
    • User-Agent: Cloudflare Custom Hostname Verification
    • Triggered when: You choose to validate a custom hostname with an HTTP ownership token.