Skip to content
Cloudflare Docs

Direct AI crawlers with managed robots.txt

Protect your website or application from AI crawlers by implementing a robots.txt file on your domain to direct AI bot operators on what content they can and cannot scrape for AI model training.

Cloudflare's managed robots.txt explicitly disallows known bots engaged in scraping for AI purposes and AI agent activity.

AI bots are expected to follow the robots.txt directives. Otherwise, they risk getting banned.

Compatibility with existing robots.txt files

Cloudflare will independently check whether your website has an existing robots.txt file.

  • If your website already has a robots.txt file — verified by a HTTP 200 response — Cloudflare will prepend our managed robots.txt before your existing robots.txt, combining both into a single response.
  • If your website does not have a robots.txt file, it will receive Cloudflare's managed block directives.

To implement a robots.txt file on your domain based on your plan:

  1. Log in to the Cloudflare dashboard ↗ and select your account and domain.
  2. Go to Security > Bots.
  3. Select Configure Bot Fight Mode.
  4. On Manage bot traffic with robots.txt, select the toggle.

Availability

Managed robots.txt for AI crawlers is available on all plans.