--- title: robots.txt setting description: Manage a robots.txt file to direct AI bot operators on content scraping permissions. image: https://developers.cloudflare.com/core-services-preview.png --- > Documentation Index > Fetch the complete documentation index at: https://developers.cloudflare.com/bots/llms.txt > Use this file to discover all available pages before exploring further. [Skip to content](#%5Ftop) # robots.txt setting AI companies use crawlers to collect website content for training language models, generating search answers, and other purposes. A `robots.txt` file at the root of your domain tells these crawlers which content they should or should not access. When you turn on the managed `robots.txt` setting, Cloudflare generates and maintains a `robots.txt` file that instructs known AI crawlers to stay away from your content. `robots.txt` compliance is voluntary. The file expresses your preferences, but it does not prevent crawlers from accessing your content at a technical level. Some crawler operators may disregard your `robots.txt` directives (instructions like `Disallow: /`) and crawl your content regardless. Note If you want to enforce crawl blocking rather than request it, use [AI Crawl Control](https://developers.cloudflare.com/ai-crawl-control/features/manage-ai-crawlers/). You can also use both features together — `robots.txt` to express your preferences and AI Crawl Control to enforce them. ## Compatibility with existing `robots.txt` files Cloudflare detects whether your origin server already has a `robots.txt` file and adjusts accordingly — either merging with your existing file or creating one from scratch. ### Existing robots.txt file If your website already has a `robots.txt` file — verified by an HTTP `200` response — Cloudflare will prepend our managed `robots.txt` before your existing `robots.txt`, combining both into a single response. For example, without this feature enabled, the `robots.txt` content of `crawlstop.com` would be: Feature not enabled ``` User-agent: *Disallow: /lpDisallow: /feedbackDisallow: /langtest Sitemap: https://www.crawlstop.com/sitemap.xml ``` With the managed `robots.txt` enabled, Cloudflare will prepend our managed content before your original content, resulting in what you can view at [https://www.crawlstop.com/robots.txt ↗](https://www.crawlstop.com/robots.txt). Feature enabled ``` # As a condition of accessing this website, you agree to abide by the# following content signals: # (a) If a content-signal = yes, you may collect content for the# corresponding use.# (b) If a content-signal = no, you may not collect content for the# corresponding use.# (c) If the website operator does not include a content signal for a# corresponding use, the website operator neither grants nor restricts# permission via content signal with respect to the corresponding use. # The content signals and their meanings are: # search: building a search index and providing search results (e.g., returning# hyperlinks and short excerpts from your website's contents). Search# does not include providing AI-generated search summaries.# ai-input: inputting content into one or more AI models (e.g., retrieval# augmented generation, grounding, or other real-time taking of# content for generative AI search answers).# ai-train: training or fine-tuning AI models. # ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF# RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT# AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET. # BEGIN Cloudflare Managed content User-Agent: *Content-signal: search=yes, ai-train=noAllow: / User-agent: AmazonbotDisallow: / User-agent: Applebot-ExtendedDisallow: / User-agent: BytespiderDisallow: / User-agent: CCBotDisallow: / User-agent: ClaudeBotDisallow: / User-agent: Google-ExtendedDisallow: / User-agent: GPTBotDisallow: / User-agent: meta-externalagentDisallow: / # END Cloudflare Managed ContentUser-agent: *Disallow: /lpDisallow: /feedbackDisallow: /langtest Sitemap: https://www.crawlstop.com/sitemap.xml ``` ### No robots.txt file If your website does not have a `robots.txt` file, Cloudflare creates a new file with managed `Disallow` rules for known AI crawlers and serves it for you. ## Implementation To implement a `robots.txt` file on your domain: * [ New dashboard ](#tab-panel-7138) * [ Old dashboard ](#tab-panel-7139) 1. In the Cloudflare dashboard, go to the **Security Settings** page. [ Go to **Settings** ](https://dash.cloudflare.com/?to=/:account/:zone/security/settings) 2. Filter by **Bot traffic**. 3. Go to **Instruct AI bot traffic with robots.txt**. 4. Turn on **Instruct AI bot traffic with robots.txt**. 1. Log in to the [Cloudflare dashboard ↗](https://dash.cloudflare.com/), and select your account and domain. 2. Go to **Security** \> **Bots**. 3. Select **Configure Bot Fight Mode**. 4. Turn on **Instruct bot traffic with robots.txt**. ## Content Signals Policy Content Signals are a set of machine-readable directives in a `robots.txt` file that categorize how crawlers may use your content. The three categories are `search` (building a search index), `ai-input` (feeding content into AI models for real-time answers), and `ai-train` (training or fine-tuning AI models). Domains on the Free plan that do not have their own `robots.txt` file and do not use the managed `robots.txt` feature will display the Content Signals Policy when a crawler requests the `robots.txt` file for your domain. The Content Signals Policy defines these categories but does not express any specific preferences about your content. To set preferences (for example, `ai-train=no`), turn on the managed `robots.txt` feature. Content Signals Policy ``` # As a condition of accessing this website, you agree to abide by the# following content signals: # (a) If a content-signal = yes, you may collect content for the# corresponding use.# (b) If a content-signal = no, you may not collect content for the# corresponding use.# (c) If the website operator does not include a content signal for a# corresponding use, the website operator neither grants nor restricts# permission via content signal with respect to the corresponding use. # The content signals and their meanings are: # search: building a search index and providing search results (e.g., returning# hyperlinks and short excerpts from your website's contents). Search# does not include providing AI-generated search summaries.# ai-input: inputting content into one or more AI models (e.g., retrieval# augmented generation, grounding, or other real-time taking of# content for generative AI search answers).# ai-train: training or fine-tuning AI models. # ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF# RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT# AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET. ``` Cloudflare's Content Signals Policy is included by default in the `robots.txt` file when you turn on **robots.txt setting**. If you would like to opt out of displaying the policy in your `robots.txt` file, you can uncheck **Display Content Signals Policy** under **Control AI Crawlers** in your zone's overview. [ Go to **Overview** ](https://dash.cloudflare.com/?to=/:account/:zone/) Alternatively, you can use [Security Settings](#implementation). Warning Google Search Console may occasionally report `Syntax not understood` for Content Signals and newer directives in the `robots.txt` standard. However, we have observed no impact on crawling rates or SEO as a result of these reports. ## Availability Managed `robots.txt` for AI crawlers is available on all plans. ```json {"@context":"https://schema.org","@type":"TechArticle","@id":"https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/#page","headline":"robots.txt setting · Cloudflare bot solutions docs","description":"Manage a robots.txt file to direct AI bot operators on content scraping permissions.","url":"https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/","inLanguage":"en","image":"https://developers.cloudflare.com/core-services-preview.png","dateModified":"2026-05-05","publisher":{"@type":"Organization","name":"Cloudflare","url":"https://www.cloudflare.com/"},"isPartOf":{"@type":"WebSite","@id":"https://developers.cloudflare.com/#website","name":"Cloudflare Docs","url":"https://developers.cloudflare.com/"},"keywords":["AI"]} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/bots/","name":"Bots"}},{"@type":"ListItem","position":3,"item":{"@id":"/bots/additional-configurations/","name":"Additional configurations"}},{"@type":"ListItem","position":4,"item":{"@id":"/bots/additional-configurations/managed-robots-txt/","name":"robots.txt setting"}}]} ```