---
title: Bulk get robots.txt rules
---

[Skip to content](#_top)

[API Reference](https://developers.cloudflare.com/api/python)

[AI Audit](https://developers.cloudflare.com/api/python/resources/ai_audit)

[Robots](https://developers.cloudflare.com/api/python/resources/ai_audit/subresources/robots)

Copy Markdown

Open in **Claude**Open in **ChatGPT**Open in **Cursor**

---

**Copy Markdown****View as Markdown**

# Bulk get robots.txt rules

ai\_audit.robots.bulk\_get(RobotBulkGetParams\*\*kwargs) -> [RobotBulkGetResponse](<https://developers.cloudflare.com/api/python/resources/ai_audit#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_bulk_get_response%20%3E%20(schema)>)

POST/zones/{zone\_id}/ai-audit/robots/bulk

Fetches and parses robots.txt files for multiple domains within a zone in a single request. Each domain must belong to the specified zone. Results are keyed by hostname.

##### Security

<details>

<summary>API Token</summary>



The preferred authorization scheme for interacting with the Cloudflare API. <a href="https://developers.cloudflare.com/fundamentals/api/get-started/create-token/">Create a token</a>.

**Example:**<code>Authorization: Bearer Sn3lZJTBX6kkg7OdcBUAxOO963GEIyGQqnFTOFYY</code>

</details>

<details>

<summary>API Email + API Key</summary>



The previous authorization scheme for interacting with the Cloudflare API, used in conjunction with a Global API key.

**Example:**<code>X-Auth-Email: user@example.com</code>

The previous authorization scheme for interacting with the Cloudflare API. When possible, use API tokens instead of Global API keys.

**Example:**<code>X-Auth-Key: 144c9defac04969c7bfad8efaa8ea194</code>

</details>

##### Accepted Permissions (at least one required)

`Zone Settings Write``Zone Settings Read`

##### ParametersExpand Collapse

zone\_id: str

[Link to this property](<#(resource)%20ai_audit.robots%20%3E%20(method)%20bulk_get%20%3E%20(params)%20default%20%3E%20(param)%20zone_id%20%3E%20(schema)>)

body: Sequence\[str]

Array of domain hostnames to fetch robots.txt for. Each domain must end with the zone name. Maximum 25 domains per request.

[Link to this property](<#(resource)%20ai_audit.robots%20%3E%20(method)%20bulk_get%20%3E%20(params)%20default%20%3E%20(param)%20body%20%3E%20(schema)>)

##### ReturnsExpand Collapse

<details>

<summary>

Dict\[str, RobotBulkGetResponseItem]

Map of hostname to parsed robots.txt rules.

</summary>

<details>

<summary>

user\_agents: Dict\[str, RobotBulkGetResponseItemUserAgents]

Map of user-agent string to its parsed rules.

</summary>

allow: List\[str]

List of allowed path patterns.

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_bulk_get_response%20%3E%20(schema)%20%3E%20(items)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20allow">Link to this property</a>

disallow: List\[str]

List of disallowed path patterns.

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_bulk_get_response%20%3E%20(schema)%20%3E%20(items)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20disallow">Link to this property</a>

<details>

<summary>

content\_signals: Optional\[RobotBulkGetResponseItemUserAgentsContentSignals]

Content signal directives from robots.txt.

</summary>

<details>

<summary>

ai\_input: Optional\[Literal\["yes", "no"]]

Whether AI input usage is permitted.

</summary>

One of the following:

"yes"

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_bulk_get_response%20%3E%20(schema)%20%3E%20(items)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20contentSignals%20%3E%20(property)%20ai-input%20%3E%20(member)%200">Link to this property</a>

"no"

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_bulk_get_response%20%3E%20(schema)%20%3E%20(items)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20contentSignals%20%3E%20(property)%20ai-input%20%3E%20(member)%201">Link to this property</a>

</details>

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_bulk_get_response%20%3E%20(schema)%20%3E%20(items)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20contentSignals%20%3E%20(property)%20ai-input">Link to this property</a>

<details>

<summary>

ai\_train: Optional\[Literal\["yes", "no"]]

Whether AI training is permitted.

</summary>

One of the following:

"yes"

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_bulk_get_response%20%3E%20(schema)%20%3E%20(items)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20contentSignals%20%3E%20(property)%20ai-train%20%3E%20(member)%200">Link to this property</a>

"no"

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_bulk_get_response%20%3E%20(schema)%20%3E%20(items)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20contentSignals%20%3E%20(property)%20ai-train%20%3E%20(member)%201">Link to this property</a>

</details>

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_bulk_get_response%20%3E%20(schema)%20%3E%20(items)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20contentSignals%20%3E%20(property)%20ai-train">Link to this property</a>

<details>

<summary>

search: Optional\[Literal\["yes", "no"]]

Whether search indexing is permitted.

</summary>

One of the following:

"yes"

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_bulk_get_response%20%3E%20(schema)%20%3E%20(items)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20contentSignals%20%3E%20(property)%20search%20%3E%20(member)%200">Link to this property</a>

"no"

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_bulk_get_response%20%3E%20(schema)%20%3E%20(items)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20contentSignals%20%3E%20(property)%20search%20%3E%20(member)%201">Link to this property</a>

</details>

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_bulk_get_response%20%3E%20(schema)%20%3E%20(items)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20contentSignals%20%3E%20(property)%20search">Link to this property</a>

</details>

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_bulk_get_response%20%3E%20(schema)%20%3E%20(items)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20contentSignals">Link to this property</a>

crawl\_delay: Optional\[float]

Crawl delay in seconds.

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_bulk_get_response%20%3E%20(schema)%20%3E%20(items)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20crawlDelay">Link to this property</a>

</details>

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_bulk_get_response%20%3E%20(schema)%20%3E%20(items)%20%3E%20(property)%20userAgents">Link to this property</a>

sitemaps: Optional\[List\[str]]

List of sitemap URLs found in robots.txt.

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_bulk_get_response%20%3E%20(schema)%20%3E%20(items)%20%3E%20(property)%20sitemaps">Link to this property</a>

status: Optional\[int]

HTTP status code from fetching the robots.txt file.

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_bulk_get_response%20%3E%20(schema)%20%3E%20(items)%20%3E%20(property)%20status">Link to this property</a>

</details>

[Link to this property](<#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_bulk_get_response%20%3E%20(schema)>)

### Bulk get robots.txt rules

Python

HTTPTypeScriptPythonGoTerraform

```
import os
from cloudflare import Cloudflare

client = Cloudflare(
    api_token=os.environ.get("CLOUDFLARE_API_TOKEN"),  # This is the default and can be omitted
)
response = client.ai_audit.robots.bulk_get(
    zone_id="zone_id",
    body=["example.com", "blog.example.com"],
)
print(response)
```

200 example

```
{
  "errors": [],
  "messages": [
    {
      "code": 1000,
      "message": "message",
      "documentation_url": "documentation_url",
      "source": {
        "pointer": "pointer"
      }
    }
  ],
  "success": true,
  "result": {
    "foo": {
      "userAgents": {
        "foo": {
          "allow": [
            "string"
          ],
          "disallow": [
            "string"
          ],
          "contentSignals": {
            "ai-input": "yes",
            "ai-train": "yes",
            "search": "yes"
          },
          "crawlDelay": 0
        }
      },
      "sitemaps": [
        "string"
      ],
      "status": 0
    }
  }
}
```

##### Returns Examples

200 example

```
{
  "errors": [],
  "messages": [
    {
      "code": 1000,
      "message": "message",
      "documentation_url": "documentation_url",
      "source": {
        "pointer": "pointer"
      }
    }
  ],
  "success": true,
  "result": {
    "foo": {
      "userAgents": {
        "foo": {
          "allow": [
            "string"
          ],
          "disallow": [
            "string"
          ],
          "contentSignals": {
            "ai-input": "yes",
            "ai-train": "yes",
            "search": "yes"
          },
          "crawlDelay": 0
        }
      },
      "sitemaps": [
        "string"
      ],
      "status": 0
    }
  }
}
```