---
title: Robots
---

[Skip to content](#_top)

[API Reference](https://developers.cloudflare.com/api/typescript)

[AI Audit](https://developers.cloudflare.com/api/typescript/resources/ai_audit)

Copy Markdown

Open in **Claude**Open in **ChatGPT**Open in **Cursor**

---

**Copy Markdown****View as Markdown**

# Robots

##### [Get robots.txt rules](https://developers.cloudflare.com/api/typescript/resources/ai_audit/subresources/robots/methods/get)

client.aiAudit.robots.get(RobotGetParams {zone\_id, subdomain } params, RequestOptionsoptions?): [RobotGetResponse](<https://developers.cloudflare.com/api/typescript/resources/ai_audit#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_get_response%20%3E%20(schema)>) {userAgents, sitemaps, status }

GET/zones/{zone\_id}/ai-audit/robots

##### [Bulk get robots.txt rules](https://developers.cloudflare.com/api/typescript/resources/ai_audit/subresources/robots/methods/bulk_get)

client.aiAudit.robots.bulkGet(RobotBulkGetParams {zone\_id, body } params, RequestOptionsoptions?): [RobotBulkGetResponse](<https://developers.cloudflare.com/api/typescript/resources/ai_audit#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_bulk_get_response%20%3E%20(schema)>) {userAgents, sitemaps, status }

POST/zones/{zone\_id}/ai-audit/robots/bulk

##### ModelsExpand Collapse

<details>

<summary>

RobotGetResponse {userAgents, sitemaps, status }

Parsed robots.txt rules for a single domain.

</summary>

<details>

<summary>

userAgents: Record&lt;string, UserAgents&gt;

Map of user-agent string to its parsed rules.

</summary>

allow: Array&lt;string&gt;

List of allowed path patterns.

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_get_response%20%3E%20(schema)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20allow">Link to this property</a>

disallow: Array&lt;string&gt;

List of disallowed path patterns.

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_get_response%20%3E%20(schema)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20disallow">Link to this property</a>

<details>

<summary>

contentSignals?: ContentSignals {ai-input, ai-train, search }

Content signal directives from robots.txt.

</summary>

<details>

<summary>

"ai-input"?: "yes"| "no"

Whether AI input usage is permitted.

</summary>

One of the following:

"yes"

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_get_response%20%3E%20(schema)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20contentSignals%20%3E%20(property)%20ai-input%20%3E%20(member)%200">Link to this property</a>

"no"

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_get_response%20%3E%20(schema)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20contentSignals%20%3E%20(property)%20ai-input%20%3E%20(member)%201">Link to this property</a>

</details>

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_get_response%20%3E%20(schema)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20contentSignals%20%3E%20(property)%20ai-input">Link to this property</a>

<details>

<summary>

"ai-train"?: "yes"| "no"

Whether AI training is permitted.

</summary>

One of the following:

"yes"

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_get_response%20%3E%20(schema)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20contentSignals%20%3E%20(property)%20ai-train%20%3E%20(member)%200">Link to this property</a>

"no"

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_get_response%20%3E%20(schema)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20contentSignals%20%3E%20(property)%20ai-train%20%3E%20(member)%201">Link to this property</a>

</details>

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_get_response%20%3E%20(schema)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20contentSignals%20%3E%20(property)%20ai-train">Link to this property</a>

<details>

<summary>

search?: "yes"| "no"

Whether search indexing is permitted.

</summary>

One of the following:

"yes"

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_get_response%20%3E%20(schema)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20contentSignals%20%3E%20(property)%20search%20%3E%20(member)%200">Link to this property</a>

"no"

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_get_response%20%3E%20(schema)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20contentSignals%20%3E%20(property)%20search%20%3E%20(member)%201">Link to this property</a>

</details>

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_get_response%20%3E%20(schema)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20contentSignals%20%3E%20(property)%20search">Link to this property</a>

</details>

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_get_response%20%3E%20(schema)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20contentSignals">Link to this property</a>

crawlDelay?: number

Crawl delay in seconds.

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_get_response%20%3E%20(schema)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20crawlDelay">Link to this property</a>

</details>

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_get_response%20%3E%20(schema)%20%3E%20(property)%20userAgents">Link to this property</a>

sitemaps?: Array&lt;string&gt;

List of sitemap URLs found in robots.txt.

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_get_response%20%3E%20(schema)%20%3E%20(property)%20sitemaps">Link to this property</a>

status?: number

HTTP status code from fetching the robots.txt file.

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_get_response%20%3E%20(schema)%20%3E%20(property)%20status">Link to this property</a>

</details>

[Link to this property](<#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_get_response%20%3E%20(schema)>)

<details>

<summary>

RobotBulkGetResponse = Record&lt;string, item&gt;

Map of hostname to parsed robots.txt rules.

</summary>

<details>

<summary>

userAgents: Record&lt;string, UserAgents&gt;

Map of user-agent string to its parsed rules.

</summary>

allow: Array&lt;string&gt;

List of allowed path patterns.

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_bulk_get_response%20%3E%20(schema)%20%3E%20(items)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20allow">Link to this property</a>

disallow: Array&lt;string&gt;

List of disallowed path patterns.

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_bulk_get_response%20%3E%20(schema)%20%3E%20(items)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20disallow">Link to this property</a>

<details>

<summary>

contentSignals?: ContentSignals {ai-input, ai-train, search }

Content signal directives from robots.txt.

</summary>

<details>

<summary>

"ai-input"?: "yes"| "no"

Whether AI input usage is permitted.

</summary>

One of the following:

"yes"

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_bulk_get_response%20%3E%20(schema)%20%3E%20(items)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20contentSignals%20%3E%20(property)%20ai-input%20%3E%20(member)%200">Link to this property</a>

"no"

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_bulk_get_response%20%3E%20(schema)%20%3E%20(items)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20contentSignals%20%3E%20(property)%20ai-input%20%3E%20(member)%201">Link to this property</a>

</details>

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_bulk_get_response%20%3E%20(schema)%20%3E%20(items)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20contentSignals%20%3E%20(property)%20ai-input">Link to this property</a>

<details>

<summary>

"ai-train"?: "yes"| "no"

Whether AI training is permitted.

</summary>

One of the following:

"yes"

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_bulk_get_response%20%3E%20(schema)%20%3E%20(items)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20contentSignals%20%3E%20(property)%20ai-train%20%3E%20(member)%200">Link to this property</a>

"no"

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_bulk_get_response%20%3E%20(schema)%20%3E%20(items)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20contentSignals%20%3E%20(property)%20ai-train%20%3E%20(member)%201">Link to this property</a>

</details>

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_bulk_get_response%20%3E%20(schema)%20%3E%20(items)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20contentSignals%20%3E%20(property)%20ai-train">Link to this property</a>

<details>

<summary>

search?: "yes"| "no"

Whether search indexing is permitted.

</summary>

One of the following:

"yes"

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_bulk_get_response%20%3E%20(schema)%20%3E%20(items)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20contentSignals%20%3E%20(property)%20search%20%3E%20(member)%200">Link to this property</a>

"no"

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_bulk_get_response%20%3E%20(schema)%20%3E%20(items)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20contentSignals%20%3E%20(property)%20search%20%3E%20(member)%201">Link to this property</a>

</details>

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_bulk_get_response%20%3E%20(schema)%20%3E%20(items)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20contentSignals%20%3E%20(property)%20search">Link to this property</a>

</details>

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_bulk_get_response%20%3E%20(schema)%20%3E%20(items)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20contentSignals">Link to this property</a>

crawlDelay?: number

Crawl delay in seconds.

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_bulk_get_response%20%3E%20(schema)%20%3E%20(items)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20crawlDelay">Link to this property</a>

</details>

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_bulk_get_response%20%3E%20(schema)%20%3E%20(items)%20%3E%20(property)%20userAgents">Link to this property</a>

sitemaps?: Array&lt;string&gt;

List of sitemap URLs found in robots.txt.

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_bulk_get_response%20%3E%20(schema)%20%3E%20(items)%20%3E%20(property)%20sitemaps">Link to this property</a>

status?: number

HTTP status code from fetching the robots.txt file.

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_bulk_get_response%20%3E%20(schema)%20%3E%20(items)%20%3E%20(property)%20status">Link to this property</a>

</details>

[Link to this property](<#(resource)%20ai_audit.robots%20%3E%20(model)%20robot_bulk_get_response%20%3E%20(schema)>)