---
title: Get robots.txt rules
---
[Skip to content](#_top)
[API Reference](https://developers.cloudflare.com/api)
[AI Audit](https://developers.cloudflare.com/api/resources/ai_audit)
[Robots](https://developers.cloudflare.com/api/resources/ai_audit/subresources/robots)
Copy Markdown
Open in **Claude**Open in **ChatGPT**Open in **Cursor**
---
**Copy Markdown****View as Markdown**
# Get robots.txt rules
GET/zones/{zone\_id}/ai-audit/robots
Fetches and parses the robots.txt file for a zone or a specific subdomain within the zone. Returns parsed user-agent rules, content signals, and sitemaps.
##### Security
API Token
The preferred authorization scheme for interacting with the Cloudflare API. Create a token.
**Example:**Authorization: Bearer Sn3lZJTBX6kkg7OdcBUAxOO963GEIyGQqnFTOFYY
API Email + API Key
The previous authorization scheme for interacting with the Cloudflare API, used in conjunction with a Global API key.
**Example:**X-Auth-Email: user@example.com
The previous authorization scheme for interacting with the Cloudflare API. When possible, use API tokens instead of Global API keys.
**Example:**X-Auth-Key: 144c9defac04969c7bfad8efaa8ea194
##### Accepted Permissions (at least one required)
`Zone Settings Write``Zone Settings Read`
##### P ath ParametersExpand Collapse
zone\_id: string
[Link to this property](<#(resource)%20ai_audit.robots%20%3E%20(method)%20get%20%3E%20(params)%20default%20%3E%20(param)%20zone_id%20%3E%20(schema)>)
##### Q uery ParametersExpand Collapse
subdomain: optional string
Optional subdomain to fetch robots.txt for. If omitted, fetches robots.txt for the zone apex domain.
[Link to this property](<#(resource)%20ai_audit.robots%20%3E%20(method)%20get%20%3E%20(params)%20default%20%3E%20(param)%20subdomain%20%3E%20(schema)>)
##### ReturnsExpand Collapse
errors: array of ResponseInfo { code, message, documentation\_url, source }
code: number
minimum1000
Link to this property
message: string
Link to this property
documentation\_url: optional string
Link to this property
source: optional object {pointer }
pointer: optional string
Link to this property
Link to this property
[Link to this property](<#(resource)%20ai_audit.robots%20%3E%20(method)%20get%20%3E%20(network%20schema)%20%3E%20(property)%20errors>)
messages: array of ResponseInfo { code, message, documentation\_url, source }
code: number
minimum1000
Link to this property
message: string
Link to this property
documentation\_url: optional string
Link to this property
source: optional object {pointer }
pointer: optional string
Link to this property
Link to this property
[Link to this property](<#(resource)%20ai_audit.robots%20%3E%20(method)%20get%20%3E%20(network%20schema)%20%3E%20(property)%20messages>)
success: boolean
[Link to this property](<#(resource)%20ai_audit.robots%20%3E%20(method)%20get%20%3E%20(network%20schema)%20%3E%20(property)%20success>)
result: optional object {userAgents, sitemaps, status }
Parsed robots.txt rules for a single domain.
userAgents: map\[object {allow, disallow, contentSignals, crawlDelay } ]
Map of user-agent string to its parsed rules.
allow: array of string
List of allowed path patterns.
Link to this property
disallow: array of string
List of disallowed path patterns.
Link to this property
contentSignals: optional object {"ai-input", "ai-train", search }
Content signal directives from robots.txt.
"ai-input": optional "yes"or "no"
Whether AI input usage is permitted.
One of the following:
"yes"
Link to this property
"no"
Link to this property
Link to this property
"ai-train": optional "yes"or "no"
Whether AI training is permitted.
One of the following:
"yes"
Link to this property
"no"
Link to this property
Link to this property
search: optional "yes"or "no"
Whether search indexing is permitted.
One of the following:
"yes"
Link to this property
"no"
Link to this property
Link to this property
Link to this property
crawlDelay: optional number
Crawl delay in seconds.
Link to this property
Link to this property
sitemaps: optional array of string
List of sitemap URLs found in robots.txt.
Link to this property
status: optional number
HTTP status code from fetching the robots.txt file.
Link to this property
[Link to this property](<#(resource)%20ai_audit.robots%20%3E%20(method)%20get%20%3E%20(network%20schema)%20%3E%20(property)%20result>)
### Get robots.txt rules
HTTP
HTTPTypeScriptPythonGoTerraform
```
curl https://api.cloudflare.com/client/v4/zones/$ZONE_ID/ai-audit/robots \
-H "Authorization: Bearer $CLOUDFLARE_API_TOKEN"
```
200 example
```
{
"errors": [],
"messages": [
{
"code": 1000,
"message": "message",
"documentation_url": "documentation_url",
"source": {
"pointer": "pointer"
}
}
],
"success": true,
"result": {
"userAgents": {
"foo": {
"allow": [
"string"
],
"disallow": [
"string"
],
"contentSignals": {
"ai-input": "yes",
"ai-train": "yes",
"search": "yes"
},
"crawlDelay": 0
}
},
"sitemaps": [
"string"
],
"status": 0
}
}
```
##### Returns Examples
200 example
```
{
"errors": [],
"messages": [
{
"code": 1000,
"message": "message",
"documentation_url": "documentation_url",
"source": {
"pointer": "pointer"
}
}
],
"success": true,
"result": {
"userAgents": {
"foo": {
"allow": [
"string"
],
"disallow": [
"string"
],
"contentSignals": {
"ai-input": "yes",
"ai-train": "yes",
"search": "yes"
},
"crawlDelay": 0
}
},
"sitemaps": [
"string"
],
"status": 0
}
}
```