---
title: Get robots.txt rules
---

[Skip to content](#_top)

[API Reference](https://developers.cloudflare.com/api/go)

[AI Audit](https://developers.cloudflare.com/api/go/resources/ai_audit)

[Robots](https://developers.cloudflare.com/api/go/resources/ai_audit/subresources/robots)

Copy Markdown

Open in **Claude**Open in **ChatGPT**Open in **Cursor**

---

**Copy Markdown****View as Markdown**

# Get robots.txt rules

client.AIAudit.Robots.Get(ctx, params) (\*[RobotGetResponse](<https://developers.cloudflare.com/api/go/resources/ai_audit#(resource)%20ai_audit.robots%20%3E%20(model)%20RobotGetResponse%20%3E%20(schema)>), error)

GET/zones/{zone\_id}/ai-audit/robots

Fetches and parses the robots.txt file for a zone or a specific subdomain within the zone. Returns parsed user-agent rules, content signals, and sitemaps.

##### Security

<details>

<summary>API Token</summary>



The preferred authorization scheme for interacting with the Cloudflare API. <a href="https://developers.cloudflare.com/fundamentals/api/get-started/create-token/">Create a token</a>.

**Example:**<code>Authorization: Bearer Sn3lZJTBX6kkg7OdcBUAxOO963GEIyGQqnFTOFYY</code>

</details>

<details>

<summary>API Email + API Key</summary>



The previous authorization scheme for interacting with the Cloudflare API, used in conjunction with a Global API key.

**Example:**<code>X-Auth-Email: user@example.com</code>

The previous authorization scheme for interacting with the Cloudflare API. When possible, use API tokens instead of Global API keys.

**Example:**<code>X-Auth-Key: 144c9defac04969c7bfad8efaa8ea194</code>

</details>

##### Accepted Permissions (at least one required)

`Zone Settings Write``Zone Settings Read`

##### ParametersExpand Collapse

<details>

<summary>

params RobotGetParams

</summary>

ZoneID param.Field\[string]

Path param: Identifier of the zone.

<a href="#(resource)%20ai_audit.robots%20%3E%20(method)%20get%20%3E%20(params)%20default%20%3E%20(param)%20zone_id">Link to this property</a>

Subdomain param.Field\[string]Optional

Query param: Optional subdomain to fetch robots.txt for. If omitted, fetches robots.txt for the zone apex domain.

<a href="#(resource)%20ai_audit.robots%20%3E%20(method)%20get%20%3E%20(params)%20default%20%3E%20(param)%20subdomain">Link to this property</a>

</details>

[Link to this property](<#(resource)%20ai_audit.robots%20%3E%20(method)%20get%20%3E%20(params)%20default>)

##### ReturnsExpand Collapse

<details>

<summary>

type RobotGetResponse struct{…}

Parsed robots.txt rules for a single domain.

</summary>

<details>

<summary>

UserAgents map\[string, RobotGetResponseUserAgent]

Map of user-agent string to its parsed rules.

</summary>

Allow \[]string

List of allowed path patterns.

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20RobotGetResponse%20%3E%20(schema)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20allow">Link to this property</a>

Disallow \[]string

List of disallowed path patterns.

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20RobotGetResponse%20%3E%20(schema)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20disallow">Link to this property</a>

<details>

<summary>

ContentSignals RobotGetResponseUserAgentsContentSignalsOptional

Content signal directives from robots.txt.

</summary>

<details>

<summary>

AIInput RobotGetResponseUserAgentsContentSignalsAIInputOptional

Whether AI input usage is permitted.

</summary>

One of the following:

const RobotGetResponseUserAgentsContentSignalsAIInputYes RobotGetResponseUserAgentsContentSignalsAIInput = "yes"

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20RobotGetResponse%20%3E%20(schema)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20contentSignals%20%3E%20(property)%20ai-input%20%3E%20(member)%200">Link to this property</a>

const RobotGetResponseUserAgentsContentSignalsAIInputNo RobotGetResponseUserAgentsContentSignalsAIInput = "no"

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20RobotGetResponse%20%3E%20(schema)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20contentSignals%20%3E%20(property)%20ai-input%20%3E%20(member)%201">Link to this property</a>

</details>

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20RobotGetResponse%20%3E%20(schema)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20contentSignals%20%3E%20(property)%20ai-input">Link to this property</a>

<details>

<summary>

AITrain RobotGetResponseUserAgentsContentSignalsAITrainOptional

Whether AI training is permitted.

</summary>

One of the following:

const RobotGetResponseUserAgentsContentSignalsAITrainYes RobotGetResponseUserAgentsContentSignalsAITrain = "yes"

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20RobotGetResponse%20%3E%20(schema)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20contentSignals%20%3E%20(property)%20ai-train%20%3E%20(member)%200">Link to this property</a>

const RobotGetResponseUserAgentsContentSignalsAITrainNo RobotGetResponseUserAgentsContentSignalsAITrain = "no"

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20RobotGetResponse%20%3E%20(schema)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20contentSignals%20%3E%20(property)%20ai-train%20%3E%20(member)%201">Link to this property</a>

</details>

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20RobotGetResponse%20%3E%20(schema)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20contentSignals%20%3E%20(property)%20ai-train">Link to this property</a>

<details>

<summary>

Search RobotGetResponseUserAgentsContentSignalsSearchOptional

Whether search indexing is permitted.

</summary>

One of the following:

const RobotGetResponseUserAgentsContentSignalsSearchYes RobotGetResponseUserAgentsContentSignalsSearch = "yes"

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20RobotGetResponse%20%3E%20(schema)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20contentSignals%20%3E%20(property)%20search%20%3E%20(member)%200">Link to this property</a>

const RobotGetResponseUserAgentsContentSignalsSearchNo RobotGetResponseUserAgentsContentSignalsSearch = "no"

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20RobotGetResponse%20%3E%20(schema)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20contentSignals%20%3E%20(property)%20search%20%3E%20(member)%201">Link to this property</a>

</details>

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20RobotGetResponse%20%3E%20(schema)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20contentSignals%20%3E%20(property)%20search">Link to this property</a>

</details>

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20RobotGetResponse%20%3E%20(schema)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20contentSignals">Link to this property</a>

CrawlDelay float64Optional

Crawl delay in seconds.

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20RobotGetResponse%20%3E%20(schema)%20%3E%20(property)%20userAgents%20%3E%20(items)%20%3E%20(property)%20crawlDelay">Link to this property</a>

</details>

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20RobotGetResponse%20%3E%20(schema)%20%3E%20(property)%20userAgents">Link to this property</a>

Sitemaps \[]stringOptional

List of sitemap URLs found in robots.txt.

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20RobotGetResponse%20%3E%20(schema)%20%3E%20(property)%20sitemaps">Link to this property</a>

Status int64Optional

HTTP status code from fetching the robots.txt file.

<a href="#(resource)%20ai_audit.robots%20%3E%20(model)%20RobotGetResponse%20%3E%20(schema)%20%3E%20(property)%20status">Link to this property</a>

</details>

[Link to this property](<#(resource)%20ai_audit.robots%20%3E%20(model)%20RobotGetResponse%20%3E%20(schema)>)

### Get robots.txt rules

Go

HTTPTypeScriptPythonGoTerraform

```
package main

import (
  "context"
  "fmt"

  "github.com/cloudflare/cloudflare-go"
  "github.com/cloudflare/cloudflare-go/ai_audit"
  "github.com/cloudflare/cloudflare-go/option"
)

func main() {
  client := cloudflare.NewClient(
    option.WithAPIToken("Sn3lZJTBX6kkg7OdcBUAxOO963GEIyGQqnFTOFYY"),
  )
  robot, err := client.AIAudit.Robots.Get(context.TODO(), ai_audit.RobotGetParams{
    ZoneID: cloudflare.F("zone_id"),
  })
  if err != nil {
    panic(err.Error())
  }
  fmt.Printf("%+v\n", robot.UserAgents)
}
```

200 example

```
{
  "errors": [],
  "messages": [
    {
      "code": 1000,
      "message": "message",
      "documentation_url": "documentation_url",
      "source": {
        "pointer": "pointer"
      }
    }
  ],
  "success": true,
  "result": {
    "userAgents": {
      "foo": {
        "allow": [
          "string"
        ],
        "disallow": [
          "string"
        ],
        "contentSignals": {
          "ai-input": "yes",
          "ai-train": "yes",
          "search": "yes"
        },
        "crawlDelay": 0
      }
    },
    "sitemaps": [
      "string"
    ],
    "status": 0
  }
}
```

##### Returns Examples

200 example

```
{
  "errors": [],
  "messages": [
    {
      "code": 1000,
      "message": "message",
      "documentation_url": "documentation_url",
      "source": {
        "pointer": "pointer"
      }
    }
  ],
  "success": true,
  "result": {
    "userAgents": {
      "foo": {
        "allow": [
          "string"
        ],
        "disallow": [
          "string"
        ],
        "contentSignals": {
          "ai-input": "yes",
          "ai-train": "yes",
          "search": "yes"
        },
        "crawlDelay": 0
      }
    },
    "sitemaps": [
      "string"
    ],
    "status": 0
  }
}
```