---
title: Crawl websites.
---
[Skip to content](#_top)
[API Reference](https://developers.cloudflare.com/api)
[Browser Rendering](https://developers.cloudflare.com/api/resources/browser_rendering)
[Crawl](https://developers.cloudflare.com/api/resources/browser_rendering/subresources/crawl)
Copy Markdown
Open in **Claude**Open in **ChatGPT**Open in **Cursor**
---
**Copy Markdown****View as Markdown**
# Crawl websites.
POST/accounts/{account\_id}/browser-rendering/crawl
Starts a crawl job for the provided URL and its children. Check available options like `gotoOptions` and `waitFor*` to control page load behaviour.
##### Security
API Token
The preferred authorization scheme for interacting with the Cloudflare API. Create a token.
**Example:**Authorization: Bearer Sn3lZJTBX6kkg7OdcBUAxOO963GEIyGQqnFTOFYY
API Email + API Key
The previous authorization scheme for interacting with the Cloudflare API, used in conjunction with a Global API key.
**Example:**X-Auth-Email: user@example.com
The previous authorization scheme for interacting with the Cloudflare API. When possible, use API tokens instead of Global API keys.
**Example:**X-Auth-Key: 144c9defac04969c7bfad8efaa8ea194
##### Accepted Permissions (at least one required)
`Browser Rendering Write`
##### P ath ParametersExpand Collapse
account\_id: string
Account ID.
[Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(method)%20create%20%3E%20(params)%200%20%3E%20(param)%20account_id%20%3E%20(schema)>)
##### Q uery ParametersExpand Collapse
cacheTTL: optional number
Cache TTL default is 5s. Set to 0 to disable.
maximum86400
minimum0
[Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(method)%20create%20%3E%20(params)%200%20%3E%20(param)%20cacheTTL%20%3E%20(schema)>)
##### Body ParametersJSONExpand Collapse
body: object {url, actionTimeout, addScriptTag, 27 more } or object {render, url, contentUse, 9 more }
One of the following:
object {url, actionTimeout, addScriptTag, 27 more }
url: string
URL to navigate to, eg. https://example.com.
formaturi
Link to this property
actionTimeout: optional number
The maximum duration allowed for the browser action to complete after the page has loaded (such as taking screenshots, extracting content, or generating PDFs). If this time limit is exceeded, the action stops and returns a timeout error.
maximum120000
Link to this property
addScriptTag: optional array of object {id, content, type, url }
Adds a script element into the page with the desired URL or content.
id: optional string
Link to this property
content: optional string
Link to this property
type: optional string
Link to this property
url: optional string
formaturi
Link to this property
Link to this property
addStyleTag: optional array of object {content, url }
Adds a <link rel="stylesheet"> tag into the page with the desired URL or a <style type="text/css"> tag with the content.
content: optional string
Link to this property
url: optional string
formaturi
Link to this property
Link to this property
allowRequestPattern: optional array of string
Only allow requests that match the provided regex patterns, eg. ’/^.\*.(css)’. Reject rules are applied first.
Link to this property
allowResourceTypes: optional array of "document"or "stylesheet"or "image"or 15 more
Only allow requests that match the provided resource types, eg. ‘image’ or ‘script’. Reject rules are applied first.
One of the following:
"document"
Link to this property
"stylesheet"
Link to this property
"image"
Link to this property
"media"
Link to this property
"font"
Link to this property
"script"
Link to this property
"texttrack"
Link to this property
"xhr"
Link to this property
"fetch"
Link to this property
"prefetch"
Link to this property
"eventsource"
Link to this property
"websocket"
Link to this property
"manifest"
Link to this property
"signedexchange"
Link to this property
"ping"
Link to this property
"cspviolationreport"
Link to this property
"preflight"
Link to this property
"other"
Link to this property
Link to this property
authenticate: optional object {password, username }
Provide credentials for HTTP authentication.
password: string
minLength1
Link to this property
username: string
minLength1
Link to this property
Link to this property
bestAttempt: optional boolean
Attempt to proceed when ‘awaited’ events fail or timeout.
Link to this property
browser: optional "kitesurf"
Rendering backend for this crawl. Set to kitesurf to render pages with Kitesurf (beta). Only valid when render is true.
Link to this property
contentUse: optional "reference"or "full"
Intended content use level to respect the use Content-Signal directive in robots.txt. Levels (least to most permissive): ‘reference’, ‘full’. A URL is disallowed when the publisher’s declared use level is lower than this intent. Learn more: https://contentsignals.org/. Default: ‘full’.
One of the following:
"reference"
Link to this property
"full"
Link to this property
Link to this property
cookies: optional array of object {name, value, domain, 11 more }
Check options.
name: string
Cookie name.
Link to this property
value: string
Link to this property
domain: optional string
Link to this property
expires: optional number
Link to this property
httpOnly: optional boolean
Link to this property
partitionKey: optional string
Link to this property
path: optional string
Link to this property
priority: optional "Low"or "Medium"or "High"
One of the following:
"Low"
Link to this property
"Medium"
Link to this property
"High"
Link to this property
Link to this property
sameParty: optional boolean
Link to this property
sameSite: optional "Strict"or "Lax"or "None"
One of the following:
"Strict"
Link to this property
"Lax"
Link to this property
"None"
Link to this property
Link to this property
secure: optional boolean
Link to this property
sourcePort: optional number
Link to this property
sourceScheme: optional "Unset"or "NonSecure"or "Secure"
One of the following:
"Unset"
Link to this property
"NonSecure"
Link to this property
"Secure"
Link to this property
Link to this property
url: optional string
Link to this property
Link to this property
crawlPurposes: optional array of "search"or "ai-input"or "ai-train"
List of crawl purposes to respect Content-Signal directives in robots.txt. Allowed values: ‘search’, ‘ai-input’, ‘ai-train’. Learn more: https://contentsignals.org/. Default: \[‘search’, ‘ai-input’, ‘ai-train’].
One of the following:
"search"
Link to this property
"ai-input"
Link to this property
"ai-train"
Link to this property
Link to this property
depth: optional number
Maximum number of levels deep the crawler will traverse from the starting URL.
maximum100000
minimum1
Link to this property
emulateMediaType: optional string
Link to this property
formats: optional array of "html"or "markdown"or "json"
Formats to return. Default is html.
One of the following:
"html"
Link to this property
"markdown"
Link to this property
"json"
Link to this property
Link to this property
gotoOptions: optional object {referer, referrerPolicy, timeout, waitUntil }
Check options.
referer: optional string
Link to this property
referrerPolicy: optional string
Link to this property
timeout: optional number
maximum60000
Link to this property
waitUntil: optional "load"or "domcontentloaded"or "networkidle0"or "networkidle2"or array of "load"or "domcontentloaded"or "networkidle0"or "networkidle2"
One of the following:
"load"or "domcontentloaded"or "networkidle0"or "networkidle2"
One of the following:
"load"
Link to this property
"domcontentloaded"
Link to this property
"networkidle0"
Link to this property
"networkidle2"
Link to this property
Link to this property
array of "load"or "domcontentloaded"or "networkidle0"or "networkidle2"
One of the following:
"load"
Link to this property
"domcontentloaded"
Link to this property
"networkidle0"
Link to this property
"networkidle2"
Link to this property
Link to this property
Link to this property
Link to this property
jsonOptions: optional object {custom\_ai, prompt, response\_format }
Options for JSON extraction.
custom\_ai: optional array of object {model, authorization }
Optional list of custom AI models to use for the request. The models will be tried in the order provided, and in case a model returns an error, the next one will be used as fallback.
model: string
AI model to use for the request. Must be formed as <provider>/<model_name>, e.g. workers-ai/@cf/meta/llama-3.3-70b-instruct-fp8-fast.
Link to this property
authorization: optional string
Authorization token for the AI model: Bearer <token>. Not needed for workers-ai models.
Link to this property
Link to this property
prompt: optional string
Link to this property
response\_format: optional object {type, json\_schema }
type: string
Link to this property
json\_schema: optional map\[unknown]
Schema for the response format. More information here: https://developers.cloudflare.com/workers-ai/json-mode/
Link to this property
Link to this property
Link to this property
limit: optional number
Maximum number of URLs to crawl.
maximum100000
minimum1
Link to this property
maxAge: optional number
Maximum age of a resource that can be returned from cache in seconds. Default is 1 day.
maximum604800
minimum0
Link to this property
modifiedSince: optional number
Unix timestamp (seconds since epoch) indicating to only crawl pages that were modified since this time. For sitemap URLs with a lastmod field, this is compared directly. For other URLs, the crawler will use If-Modified-Since header when fetching. URLs without modification information (no lastmod in sitemap and no Last-Modified header support) will be crawled. Note: This works in conjunction with maxAge - both filters must pass for a cached resource to be used. Must be within the last year and not in the future.
exclusiveMinimum
minimum0
Link to this property
options: optional object {excludePatterns, includeExternalLinks, includePatterns, includeSubdomains }
Additional options for the crawler.
excludePatterns: optional array of string
Exclude links matching the provided wildcard patterns in the crawl job. Example: ‘https://example.com/privacy/\*\*’.
Link to this property
includeExternalLinks: optional boolean
Include external links in the crawl job. If set to true, includeSubdomains is ignored.
Link to this property
includePatterns: optional array of string
Include only links matching the provided wildcard patterns in the crawl job. Include patterns are evaluated before exclude patterns. URLs that match any of the specified include patterns will be included in the crawl job. Example: ‘https://example.com/blog/\*\*’.
Link to this property
includeSubdomains: optional boolean
Include links to subdomains in the crawl job. This option is ignored if includeExternalLinks is true.
Link to this property
Link to this property
rejectRequestPattern: optional array of string
Block undesired requests that match the provided regex patterns, eg. ’/^.\*.(css)’.
Link to this property
rejectResourceTypes: optional array of "document"or "stylesheet"or "image"or 15 more
Block undesired requests that match the provided resource types, eg. ‘image’ or ‘script’.
One of the following:
"document"
Link to this property
"stylesheet"
Link to this property
"image"
Link to this property
"media"
Link to this property
"font"
Link to this property
"script"
Link to this property
"texttrack"
Link to this property
"xhr"
Link to this property
"fetch"
Link to this property
"prefetch"
Link to this property
"eventsource"
Link to this property
"websocket"
Link to this property
"manifest"
Link to this property
"signedexchange"
Link to this property
"ping"
Link to this property
"cspviolationreport"
Link to this property
"preflight"
Link to this property
"other"
Link to this property
Link to this property
render: optional true
Whether to render the page or fetch static content. True by default.
Link to this property
setExtraHTTPHeaders: optional map\[string]
Link to this property
setJavaScriptEnabled: optional boolean
Link to this property
source: optional "sitemaps"or "links"or "all"
Source of links to crawl. ‘sitemaps’ - only crawl URLs from sitemaps, ‘links’ - only crawl URLs scraped from pages, ‘all’ - crawl both sitemap and scraped links (default).
One of the following:
"sitemaps"
Link to this property
"links"
Link to this property
"all"
Link to this property
Link to this property
viewport: optional object {height, width, deviceScaleFactor, 3 more }
Check options.
height: number
Link to this property
width: number
Link to this property
deviceScaleFactor: optional number
Link to this property
hasTouch: optional boolean
Link to this property
isLandscape: optional boolean
Link to this property
isMobile: optional boolean
Link to this property
Link to this property
waitForSelector: optional object {selector, hidden, timeout, visible }
Wait for the selector to appear in page. Check options.
selector: string
Link to this property
hidden: optional true
Link to this property
timeout: optional number
maximum120000
Link to this property
visible: optional true
Link to this property
Link to this property
waitForTimeout: optional number
Waits for a specified timeout before continuing.
maximum120000
Link to this property
Link to this property
object {render, url, contentUse, 9 more }
render: false
Whether to render the page or fetch static content. True by default.
Link to this property
url: string
URL to navigate to, eg. https://example.com.
formaturi
Link to this property
contentUse: optional "reference"or "full"
Intended content use level to respect the use Content-Signal directive in robots.txt. Levels (least to most permissive): ‘reference’, ‘full’. A URL is disallowed when the publisher’s declared use level is lower than this intent. Learn more: https://contentsignals.org/. Default: ‘full’.
One of the following:
"reference"
Link to this property
"full"
Link to this property
Link to this property
crawlPurposes: optional array of "search"or "ai-input"or "ai-train"
List of crawl purposes to respect Content-Signal directives in robots.txt. Allowed values: ‘search’, ‘ai-input’, ‘ai-train’. Learn more: https://contentsignals.org/. Default: \[‘search’, ‘ai-input’, ‘ai-train’].
One of the following:
"search"
Link to this property
"ai-input"
Link to this property
"ai-train"
Link to this property
Link to this property
depth: optional number
Maximum number of levels deep the crawler will traverse from the starting URL.
maximum100000
minimum1
Link to this property
formats: optional array of "html"or "markdown"or "json"
Formats to return. Default is html.
One of the following:
"html"
Link to this property
"markdown"
Link to this property
"json"
Link to this property
Link to this property
jsonOptions: optional object {custom\_ai, prompt, response\_format }
Options for JSON extraction.
custom\_ai: optional array of object {model, authorization }
Optional list of custom AI models to use for the request. The models will be tried in the order provided, and in case a model returns an error, the next one will be used as fallback.
model: string
AI model to use for the request. Must be formed as <provider>/<model_name>, e.g. workers-ai/@cf/meta/llama-3.3-70b-instruct-fp8-fast.
Link to this property
authorization: optional string
Authorization token for the AI model: Bearer <token>. Not needed for workers-ai models.
Link to this property
Link to this property
prompt: optional string
Link to this property
response\_format: optional object {type, json\_schema }
type: string
Link to this property
json\_schema: optional map\[unknown]
Schema for the response format. More information here: https://developers.cloudflare.com/workers-ai/json-mode/
Link to this property
Link to this property
Link to this property
limit: optional number
Maximum number of URLs to crawl.
maximum100000
minimum1
Link to this property
maxAge: optional number
Maximum age of a resource that can be returned from cache in seconds. Default is 1 day.
maximum604800
minimum0
Link to this property
modifiedSince: optional number
Unix timestamp (seconds since epoch) indicating to only crawl pages that were modified since this time. For sitemap URLs with a lastmod field, this is compared directly. For other URLs, the crawler will use If-Modified-Since header when fetching. URLs without modification information (no lastmod in sitemap and no Last-Modified header support) will be crawled. Note: This works in conjunction with maxAge - both filters must pass for a cached resource to be used. Must be within the last year and not in the future.
exclusiveMinimum
minimum0
Link to this property
options: optional object {excludePatterns, includeExternalLinks, includePatterns, includeSubdomains }
Additional options for the crawler.
excludePatterns: optional array of string
Exclude links matching the provided wildcard patterns in the crawl job. Example: ‘https://example.com/privacy/\*\*’.
Link to this property
includeExternalLinks: optional boolean
Include external links in the crawl job. If set to true, includeSubdomains is ignored.
Link to this property
includePatterns: optional array of string
Include only links matching the provided wildcard patterns in the crawl job. Include patterns are evaluated before exclude patterns. URLs that match any of the specified include patterns will be included in the crawl job. Example: ‘https://example.com/blog/\*\*’.
Link to this property
includeSubdomains: optional boolean
Include links to subdomains in the crawl job. This option is ignored if includeExternalLinks is true.
Link to this property
Link to this property
source: optional "sitemaps"or "links"or "all"
Source of links to crawl. ‘sitemaps’ - only crawl URLs from sitemaps, ‘links’ - only crawl URLs scraped from pages, ‘all’ - crawl both sitemap and scraped links (default).
One of the following:
"sitemaps"
Link to this property
"links"
Link to this property
"all"
Link to this property
Link to this property
Link to this property
[Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(method)%20create%20%3E%20(params)%200%20%3E%20(param)%20body%20%3E%20(schema)>)
##### ReturnsExpand Collapse
result: string
Crawl job ID.
[Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(method)%20create%20%3E%20(network%20schema)%20%3E%20(property)%20result>)
success: boolean
Response status.
[Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(method)%20create%20%3E%20(network%20schema)%20%3E%20(property)%20success>)
errors: optional array of object {code, message }
code: number
Error code.
Link to this property
message: string
Error message.
Link to this property
[Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(method)%20create%20%3E%20(network%20schema)%20%3E%20(property)%20errors>)
### Crawl websites.
HTTP
HTTPTypeScriptPythonGoTerraform
```
curl https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/browser-rendering/crawl \
-H 'Content-Type: application/json' \
-H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
-d '{
"url": "https://example.com"
}'
```
200 example
429 example
```
{
"result": "result",
"success": true,
"errors": [
{
"code": 0,
"message": "message"
}
]
}
```
```
{
"errors": [
{
"code": 2001,
"message": "Rate limit exceeded"
}
],
"success": false
}
```
##### Returns Examples
200 example
429 example
```
{
"result": "result",
"success": true,
"errors": [
{
"code": 0,
"message": "message"
}
]
}
```
```
{
"errors": [
{
"code": 2001,
"message": "Rate limit exceeded"
}
],
"success": false
}
```