---
title: Crawl websites.
---
[Skip to content](#_top)
[API Reference](https://developers.cloudflare.com/api/go)
[Browser Rendering](https://developers.cloudflare.com/api/go/resources/browser_rendering)
[Crawl](https://developers.cloudflare.com/api/go/resources/browser_rendering/subresources/crawl)
Copy Markdown
Open in **Claude**Open in **ChatGPT**Open in **Cursor**
---
**Copy Markdown****View as Markdown**
# Crawl websites.
client.BrowserRendering.Crawl.New(ctx, params) (\*string, error)
POST/accounts/{account\_id}/browser-rendering/crawl
Starts a crawl job for the provided URL and its children. Check available options like `gotoOptions` and `waitFor*` to control page load behaviour.
##### Security
API Token
The preferred authorization scheme for interacting with the Cloudflare API. Create a token.
**Example:**Authorization: Bearer Sn3lZJTBX6kkg7OdcBUAxOO963GEIyGQqnFTOFYY
API Email + API Key
The previous authorization scheme for interacting with the Cloudflare API, used in conjunction with a Global API key.
**Example:**X-Auth-Email: user@example.com
The previous authorization scheme for interacting with the Cloudflare API. When possible, use API tokens instead of Global API keys.
**Example:**X-Auth-Key: 144c9defac04969c7bfad8efaa8ea194
##### Accepted Permissions (at least one required)
`Browser Rendering Write`
##### ParametersExpand Collapse
params CrawlNewParams
AccountID param.Field\[string]
Path param: Account ID.
Link to this property
URL param.Field\[string]
Body param: URL to navigate to, eg. https://example.com.
formaturi
Link to this property
CacheTTL param.Field\[float64]Optional
Query param: Cache TTL default is 5s. Set to 0 to disable.
maximum86400
minimum0
Link to this property
ActionTimeout param.Field\[float64]Optional
Body param: The maximum duration allowed for the browser action to complete after the page has loaded (such as taking screenshots, extracting content, or generating PDFs). If this time limit is exceeded, the action stops and returns a timeout error.
maximum120000
Link to this property
AddScriptTag param.Field\[\[]CrawlNewParamsVariant0AddScriptTag]Optional
Body param: Adds a script element into the page with the desired URL or content.
ID stringOptional
Link to this property
Content stringOptional
Link to this property
Type stringOptional
Link to this property
URL stringOptional
formaturi
Link to this property
Link to this property
AddStyleTag param.Field\[\[]CrawlNewParamsVariant0AddStyleTag]Optional
Body param: Adds a <link rel="stylesheet"> tag into the page with the desired URL or a <style type="text/css"> tag with the content.
Content stringOptional
Link to this property
URL stringOptional
formaturi
Link to this property
Link to this property
AllowRequestPattern param.Field\[\[]string]Optional
Body param: Only allow requests that match the provided regex patterns, eg. ’/^.\*.(css)’. Reject rules are applied first.
Link to this property
AllowResourceTypes param.Field\[\[]CrawlNewParamsVariant0AllowResourceType]Optional
Body param: Only allow requests that match the provided resource types, eg. ‘image’ or ‘script’. Reject rules are applied first.
const CrawlNewParamsVariant0AllowResourceTypeDocument CrawlNewParamsVariant0AllowResourceType = "document"
Link to this property
const CrawlNewParamsVariant0AllowResourceTypeStylesheet CrawlNewParamsVariant0AllowResourceType = "stylesheet"
Link to this property
const CrawlNewParamsVariant0AllowResourceTypeImage CrawlNewParamsVariant0AllowResourceType = "image"
Link to this property
const CrawlNewParamsVariant0AllowResourceTypeMedia CrawlNewParamsVariant0AllowResourceType = "media"
Link to this property
const CrawlNewParamsVariant0AllowResourceTypeFont CrawlNewParamsVariant0AllowResourceType = "font"
Link to this property
const CrawlNewParamsVariant0AllowResourceTypeScript CrawlNewParamsVariant0AllowResourceType = "script"
Link to this property
const CrawlNewParamsVariant0AllowResourceTypeTexttrack CrawlNewParamsVariant0AllowResourceType = "texttrack"
Link to this property
const CrawlNewParamsVariant0AllowResourceTypeXHR CrawlNewParamsVariant0AllowResourceType = "xhr"
Link to this property
const CrawlNewParamsVariant0AllowResourceTypeFetch CrawlNewParamsVariant0AllowResourceType = "fetch"
Link to this property
const CrawlNewParamsVariant0AllowResourceTypePrefetch CrawlNewParamsVariant0AllowResourceType = "prefetch"
Link to this property
const CrawlNewParamsVariant0AllowResourceTypeEventsource CrawlNewParamsVariant0AllowResourceType = "eventsource"
Link to this property
const CrawlNewParamsVariant0AllowResourceTypeWebsocket CrawlNewParamsVariant0AllowResourceType = "websocket"
Link to this property
const CrawlNewParamsVariant0AllowResourceTypeManifest CrawlNewParamsVariant0AllowResourceType = "manifest"
Link to this property
const CrawlNewParamsVariant0AllowResourceTypeSignedexchange CrawlNewParamsVariant0AllowResourceType = "signedexchange"
Link to this property
const CrawlNewParamsVariant0AllowResourceTypePing CrawlNewParamsVariant0AllowResourceType = "ping"
Link to this property
const CrawlNewParamsVariant0AllowResourceTypeCspviolationreport CrawlNewParamsVariant0AllowResourceType = "cspviolationreport"
Link to this property
const CrawlNewParamsVariant0AllowResourceTypePreflight CrawlNewParamsVariant0AllowResourceType = "preflight"
Link to this property
const CrawlNewParamsVariant0AllowResourceTypeOther CrawlNewParamsVariant0AllowResourceType = "other"
Link to this property
Link to this property
Authenticate param.Field\[CrawlNewParamsVariant0Authenticate]Optional
Body param: Provide credentials for HTTP authentication.
Password string
minLength1
Link to this property
Username string
minLength1
Link to this property
Link to this property
BestAttempt param.Field\[bool]Optional
Body param: Attempt to proceed when ‘awaited’ events fail or timeout.
Link to this property
Browser param.Field\[CrawlNewParamsVariant0Browser]Optional
Body param: Rendering backend for this crawl. Set to kitesurf to render pages with Kitesurf (beta). Only valid when render is true.
const CrawlNewParamsVariant0BrowserKitesurf CrawlNewParamsVariant0Browser = "kitesurf"
Link to this property
Link to this property
ContentUse param.Field\[CrawlNewParamsVariant0ContentUse]Optional
Body param: Intended content use level to respect the use Content-Signal directive in robots.txt. Levels (least to most permissive): ‘reference’, ‘full’. A URL is disallowed when the publisher’s declared use level is lower than this intent. Learn more: https://contentsignals.org/. Default: ‘full’.
const CrawlNewParamsVariant0ContentUseReference CrawlNewParamsVariant0ContentUse = "reference"
Link to this property
const CrawlNewParamsVariant0ContentUseFull CrawlNewParamsVariant0ContentUse = "full"
Link to this property
Link to this property
Cookies param.Field\[\[]CrawlNewParamsVariant0Cookie]Optional
Body param: Check options.
Name string
Cookie name.
Link to this property
Value string
Link to this property
Domain stringOptional
Link to this property
Expires float64Optional
Link to this property
HTTPOnly boolOptional
Link to this property
PartitionKey stringOptional
Link to this property
Path stringOptional
Link to this property
Priority CrawlNewParamsVariant0CookiesPriorityOptional
One of the following:
const CrawlNewParamsVariant0CookiesPriorityLow CrawlNewParamsVariant0CookiesPriority = "Low"
Link to this property
const CrawlNewParamsVariant0CookiesPriorityMedium CrawlNewParamsVariant0CookiesPriority = "Medium"
Link to this property
const CrawlNewParamsVariant0CookiesPriorityHigh CrawlNewParamsVariant0CookiesPriority = "High"
Link to this property
Link to this property
SameParty boolOptional
Link to this property
SameSite CrawlNewParamsVariant0CookiesSameSiteOptional
One of the following:
const CrawlNewParamsVariant0CookiesSameSiteStrict CrawlNewParamsVariant0CookiesSameSite = "Strict"
Link to this property
const CrawlNewParamsVariant0CookiesSameSiteLax CrawlNewParamsVariant0CookiesSameSite = "Lax"
Link to this property
const CrawlNewParamsVariant0CookiesSameSiteNone CrawlNewParamsVariant0CookiesSameSite = "None"
Link to this property
Link to this property
Secure boolOptional
Link to this property
SourcePort float64Optional
Link to this property
SourceScheme CrawlNewParamsVariant0CookiesSourceSchemeOptional
One of the following:
const CrawlNewParamsVariant0CookiesSourceSchemeUnset CrawlNewParamsVariant0CookiesSourceScheme = "Unset"
Link to this property
const CrawlNewParamsVariant0CookiesSourceSchemeNonSecure CrawlNewParamsVariant0CookiesSourceScheme = "NonSecure"
Link to this property
const CrawlNewParamsVariant0CookiesSourceSchemeSecure CrawlNewParamsVariant0CookiesSourceScheme = "Secure"
Link to this property
Link to this property
URL stringOptional
Link to this property
Link to this property
CrawlPurposes param.Field\[\[]CrawlNewParamsVariant0CrawlPurpose]Optional
Body param: List of crawl purposes to respect Content-Signal directives in robots.txt. Allowed values: ‘search’, ‘ai-input’, ‘ai-train’. Learn more: https://contentsignals.org/. Default: \[‘search’, ‘ai-input’, ‘ai-train’].
const CrawlNewParamsVariant0CrawlPurposeSearch CrawlNewParamsVariant0CrawlPurpose = "search"
Link to this property
const CrawlNewParamsVariant0CrawlPurposeAIInput CrawlNewParamsVariant0CrawlPurpose = "ai-input"
Link to this property
const CrawlNewParamsVariant0CrawlPurposeAITrain CrawlNewParamsVariant0CrawlPurpose = "ai-train"
Link to this property
Link to this property
Depth param.Field\[float64]Optional
Body param: Maximum number of levels deep the crawler will traverse from the starting URL.
maximum100000
minimum1
Link to this property
EmulateMediaType param.Field\[string]Optional
Body param
Link to this property
Formats param.Field\[\[]CrawlNewParamsVariant0Format]Optional
Body param: Formats to return. Default is html.
const CrawlNewParamsVariant0FormatHTML CrawlNewParamsVariant0Format = "html"
Link to this property
const CrawlNewParamsVariant0FormatMarkdown CrawlNewParamsVariant0Format = "markdown"
Link to this property
const CrawlNewParamsVariant0FormatJson CrawlNewParamsVariant0Format = "json"
Link to this property
Link to this property
GotoOptions param.Field\[CrawlNewParamsVariant0GotoOptions]Optional
Body param: Check options.
Referer stringOptional
Link to this property
ReferrerPolicy stringOptional
Link to this property
Timeout float64Optional
maximum60000
Link to this property
WaitUntil CrawlNewParamsVariant0GotoOptionsWaitUntilUnionOptional
One of the following:
CrawlNewParamsVariant0GotoOptionsWaitUntilString
One of the following:
const CrawlNewParamsVariant0GotoOptionsWaitUntilStringLoad CrawlNewParamsVariant0GotoOptionsWaitUntilString = "load"
Link to this property
const CrawlNewParamsVariant0GotoOptionsWaitUntilStringDomcontentloaded CrawlNewParamsVariant0GotoOptionsWaitUntilString = "domcontentloaded"
Link to this property
const CrawlNewParamsVariant0GotoOptionsWaitUntilStringNetworkidle0 CrawlNewParamsVariant0GotoOptionsWaitUntilString = "networkidle0"
Link to this property
const CrawlNewParamsVariant0GotoOptionsWaitUntilStringNetworkidle2 CrawlNewParamsVariant0GotoOptionsWaitUntilString = "networkidle2"
Link to this property
Link to this property
CrawlNewParamsVariant0GotoOptionsWaitUntilArray
One of the following:
const CrawlNewParamsVariant0GotoOptionsWaitUntilArrayLoad CrawlNewParamsVariant0GotoOptionsWaitUntilArray = "load"
Link to this property
const CrawlNewParamsVariant0GotoOptionsWaitUntilArrayDomcontentloaded CrawlNewParamsVariant0GotoOptionsWaitUntilArray = "domcontentloaded"
Link to this property
const CrawlNewParamsVariant0GotoOptionsWaitUntilArrayNetworkidle0 CrawlNewParamsVariant0GotoOptionsWaitUntilArray = "networkidle0"
Link to this property
const CrawlNewParamsVariant0GotoOptionsWaitUntilArrayNetworkidle2 CrawlNewParamsVariant0GotoOptionsWaitUntilArray = "networkidle2"
Link to this property
Link to this property
Link to this property
Link to this property
JsonOptions param.Field\[CrawlNewParamsVariant0JsonOptions]Optional
Body param: Options for JSON extraction.
CustomAI \[]CrawlNewParamsVariant0JsonOptionsCustomAIOptional
Optional list of custom AI models to use for the request. The models will be tried in the order provided, and in case a model returns an error, the next one will be used as fallback.
Model string
AI model to use for the request. Must be formed as <provider>/<model_name>, e.g. workers-ai/@cf/meta/llama-3.3-70b-instruct-fp8-fast.
Link to this property
Authorization stringOptional
Authorization token for the AI model: Bearer <token>. Not needed for workers-ai models.
Link to this property
Link to this property
Prompt stringOptional
Link to this property
ResponseFormat CrawlNewParamsVariant0JsonOptionsResponseFormatOptional
Type string
Link to this property
JsonSchema map\[string, unknown]Optional
Schema for the response format. More information here: https://developers.cloudflare.com/workers-ai/json-mode/
Link to this property
Link to this property
Link to this property
Limit param.Field\[float64]Optional
Body param: Maximum number of URLs to crawl.
maximum100000
minimum1
Link to this property
MaxAge param.Field\[float64]Optional
Body param: Maximum age of a resource that can be returned from cache in seconds. Default is 1 day.
maximum604800
minimum0
Link to this property
ModifiedSince param.Field\[int64]Optional
Body param: Unix timestamp (seconds since epoch) indicating to only crawl pages that were modified since this time. For sitemap URLs with a lastmod field, this is compared directly. For other URLs, the crawler will use If-Modified-Since header when fetching. URLs without modification information (no lastmod in sitemap and no Last-Modified header support) will be crawled. Note: This works in conjunction with maxAge - both filters must pass for a cached resource to be used. Must be within the last year and not in the future.
exclusiveMinimum
minimum0
Link to this property
Options param.Field\[CrawlNewParamsVariant0Options]Optional
Body param: Additional options for the crawler.
ExcludePatterns \[]stringOptional
Exclude links matching the provided wildcard patterns in the crawl job. Example: ‘https://example.com/privacy/\*\*’.
Link to this property
IncludeExternalLinks boolOptional
Include external links in the crawl job. If set to true, includeSubdomains is ignored.
Link to this property
IncludePatterns \[]stringOptional
Include only links matching the provided wildcard patterns in the crawl job. Include patterns are evaluated before exclude patterns. URLs that match any of the specified include patterns will be included in the crawl job. Example: ‘https://example.com/blog/\*\*’.
Link to this property
IncludeSubdomains boolOptional
Include links to subdomains in the crawl job. This option is ignored if includeExternalLinks is true.
Link to this property
Link to this property
RejectRequestPattern param.Field\[\[]string]Optional
Body param: Block undesired requests that match the provided regex patterns, eg. ’/^.\*.(css)’.
Link to this property
RejectResourceTypes param.Field\[\[]CrawlNewParamsVariant0RejectResourceType]Optional
Body param: Block undesired requests that match the provided resource types, eg. ‘image’ or ‘script’.
const CrawlNewParamsVariant0RejectResourceTypeDocument CrawlNewParamsVariant0RejectResourceType = "document"
Link to this property
const CrawlNewParamsVariant0RejectResourceTypeStylesheet CrawlNewParamsVariant0RejectResourceType = "stylesheet"
Link to this property
const CrawlNewParamsVariant0RejectResourceTypeImage CrawlNewParamsVariant0RejectResourceType = "image"
Link to this property
const CrawlNewParamsVariant0RejectResourceTypeMedia CrawlNewParamsVariant0RejectResourceType = "media"
Link to this property
const CrawlNewParamsVariant0RejectResourceTypeFont CrawlNewParamsVariant0RejectResourceType = "font"
Link to this property
const CrawlNewParamsVariant0RejectResourceTypeScript CrawlNewParamsVariant0RejectResourceType = "script"
Link to this property
const CrawlNewParamsVariant0RejectResourceTypeTexttrack CrawlNewParamsVariant0RejectResourceType = "texttrack"
Link to this property
const CrawlNewParamsVariant0RejectResourceTypeXHR CrawlNewParamsVariant0RejectResourceType = "xhr"
Link to this property
const CrawlNewParamsVariant0RejectResourceTypeFetch CrawlNewParamsVariant0RejectResourceType = "fetch"
Link to this property
const CrawlNewParamsVariant0RejectResourceTypePrefetch CrawlNewParamsVariant0RejectResourceType = "prefetch"
Link to this property
const CrawlNewParamsVariant0RejectResourceTypeEventsource CrawlNewParamsVariant0RejectResourceType = "eventsource"
Link to this property
const CrawlNewParamsVariant0RejectResourceTypeWebsocket CrawlNewParamsVariant0RejectResourceType = "websocket"
Link to this property
const CrawlNewParamsVariant0RejectResourceTypeManifest CrawlNewParamsVariant0RejectResourceType = "manifest"
Link to this property
const CrawlNewParamsVariant0RejectResourceTypeSignedexchange CrawlNewParamsVariant0RejectResourceType = "signedexchange"
Link to this property
const CrawlNewParamsVariant0RejectResourceTypePing CrawlNewParamsVariant0RejectResourceType = "ping"
Link to this property
const CrawlNewParamsVariant0RejectResourceTypeCspviolationreport CrawlNewParamsVariant0RejectResourceType = "cspviolationreport"
Link to this property
const CrawlNewParamsVariant0RejectResourceTypePreflight CrawlNewParamsVariant0RejectResourceType = "preflight"
Link to this property
const CrawlNewParamsVariant0RejectResourceTypeOther CrawlNewParamsVariant0RejectResourceType = "other"
Link to this property
Link to this property
Render param.Field\[CrawlNewParamsVariant0Render]Optional
Body param: Whether to render the page or fetch static content. True by default.
const CrawlNewParamsVariant0RenderTrue CrawlNewParamsVariant0Render = true
Link to this property
Link to this property
SetExtraHTTPHeaders param.Field\[map\[string, string]]Optional
Body param
Link to this property
SetJavaScriptEnabled param.Field\[bool]Optional
Body param
Link to this property
Source param.Field\[CrawlNewParamsVariant0Source]Optional
Body param: Source of links to crawl. ‘sitemaps’ - only crawl URLs from sitemaps, ‘links’ - only crawl URLs scraped from pages, ‘all’ - crawl both sitemap and scraped links (default).
const CrawlNewParamsVariant0SourceSitemaps CrawlNewParamsVariant0Source = "sitemaps"
Link to this property
const CrawlNewParamsVariant0SourceLinks CrawlNewParamsVariant0Source = "links"
Link to this property
const CrawlNewParamsVariant0SourceAll CrawlNewParamsVariant0Source = "all"
Link to this property
Link to this property
Viewport param.Field\[CrawlNewParamsVariant0Viewport]Optional
Body param: Check options.
Height float64
Link to this property
Width float64
Link to this property
DeviceScaleFactor float64Optional
Link to this property
HasTouch boolOptional
Link to this property
IsLandscape boolOptional
Link to this property
IsMobile boolOptional
Link to this property
Link to this property
WaitForSelector param.Field\[CrawlNewParamsVariant0WaitForSelector]Optional
Body param: Wait for the selector to appear in page. Check options.
Selector string
Link to this property
Hidden CrawlNewParamsVariant0WaitForSelectorHiddenOptional
Link to this property
Timeout float64Optional
maximum120000
Link to this property
Visible CrawlNewParamsVariant0WaitForSelectorVisibleOptional
Link to this property
Link to this property
WaitForTimeout param.Field\[float64]Optional
Body param: Waits for a specified timeout before continuing.
maximum120000
Link to this property
[Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(method)%20create%20%3E%20(params)%200>)
##### ReturnsExpand Collapse
type CrawlNewResponseEnvelopeResult string
Crawl job ID.
[Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(method)%20create%20%3E%20(network%20schema)%20%3E%20(property)%20result>)
### Crawl websites.
Go
HTTPTypeScriptPythonGoTerraform
```
package main
import (
"context"
"fmt"
"github.com/cloudflare/cloudflare-go"
"github.com/cloudflare/cloudflare-go/browser_rendering"
"github.com/cloudflare/cloudflare-go/option"
)
func main() {
client := cloudflare.NewClient(
option.WithAPIToken("Sn3lZJTBX6kkg7OdcBUAxOO963GEIyGQqnFTOFYY"),
)
crawl, err := client.BrowserRendering.Crawl.New(context.TODO(), browser_rendering.CrawlNewParams{
AccountID: cloudflare.F("account_id"),
Body: browser_rendering.CrawlNewParamsBodyObject{
URL: cloudflare.F("https://example.com"),
},
})
if err != nil {
panic(err.Error())
}
fmt.Printf("%+v\n", crawl)
}
```
200 example
429 example
```
{
"result": "result",
"success": true,
"errors": [
{
"code": 0,
"message": "message"
}
]
}
```
```
{
"errors": [
{
"code": 2001,
"message": "Rate limit exceeded"
}
],
"success": false
}
```
##### Returns Examples
200 example
429 example
```
{
"result": "result",
"success": true,
"errors": [
{
"code": 0,
"message": "message"
}
]
}
```
```
{
"errors": [
{
"code": 2001,
"message": "Rate limit exceeded"
}
],
"success": false
}
```