--- title: Crawl websites. --- [Skip to content](#_top) [API Reference](https://developers.cloudflare.com/api/python) [Browser Rendering](https://developers.cloudflare.com/api/python/resources/browser_rendering) [Crawl](https://developers.cloudflare.com/api/python/resources/browser_rendering/subresources/crawl) Copy Markdown Open in **Claude**Open in **ChatGPT**Open in **Cursor** --- **Copy Markdown****View as Markdown** # Crawl websites. browser\_rendering.crawl.create(CrawlCreateParams\*\*kwargs) -> [CrawlCreateResponse]() POST/accounts/{account\_id}/browser-rendering/crawl Starts a crawl job for the provided URL and its children. Check available options like `gotoOptions` and `waitFor*` to control page load behaviour. ##### Security

API Token

The preferred authorization scheme for interacting with the Cloudflare API. Create a token. **Example:**Authorization: Bearer Sn3lZJTBX6kkg7OdcBUAxOO963GEIyGQqnFTOFYY

API Email + API Key

The previous authorization scheme for interacting with the Cloudflare API, used in conjunction with a Global API key. **Example:**X-Auth-Email: user@example.com The previous authorization scheme for interacting with the Cloudflare API. When possible, use API tokens instead of Global API keys. **Example:**X-Auth-Key: 144c9defac04969c7bfad8efaa8ea194

##### Accepted Permissions (at least one required) `Browser Rendering Write` ##### ParametersExpand Collapse account\_id: str Account ID. [Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(method)%20create%20%3E%20(params)%200%20%3E%20(param)%20account_id%20%3E%20(schema)>) url: str URL to navigate to, eg. `https://example.com`. formaturi [Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(method)%20create%20%3E%20(params)%200%20%3E%20(param)%20url%20%3E%20(schema)>) cache\_ttl: Optional\[float] Cache TTL default is 5s. Set to 0 to disable. maximum86400 minimum0 [Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(method)%20create%20%3E%20(params)%200%20%3E%20(param)%20cacheTTL%20%3E%20(schema)>) action\_timeout: Optional\[float] The maximum duration allowed for the browser action to complete after the page has loaded (such as taking screenshots, extracting content, or generating PDFs). If this time limit is exceeded, the action stops and returns a timeout error. maximum120000 [Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(method)%20create%20%3E%20(params)%200%20%3E%20(param)%20actionTimeout%20%3E%20(schema)>)

add\_script\_tag: Optional\[Iterable\[Variant0AddScriptTag]] Adds a script element into the page with the desired URL or content.

id: Optional\[str] Link to this property content: Optional\[str] Link to this property type: Optional\[str] Link to this property url: Optional\[str] formaturi Link to this property

[Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(method)%20create%20%3E%20(params)%200%20%3E%20(param)%20addScriptTag%20%3E%20(schema)>)

add\_style\_tag: Optional\[Iterable\[Variant0AddStyleTag]] Adds a <link rel="stylesheet"> tag into the page with the desired URL or a <style type="text/css"> tag with the content.

content: Optional\[str] Link to this property url: Optional\[str] formaturi Link to this property

[Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(method)%20create%20%3E%20(params)%200%20%3E%20(param)%20addStyleTag%20%3E%20(schema)>) allow\_request\_pattern: Optional\[Sequence\[str]] Only allow requests that match the provided regex patterns, eg. ’/^.\*.(css)’. Reject rules are applied first. [Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(method)%20create%20%3E%20(params)%200%20%3E%20(param)%20allowRequestPattern%20%3E%20(schema)>)

allow\_resource\_types: Optional\[List\[Literal\["document", "stylesheet", "image", 15 more]]] Only allow requests that match the provided resource types, eg. ‘image’ or ‘script’. Reject rules are applied first.

[Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(method)%20create%20%3E%20(params)%200%20%3E%20(param)%20allowResourceTypes%20%3E%20(schema)>)

authenticate: Optional\[Variant0Authenticate] Provide credentials for HTTP authentication.

password: str minLength1 Link to this property username: str minLength1 Link to this property

[Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(method)%20create%20%3E%20(params)%200%20%3E%20(param)%20authenticate%20%3E%20(schema)>) best\_attempt: Optional\[[bool]()] Attempt to proceed when ‘awaited’ events fail or timeout. [Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(method)%20create%20%3E%20(params)%200%20%3E%20(param)%20bestAttempt%20%3E%20(schema)>) browser: Optional\[Literal\["kitesurf"]] Rendering backend for this crawl. Set to `kitesurf` to render pages with Kitesurf (beta). Only valid when `render` is `true`. [Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(method)%20create%20%3E%20(params)%200%20%3E%20(param)%20browser%20%3E%20(schema)>)

content\_use: Optional\[Literal\["reference", "full"]] Intended content use level to respect the use Content-Signal directive in robots.txt. Levels (least to most permissive): ‘reference’, ‘full’. A URL is disallowed when the publisher’s declared use level is lower than this intent. Learn more: https://contentsignals.org/. Default: ‘full’.

One of the following: "reference" Link to this property "full" Link to this property

[Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(method)%20create%20%3E%20(params)%200%20%3E%20(param)%20contentUse%20%3E%20(schema)>)

cookies: Optional\[Iterable\[Variant0Cookie]] Check options.

name: str Cookie name. Link to this property value: str Link to this property domain: Optional\[str] Link to this property expires: Optional\[float] Link to this property http\_only: Optional\[bool] Link to this property partition\_key: Optional\[str] Link to this property path: Optional\[str] Link to this property

priority: Optional\[Literal\["Low", "Medium", "High"]]

One of the following: "Low" Link to this property "Medium" Link to this property "High" Link to this property

Link to this property same\_party: Optional\[bool] Link to this property

same\_site: Optional\[Literal\["Strict", "Lax", "None"]]

One of the following: "Strict" Link to this property "Lax" Link to this property "None" Link to this property

Link to this property secure: Optional\[bool] Link to this property source\_port: Optional\[float] Link to this property

source\_scheme: Optional\[Literal\["Unset", "NonSecure", "Secure"]]

One of the following: "Unset" Link to this property "NonSecure" Link to this property "Secure" Link to this property

Link to this property url: Optional\[str] Link to this property

[Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(method)%20create%20%3E%20(params)%200%20%3E%20(param)%20cookies%20%3E%20(schema)>)

crawl\_purposes: Optional\[List\[Literal\["search", "ai-input", "ai-train"]]] List of crawl purposes to respect Content-Signal directives in robots.txt. Allowed values: ‘search’, ‘ai-input’, ‘ai-train’. Learn more: https://contentsignals.org/. Default: \[‘search’, ‘ai-input’, ‘ai-train’].

One of the following: "search" Link to this property "ai-input" Link to this property "ai-train" Link to this property

[Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(method)%20create%20%3E%20(params)%200%20%3E%20(param)%20crawlPurposes%20%3E%20(schema)>) depth: Optional\[float] Maximum number of levels deep the crawler will traverse from the starting URL. maximum100000 minimum1 [Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(method)%20create%20%3E%20(params)%200%20%3E%20(param)%20depth%20%3E%20(schema)>) emulate\_media\_type: Optional\[str] [Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(method)%20create%20%3E%20(params)%200%20%3E%20(param)%20emulateMediaType%20%3E%20(schema)>)

formats: Optional\[List\[Literal\["html", "markdown", "json"]]] Formats to return. Default is html.

One of the following: "html" Link to this property "markdown" Link to this property "json" Link to this property

[Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(method)%20create%20%3E%20(params)%200%20%3E%20(param)%20formats%20%3E%20(schema)>)

goto\_options: Optional\[Variant0GotoOptions] Check options.

referer: Optional\[str] Link to this property referrer\_policy: Optional\[str] Link to this property timeout: Optional\[float] maximum60000 Link to this property

wait\_until: Optional\[Union\[Literal\["load", "domcontentloaded", "networkidle0", "networkidle2"], List\[Literal\["load", "domcontentloaded", "networkidle0", "networkidle2"]]]]

One of the following:

Literal\["load", "domcontentloaded", "networkidle0", "networkidle2"]

One of the following: "load" Link to this property "domcontentloaded" Link to this property "networkidle0" Link to this property "networkidle2" Link to this property

Link to this property

List\[Literal\["load", "domcontentloaded", "networkidle0", "networkidle2"]]

One of the following: "load" Link to this property "domcontentloaded" Link to this property "networkidle0" Link to this property "networkidle2" Link to this property

Link to this property

[Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(method)%20create%20%3E%20(params)%200%20%3E%20(param)%20gotoOptions%20%3E%20(schema)>)

json\_options: Optional\[Variant0JsonOptions] Options for JSON extraction.

custom\_ai: Optional\[Iterable\[Variant0JsonOptionsCustomAI]] Optional list of custom AI models to use for the request. The models will be tried in the order provided, and in case a model returns an error, the next one will be used as fallback.

model: str AI model to use for the request. Must be formed as <provider>/<model_name>, e.g. workers-ai/@cf/meta/llama-3.3-70b-instruct-fp8-fast. Link to this property authorization: Optional\[str] Authorization token for the AI model: Bearer <token>. Not needed for workers-ai models. Link to this property

Link to this property prompt: Optional\[str] Link to this property

response\_format: Optional\[Variant0JsonOptionsResponseFormat]

type: str Link to this property json\_schema: Optional\[Dict\[str, object]] Schema for the response format. More information here: https://developers.cloudflare.com/workers-ai/json-mode/ Link to this property

Link to this property

[Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(method)%20create%20%3E%20(params)%200%20%3E%20(param)%20jsonOptions%20%3E%20(schema)>) limit: Optional\[float] Maximum number of URLs to crawl. maximum100000 minimum1 [Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(method)%20create%20%3E%20(params)%200%20%3E%20(param)%20limit%20%3E%20(schema)>) max\_age: Optional\[float] Maximum age of a resource that can be returned from cache in seconds. Default is 1 day. maximum604800 minimum0 [Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(method)%20create%20%3E%20(params)%200%20%3E%20(param)%20maxAge%20%3E%20(schema)>) modified\_since: Optional\[int] Unix timestamp (seconds since epoch) indicating to only crawl pages that were modified since this time. For sitemap URLs with a lastmod field, this is compared directly. For other URLs, the crawler will use If-Modified-Since header when fetching. URLs without modification information (no lastmod in sitemap and no Last-Modified header support) will be crawled. Note: This works in conjunction with maxAge - both filters must pass for a cached resource to be used. Must be within the last year and not in the future. exclusiveMinimum minimum0 [Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(method)%20create%20%3E%20(params)%200%20%3E%20(param)%20modifiedSince%20%3E%20(schema)>)

options: Optional\[Variant0Options] Additional options for the crawler.

exclude\_patterns: Optional\[Sequence\[str]] Exclude links matching the provided wildcard patterns in the crawl job. Example: ‘https://example.com/privacy/\*\*’. Link to this property include\_external\_links: Optional\[bool] Include external links in the crawl job. If set to true, includeSubdomains is ignored. Link to this property include\_patterns: Optional\[Sequence\[str]] Include only links matching the provided wildcard patterns in the crawl job. Include patterns are evaluated before exclude patterns. URLs that match any of the specified include patterns will be included in the crawl job. Example: ‘https://example.com/blog/\*\*’. Link to this property include\_subdomains: Optional\[bool] Include links to subdomains in the crawl job. This option is ignored if includeExternalLinks is true. Link to this property

[Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(method)%20create%20%3E%20(params)%200%20%3E%20(param)%20options%20%3E%20(schema)>) reject\_request\_pattern: Optional\[Sequence\[str]] Block undesired requests that match the provided regex patterns, eg. ’/^.\*.(css)’. [Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(method)%20create%20%3E%20(params)%200%20%3E%20(param)%20rejectRequestPattern%20%3E%20(schema)>)

reject\_resource\_types: Optional\[List\[Literal\["document", "stylesheet", "image", 15 more]]] Block undesired requests that match the provided resource types, eg. ‘image’ or ‘script’.

[Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(method)%20create%20%3E%20(params)%200%20%3E%20(param)%20rejectResourceTypes%20%3E%20(schema)>) render: Optional\[Literal\[true]] Whether to render the page or fetch static content. True by default. [Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(method)%20create%20%3E%20(params)%200%20%3E%20(param)%20render%20%3E%20(schema)>) set\_extra\_http\_headers: Optional\[Dict\[str, str]] [Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(method)%20create%20%3E%20(params)%200%20%3E%20(param)%20setExtraHTTPHeaders%20%3E%20(schema)>) set\_java\_script\_enabled: Optional\[[bool]()] [Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(method)%20create%20%3E%20(params)%200%20%3E%20(param)%20setJavaScriptEnabled%20%3E%20(schema)>)

source: Optional\[Literal\["sitemaps", "links", "all"]] Source of links to crawl. ‘sitemaps’ - only crawl URLs from sitemaps, ‘links’ - only crawl URLs scraped from pages, ‘all’ - crawl both sitemap and scraped links (default).

One of the following: "sitemaps" Link to this property "links" Link to this property "all" Link to this property

[Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(method)%20create%20%3E%20(params)%200%20%3E%20(param)%20source%20%3E%20(schema)>)

viewport: Optional\[Variant0Viewport] Check options.

height: float Link to this property width: float Link to this property device\_scale\_factor: Optional\[float] Link to this property has\_touch: Optional\[bool] Link to this property is\_landscape: Optional\[bool] Link to this property is\_mobile: Optional\[bool] Link to this property

[Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(method)%20create%20%3E%20(params)%200%20%3E%20(param)%20viewport%20%3E%20(schema)>)

wait\_for\_selector: Optional\[Variant0WaitForSelector] Wait for the selector to appear in page. Check options.

selector: str Link to this property hidden: Optional\[Literal\[true]] Link to this property timeout: Optional\[float] maximum120000 Link to this property visible: Optional\[Literal\[true]] Link to this property

[Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(method)%20create%20%3E%20(params)%200%20%3E%20(param)%20waitForSelector%20%3E%20(schema)>) wait\_for\_timeout: Optional\[float] Waits for a specified timeout before continuing. maximum120000 [Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(method)%20create%20%3E%20(params)%200%20%3E%20(param)%20waitForTimeout%20%3E%20(schema)>) ##### ReturnsExpand Collapse str Crawl job ID. [Link to this property](<#(resource)%20browser_rendering.crawl%20%3E%20(model)%20crawl_create_response%20%3E%20(schema)>) ### Crawl websites. Python HTTPTypeScriptPythonGoTerraform ``` import os from cloudflare import Cloudflare client = Cloudflare( api_token=os.environ.get("CLOUDFLARE_API_TOKEN"), # This is the default and can be omitted ) crawl = client.browser_rendering.crawl.create( account_id="account_id", url="https://example.com", ) print(crawl) ``` 200 example 429 example ``` { "result": "result", "success": true, "errors": [ { "code": 0, "message": "message" } ] } ``` ``` { "errors": [ { "code": 2001, "message": "Rate limit exceeded" } ], "success": false } ``` ##### Returns Examples 200 example 429 example ``` { "result": "result", "success": true, "errors": [ { "code": 0, "message": "message" } ] } ``` ``` { "errors": [ { "code": 2001, "message": "Rate limit exceeded" } ], "success": false } ```