API Reference
The Spider API is based on REST. Our API is predictable, returns JSON-encoded responses, uses standardized HTTP response codes, and authentication. The API supports bulk updates. You can work on multiple objects per request for the core endpoints.
Overview
Authentication
Include your API key in the authorization header.
Authorization: Bearer sk-xxxx...Response formats
Set the content-type header to shape the response.
Just getting started? Quickstart guide →
Not a developer? Use Spider's no-code options to get started without writing code.
https://api.spider.cloudCommon Parameters
These parameters are shared across Crawl, Scrape, Search, Links, Screenshot, and Fetch. Each endpoint includes an interactive panel to configure these parameters.
Advanced(38)
blacklistarrayBlacklist a set of paths that you do not want to crawl. You can use regex patterns to help with the list.
block_booleanDefault = trueads Block advertisements when running request as
browserorsmart. This can greatly increase performance.block_booleanDefault = trueanalytics Block analytics when running request as
browserorsmart. This can greatly increase performance.block_booleanDefault = truestylesheets Block stylesheets when running request as
browserorsmart. This can greatly increase performance.budgetobjectObject that has paths with a counter for limiting the amount of pages. Use
{"*":1}for only crawling the root page. The wildcard matches all routes and you can set child paths to limit depth, e.g.{ "/docs/colors": 10, "/docs/": 100 }.chunking_objectalg Use a chunking algorithm to segment your content output. Pass an object like
{ "type": "bysentence", "value": 2 }to split the text into an array by every 2 sentences. Works well with markdown or text formats.concurrency_numberlimit Set the concurrency limit to help balance request for slower websites. The default is unlimited.
crawl_objecttimeout The
crawl_timeoutparameter allows you to put a max duration on the entire crawl. The default setting is 2 mins.The values for the timeout duration are in the object shape
{ secs: 300, nanos: 0 }.data_objectconnectors Stream crawl results directly to cloud storage and data services. Configure one or more connectors to automatically receive page data as it is crawled. Supports S3, Google Cloud Storage, Google Sheets, Azure Blob Storage, and Supabase.
{ s3: { bucket, access_key_id, secret_access_key, region?, prefix?, content_type? }, gcs: { bucket, service_account_base64, prefix? }, google_sheets: { spreadsheet_id, service_account_base64, sheet_name? }, azure_blob: { connection_string, container, prefix? }, supabase: { url, anon_key, table }, on_find: bool, on_find_metadata: bool }depthnumberDefault = 25The crawl limit for maximum depth. If
0, no limit will be applied.disable_booleanDefault = falsefirst_ party_ javascript Disable the first-party JavaScript allow through downstream blockers (intercept manager, adblock, blocklists). By default first-party scripts pass even when blocklists would skip them. Set to
trueto enforce blocklists strictly even on first-party scripts.disable_booleanDefault = falsefirst_ party_ stylesheets Disable the first-party stylesheet allow when
block_stylesheetsis on. By default first-party CSS still passes, so SPAs that load their own stylesheets via dynamic imports keep hydrating. That covers React, Next.js, AppFabric, and requirejs-style loaders. Set totruefor a strict block-all-CSS bandwidth-minimal crawl.disable_booleanDefault = falsefirst_ party_ visuals Disable the first-party visual allow when
block_visuals/ignore_visualsis on. By default first-party images, media, and fonts still pass. Set totruefor a strictly bandwidth-minimal crawl that drops ALL visuals regardless of origin.disable_booleanDefault = falseintercept Disable request interception when running request as
browserorsmart. This may help with pages that use third-party scripts or external domains.event_objecttracker Track the event request, responses, and automation output when using browser rendering. Pass in the object with the following
requestsandresponsesfor the network output of the page.automationwill send detailed information including a screenshot of each automation step used underautomation_scripts.exclude_stringselector A CSS query selector to use for ignoring content from the markup of the response.
execution_objectscripts Run custom JavaScript on certain paths. Requires
browserorsmartrequest mode. The values should be in the shape"/path_or_url": "custom js".external_arraydomains A list of external domains to treat as one domain. You can use regex paths to include the domains. Set one of the array values to
*to include all domains.full_booleanresources Crawl and download all the resources for a website.
max_numbercredits_ allowed Set the maximum number of credits to use per run. This will return a blocked by client if the initial response is empty. Credits are measured in decimal units, where 10,000 credits equal one dollar (100 credits per penny).max_numbercredits_ per_ page Set the maximum number of credits to use per page. Credits are measured in decimal units, where 10,000 credits equal one dollar (100 credits per penny).metadatabooleanDefault = falseCollect metadata about the content found like page title, description, keywords and etc. This could help improve AI interoperability.
preserve_booleanDefault = falsehost Preserve the default HOST header for the client. This may help with pages that require a HOST, and when the TLS cannot be determined.
redirect_stringDefault = Loosepolicy The network redirect policy to use when performing HTTP request.
requeststringDefault = smartThe request type to perform. Use
smartto perform HTTP request by default until JavaScript rendering is needed for the HTML.request_numberDefault = 60timeout The timeout to use for request. Timeouts can be from
5-255seconds.root_stringselector The root CSS query selector to use extracting content from the markup for the response.
run_booleanDefault = falsein_ background Run the request in the background. Useful if storing data and wanting to trigger crawls to the dashboard.
sessionbooleanDefault = truePersist the session for the client that you use on a website. This allows the HTTP headers and cookies to be set like a real browser session.
sitemapbooleanDefault = falseInclude the sitemap results to crawl.
sitemap_booleanDefault = falseonly Only include the sitemap results to crawl.
sitemap_stringDefault = sitemap.xmlpath The sitemap URL to use when using
sitemap.subdomainsbooleanDefault = falseAllow subdomains to be included.
tldbooleanDefault = falseAllow TLD's to be included.
user_stringagent Add a custom HTTP user agent to the request. By default this is set to a random agent.
wait_objectfor The
wait_forparameter allows you to specify various waiting conditions for a website operation. If provided, it contains the following sub-parameters:The key
idle_networkspecifies the conditions to wait for the network request to be idle within a period. It can include an optional timeout value.The key
idle_network0specifies the conditions to wait for the network request to be idle with a max timeout. It can include an optional timeout value.The key
almost_idle_network0specifies the conditions to wait for the network request to be almost idle with a max timeout. It can include an optional timeout value.The key
selectorspecifies the conditions to wait for a particular CSS selector to be found on the page. It includes an optional timeout value, and the CSS selector to wait for.The key
domspecifies the conditions to wait for a particular element to stop updating for a duration on the page. It includes an optional timeout value, and the CSS selector to wait for.The key
delayspecifies a delay to wait for, with an optional timeout value.The key
page_navigationsset totruethen waiting for all page navigations will be handled.If
wait_foris not provided, the default behavior is to wait for the network to be idle for 500 milliseconds. All of the durations are capped at 60 seconds.The values for the timeout duration are in the object shape
{ secs: 10, nanos: 0 }.webhooksobjectUse webhooks to get notified on events like credit depleted, new pages, metadata, and website status.
{ destination: string, on_credits_depleted: bool, on_credits_half_depleted: bool, on_website_status: bool, on_find: bool, on_find_metadata: bool }whitelistarrayWhitelist a set of paths that you want to crawl, ignoring all other routes that do not match the patterns. You can use regex patterns to help with the list.
Core(5)
disable_booleanhints Disables service-provided hints that automatically optimize request types, geo-region selection, and network filters (for example, updating
network_blacklist/network_whitelistrecommendations based on observed request-pattern outcomes). Hints are enabled by default for allsmartrequest modes.Enable this if you want fully manual control over filtering behavior, are debugging request load order/coverage, or need deterministic behavior across runs.
limitnumberDefault = 0The maximum amount of pages allowed to crawl per website. Remove the value or set it to
0to crawl all pages.max_numbersize The max content size in bytes per page response. Content exceeding this limit will be truncated with a smart head/tail strategy that preserves the beginning and end of the content.
network_string[]blacklist Blocks matching network requests from being fetched/loaded. Use this to reduce bandwidth and noise by preventing known-unneeded third-party resources from ever being requested.
Each entry is a string match pattern (commonly a hostname, domain, or URL substring). If both whitelist and blacklist are set, whitelist takes precedence.
- Good targets:
googletagmanager.com,doubleclick.net,maps.googleapis.com - Prefer specific domains over broad substrings to avoid breaking essential assets.
- Good targets:
network_string[]whitelist Allows only matching network requests to be fetched/loaded. Use this for a strict "allowlist-first" approach: keep the crawl lightweight while still permitting the essential scripts/styles needed for rendering and JS execution.
Each entry is a string match pattern (commonly a hostname, domain, or URL substring). When set, requests not matching any whitelist entry are blocked by default.
- Start with first-party:
example.com,cdn.example.com - Add only what you observe you truly need (fonts/CDNs), then iterate.
- Start with first-party:
Output(16)
clean_booleanhtml Clean the HTML of unwanted attributes.
css_objectextraction_ map Use CSS or XPath selectors to scrape contents from the web page. Set the paths and the extraction object map to perform extractions per path or page.
encodingstringThe type of encoding to use like
UTF-8,SHIFT_JIS, or etc.filter_booleanimages Filter image elements from the markup.
filter_booleanoutput_ images Filter the images from the output.
filter_booleanDefault = trueoutput_ main_ only Filter the nav, aside, and footer from the output. When unset, markdown and text are filtered and raw HTML is the whole page. Set true to filter raw HTML too.
filter_booleanoutput_ svg Filter the svg tags from the output.
filter_booleansvg Filter SVG elements from the markup.
link_jsonrewrite Optional URL rewrite rule applied to every discovered link before it's crawled. This lets you normalize or redirect URLs (for example, rewriting paths or mapping one host pattern to another).
The value must be a JSON object with a
typefield. Supported types:"replace"– simple substring replacement.
Fields:host?: string(optional) – only apply when the link's host matches this value (e.g."blog.example.com").find: string– substring to search for in the URL.replace_with: string– replacement substring.
"regex"– regex-based rewrite with capture groups.
Fields:host?: string(optional) – only apply for this host.pattern: string– regex applied to the full URL.replace_with: string– replacement string supporting$1,$2, etc.
Invalid or unsafe regex patterns (overly long, unbalanced parentheses, advanced lookbehind constructs, etc.) are rejected by the server and ignored.
readabilitybooleanDefault = falseUse readability to pre-process the content for reading. This may drastically improve the content for LLM usage.
return_booleanDefault = falsecookies Return the HTTP response cookies with the results.
return_booleanDefault = falseembeddings Include OpenAI embeddings for
titleanddescription. Requiresmetadatato be enabled.return_string | arrayDefault = rawformat The format to return the data in. Possible values are
markdown,commonmark,raw,text,xml,bytes, andempty. Userawto return the default format of the page likeHTMLetc.return_booleanDefault = falseheaders Return the HTTP response headers with the results.
return_booleanDefault = falsejson_ data Return the JSON data found in scripts used for SSR.
return_booleanDefault = falsepage_ links Return the links found on each page.
Config(9)
cookiesstringAdd HTTP cookies to use for request.
fingerprintbooleanDefault = trueGive browser requests a complete, consistent browser profile.
headersobjectForward HTTP headers to use for all requests. The object is expected to be a map of key value pairs.
providerstringName an outside scraping provider to prefer when Spider's own stack cannot reach a page.
modeon the router object is where you ask for placement. Pass a provider name such aszyte, never a route id. An unknown name is ignored and the request still runs. This is the older top-level form ofrouter.provider. Both still work, androuter.providerwins when both are sent. Provider routing is still rolling out. Spider accepts the value on every request. A provider is dispatched on it as the rollout reaches the fleet, andcosts.vendoron the response names the provider that served a page.provider_objectoptions Provider-native settings, keyed by provider name, passed through when that provider handles the request. Keep it top level, outside
router. Keys namedurl,api_key, ortokeninside it are dropped; the target goes inurland the key inrouter.tokenorrouter.credentials. Check the provider's own documentation for the settings it supports. Provider routing is still rolling out. Spider accepts the value on every request. A provider is dispatched on it as the rollout reaches the fleet, andcosts.vendoron the response names the provider that served a page.remote_stringproxy Use a remote external proxy connection. You also save 50% on data transfer costs when you bring your own proxy.
routerobjectAsk for an outside scraping provider in one object. All five fields are optional.
modeisfallback, which asks for an eligible provider to sit behind Spider's own stack, orfirst, which asks for a provider before Spider.offkeeps the URL on Spider's own stack and asks no outside provider.providertakes one of the eleven names listed underprovider.tokenis your key for that provider and needsproviderbeside it.credentialsis a map keyed by credential name, the same shape asvendor_credentials.fundingisany, the default, orown, which allows only routes your own keys can pay for. Unknown modes and provider names are ignored and the request still runs. See the provider router setup for examples and precedence. Provider routing is still rolling out. Spider accepts the value on every request. A provider is dispatched on it as the rollout reaches the fleet, andcosts.vendoron the response names the provider that served a page.stealthbooleanDefault = trueRoutes the request through a premium residential proxy pool when the target turns away datacenter or standard flat-rate IPs. The serving pool is chosen automatically by health and cost.
Reach for it when you see consistent 403s, repeated challenge pages, or "unusual activity" pages on otherwise plain HTML targets. Adds credit cost (residential pools bill at 1.5–2× the flat-rate baseline); leave it off for sites that work without it.
vendor_Record<string, string>credentials Use your own provider account for the fallback. Supply each key under the provider's own environment variable name, for example
{"ZYTE_API_KEY": "sk-..."}. The provider bills you directly, so Spider charges its markup and not the provider's fee. Spider reads a key on the request to authenticate that one dispatch. It is never echoed in a response, sent to a webhook, or written to a crawl log. This is the older top-level form ofrouter.credentials. Both still work. For the same credential name,router.tokenwins, thenrouter.credentials, then this map. Provider routing is still rolling out. Spider accepts the value on every request. A provider is dispatched on it as the rollout reaches the fleet, andcosts.vendoron the response names the provider that served a page.
Performance(5)
cacheboolean | { maxAge?: number; allowStale?: boolean; period?: string; skipBrowser?: boolean }Default = trueUse HTTP caching for the crawl to speed up repeated runs. Defaults to
true.Accepts either:
true/false- A cache control object:
maxAge(ms): freshness window (default:172800000= 2 days). Set0for always fetch fresh.allowStale: serve cached results even if stale.period: RFC3339 timestamp cutoff (overridesmaxAge), e.g."2025-11-29T12:00:00Z"skipBrowser: skip browser entirely if cached HTML exists. Returns cached HTML directly without launching Chrome for instant responses.
Default behavior by route type:
- Standard routes (
/crawl,/scrape): cache istruewithskipBrowserenabled by default. Cached pages return instantly without re-launching Chrome. To force a fresh browser fetch, setcache: falseor{ "skipBrowser": false }. - AI routes (
/ai/crawl,/ai/scrape, etc.): cache istruebutskipBrowseris not enabled. AI routes always use the browser to ensure live page content for extraction.
delaynumberDefault = 0Add a crawl delay of up to 60 seconds, disabling concurrency. The delay needs to be in milliseconds format.
respect_booleanDefault = truerobots Respect the robots.txt file for crawling.
service_booleanDefault = trueworker_ enabled Allow the website to use Service Workers as needed.
skip_booleanDefault = trueconfig_ checks Skip checking the database for website configuration. This will increase performance for requests that use limit=1.
Automation(4)
automation_objectscripts Run custom web automated tasks on certain paths. Requires
browserorsmartrequest mode.Below are the available actions for web automation:- Evaluate: Runs custom JavaScript code.
{ "Evaluate": "console.log('Hello, World!');" } - Click: Clicks on an element identified by a CSS selector.
{ "Click": "button#submit" } - ClickAll: Clicks on all elements matching a CSS selector.
{ "ClickAll": "button.loadMore" } - ClickPoint: Clicks at the position x and y coordinates.
{ "ClickPoint": { "x": 120.5, "y": 340.25 } } - ClickAllClickable: Clicks on common clickable elements (buttons/inputs/role=button/etc.).
{ "ClickAllClickable": true } - ClickHold: Clicks and holds on an element (via selector) for a duration in milliseconds.
{ "ClickHold": { "selector": "#sliderThumb", "hold_for_ms": 750 } } - ClickHoldPoint: Clicks and holds at a point for a duration in milliseconds.
{ "ClickHoldPoint": { "x": 250.0, "y": 410.0, "hold_for_ms": 750 } } - ClickDrag: Click-and-drag from one element to another (selector → selector) with optional modifier.
{ "ClickDrag": { "from": "#handle", "to": "#target", "modifier": 8 } } - ClickDragPoint: Click-and-drag from one point to another with optional modifier.
{ "ClickDragPoint": { "from_x": 100.0, "from_y": 200.0, "to_x": 500.0, "to_y": 220.0, "modifier": 0 } } - Wait: Waits for a specified duration in milliseconds.
{ "Wait": 2000 } - WaitForNavigation: Waits for the next navigation event.
{ "WaitForNavigation": true } - WaitFor: Waits for an element to appear identified by a CSS selector.
{ "WaitFor": "div#content" } - WaitForWithTimeout: Waits for an element to appear with a timeout (ms).
{ "WaitForWithTimeout": { "selector": "div#content", "timeout": 8000 } } - WaitForAndClick: Waits for an element to appear and then clicks on it, identified by a CSS selector.
{ "WaitForAndClick": "button#loadMore" } - WaitForDom: Waits for DOM updates to settle (quiet/stable) on a selector (or body) with timeout (ms).
{ "WaitForDom": { "selector": "main", "timeout": 12000 } } - ScrollX: Scrolls the screen horizontally by a specified number of pixels.
{ "ScrollX": 100 } - ScrollY: Scrolls the screen vertically by a specified number of pixels.
{ "ScrollY": 200 } - Fill: Fills an input element with a specified value.
{ "Fill": { "selector": "input#name", "value": "John Doe" } } - Type: Type a key into the browser with an optional modifier.
{ "Type": { "value": "John Doe", "modifier": 0 } } - InfiniteScroll: Scrolls the page until the end for certain duration.
{ "InfiniteScroll": 3000 } - Screenshot: Perform a screenshot on the page.
{ "Screenshot": { "full_page": true, "omit_background": true, "output": "out.png" } } - ValidateChain: Set this before a step to validate the prior action to break out of the chain.
{ "ValidateChain": true }
- Evaluate: Runs custom JavaScript code.
evaluate_stringon_ new_ document Set a custom script to evaluate on new document creation.scrollnumberInfinite scroll the page as new content loads, up to a duration in milliseconds. You may still need to use the
wait_forparameters. Requiresbrowserrequest mode.viewportobjectConfigure the viewport for browser requests.
Geolocation(2)
country_stringcode Set a ISO country code for proxy connections. View the locations list for available countries.
localestringThe locale to use for request, example
en-US.
Per-endpoint notes
Scrape exclude limit, depth, and delay. Single-page endpoints.
Screenshot exclude request, return_format, and readability. Returns image data.
Configure these via the Common Parameters panel in each endpoint section.
Provider router setup
The optional router object controls outside-provider routing. Choose fallback or first, set a provider preference, and supply your own keys. See router options, examples, and provider setup links.
Endpoints
Each endpoint has full documentation with parameters, code examples, and a live playground.
Scraper Configs Alpha
Pre-built extraction configs for popular websites. No setup required.
Queries
Check your usage and remaining credits.
Send this request
The playground posts the call on this page against a URL you choose, with your key or without one. Keyless runs work without an account and are rate limited.