API Reference

The Spider API is based on REST. Our API is predictable, returns JSON-encoded responses, uses standard HTTP response codes, authentication, and verbs. Set your API secret key in the authorization header to commence with the format Bearer $TOKEN. You can use the content-type header with application/json, application/xml, text/csv, and application/jsonl for shaping the response.

The Spider API supports multi domain actions. You can work with multiple domains per request by adding the urls comma separated.

The Spider API differs for every account as we release new versions and tailor functionality. You can add v1 before any path to pin to the version. Executing a request on the page by pressing the Run button will consume live credits and consider the response as a genuine result.

Just getting started?

Check out our development quickstart guide.

Not a developer?

Use Spiders no-code options or apps to get started with Spider and to do more with your Spider account no code required.

Base Url

https://api.spider.cloud

Client libraries

CLI Javascript Python Rust

Crawl websites

Start crawling a website(s) to collect resources.

POSThttps://api.spider.cloud/crawl

Request body

url required string
The URI resource to crawl. This can be a comma split list for multiple urls.
To reduce latency, enhance performance, and save on batch multiple URLs into a single call. For large websites with high page limits, it's best to run requests individually.
Test Url
limit number
The maximum amount of pages allowed to crawl per website. Remove the value or set it to 0 to crawl all pages.
It is better to set a limit upfront on websites where you do not know the size. Re-crawling can effectively use cache to keep costs low as new pages are found.
Crawl Limit
request string
The request type to perform. Possible values are http, chrome, and smart. Use smart to perform HTTP request by default until JavaScript rendering is needed for the HTML.
The request greatly influences how the output is going to look. If the page is server-side rendered, you can stick to the defaults for the most part.
return_format string
The format to return the data in. Possible values are markdown, commonmark, raw, text, and bytes. Use raw to return the default format of the page like HTML etc.
Usually you want to use markdown for LLM processing or text. If you need to store the files without losing any encoding, use the bytes or raw format.
readability boolean
Use readability to pre-process the content for reading. This may drastically improve the content for LLM usage.
This uses the Safari Reader Mode algorithm to extract only important information from the content.
metadata boolean
Collect metadata about the content found like page title, description, keywards and etc. This could help improve AI interoperability. Defaults to false unless you have the website already stored with the configuration enabled.
Using metadata can help extract critical information to use for AI.
anti_bot boolean
Enable anti-bot mode using various techniques to increase the chance of success. Default is false.
This config will attempt to make request resemble a real human. If the request fails on chrome it will retry using a virtual display for the request that is slower difficult to block at the cost of speed.
css_extraction_map object
Use CSS query selectors to scrape contents from the web page. Set the paths and the CSS extraction object map to perform extractions per path or page.
Scrape content using CSS or XPATH selectors to get data. You can scrape using selectors at no extra cost.
store_data boolean
Determine whether to store data. If set this takes precedence over storageless. Defaults to false.
Set to true to collect resources to download and re-use later on.
tld boolean
Allow TLD's to be included. Default is false.
depth number
The crawl limit for maximum depth. If 0, no limit will be applied.
Depth allows you to place a distance between the base URL path and subfolders.
Crawl Depth
cache boolean
Use HTTP caching for the crawl to speed up repeated runs. Default is true.
Enabling caching can save costs when you need to perform transformations on different files or handle various events on a website.
delay number
Add a crawl delay of up to 60 seconds, disabling concurrency. The delay needs to be in milliseconds format. The default is 0, which indicates it is disabled.
Using a delay can help with websites that are set on a cron and do not require immediate data retrieval.
budget object
Object that has paths with a counter for limiting the amount of pages example {"*":1} for only crawling the root page. The wildcard matches all routes and you can set child paths preventing a depth level, example of limiting { "/docs/colors": 10, "/docs/": 100 } which only allows a max of 100 pages if the route matches /docs/:pathname and only 10 pages if it matches /docs/colors/:pathname.
The budget explicitly allows you to set paths and limits for the crawl.
Crawl Budget
scroll number
Infinite scroll the page as new content loads, up to a duration in milliseconds. The duration represents the maximum time you would wait to scroll. You may still need to use the wait_for parameters. You also need to ensure that the request is made using chrome.
Use the wait_for configuration to scroll until and disable_intercept to make sure you get data from the network regardless of hostname.
locale string
The locale to use for request, example en-US.
cookies string
Add HTTP cookies to use for request.
Set the cookie value for pages that use SSR authentication.
stealth boolean
Use stealth mode for headless chrome request to help prevent being blocked. The default is false on chrome.
Set to true to almost guarantee not being detected by anything.
headers object
Forward HTTP headers to use for all request. The object is expected to be a map of key value pairs.
Using HTTP headers can help with authenticated pages that use the authorization header field.
sitemap boolean
Include the sitemap results to crawl. The default is false.
The sitemap allows you to include links that may not be exposed in the HTML.
viewport object
Configure the viewport for chrome. Defaults to 800x600.
If you need to get data from a website as a mobile, set the viewport to a phone device's size ex: 375x414.
encoding string
The type of encoding to use like UTF-8, SHIFT_JIS, or etc.
Perform the encoding on the server when you know in advance the type of website.
wait_for object
The wait_for parameter allows you to specify various waiting conditions for a website operation. If provided, it contains the following sub-parameters:
The key idle_network specifies the conditions to wait for the network to be idle. It can include an optional timeout value.
The key selector specifies the conditions to wait for a particular CSS selector to be found on the page. It includes an optional timeout value, and the CSS selector to wait for.
The key delay specifies a delay to wait for, with an optional timeout value.
The key page_navigationsset to true then waiting for all page navigations will be handled.
If wait_for is not provided, the default behavior is to wait for the network to be idle for 500 milliseconds. All of the durations are capped at capped at 60 seconds.
The values for the timeout duration are in the object shape { secs: 10, nanos: 0 }.
blacklist array
Blacklist a set of paths that you do not want to crawl. You can use Regex patterns to help with the list.
whitelist array
Whitelist a set of paths that you want to crawl, ignoring all other routes that do not match the patterns. You can use regex patterns to help with the list.
subdomains boolean
Allow subdomains to be included. Default is false.
user_agent string
Add a custom HTTP user agent to the request. By default this is set to a random agent.
gpt_config object
Use AI to generate actions to perform during the crawl. You can pass an array for the prompt to chain steps.
Perform event-driven browser actions on a web page or extract data using the gpt_config, you need to use your own API key or have at least $100 in credits to use AI.
fingerprint boolean
Use advanced fingerprint detection for chrome.
Set this value to help crawl when websites require a fingerprint.
storageless boolean
Prevent storing any type of data for the request including storage. Defaults to true unless you have the website already stored.
chunking_alg object
Use a chunking algorithm to segment your content output. Possible values are ByWords, ByLines, ByCharacterLength, and BySentence. The following is an example of the object shape for "chunking_alg": { "type": "bysentence", "value": 2 } which splits the text into an array by every 2 sentences found. This works well when used with markdown or text formats.
The chunking algorithm allows you to prepare the content for the AI quick, without needing to add extra code or loaders.
country_code string
Set a ISO country code for premium proxy connections, this only works for HTTP request at the moment. You need to also use proxy_enabled.
The country code allows you to run requests in regions where access to the website is restricted to within that specific region.
root_selector string
The CSS query selector to use when extracting content from the markup.
Test Query Selector
page_insights boolean
Gather page insights like request duration, accessibility, and other vitals. The default is false and the param requires metadata to be used in conjunction.
Page Insights provides a quick, high-level alternative to Google's PageSpeed.
return_headers boolean
Return the HTTP response headers with the results. Defaults to false unless you have the website already stored with the configuration enabled.
Getting the HTTP headers can help setup authentication flows.
return_cookies boolean
Return the HTTP response cookies with the results. Defaults to false unless you have the website already stored with the configuration enabled.
Getting the HTTP cookies can help setup authentication SSR flows.
respect_robots boolean
Respect the robots.txt file for crawling. Default is true.
If you have trouble crawling a website it may be an issue with the robots.txt file. Setting the value to false could help. Make sure to use this config sparingly.
full_resources boolean
Crawl and download all the resources for a website.
Collect all the content from the website, including assets like images, videos, etc.
request_timeout number
The timeout to use for request. Timeouts can be from 5-60. The default is 30 seconds.
The timeout of the request helps prevent long request times from hanging.
external_domains array
A list of external domains to include into the domain results. You can use regex paths to include the domains. Set one of the array values to * to include all domains.
return_page_links boolean
Return the links found on each page. Defaults to false unless you have the website already stored with the configuration enabled.
Getting the links can help index the reference locations found for the resource.
return_embeddings boolean
Include OpenAI embeddings for title and description. The default is false and the param requires metadata to be used in conjunction.
If you are embedding data, you can use these matrices as staples for most vector baseline operations.
execution_scripts object
Run custom JavaScript on certain paths. This helps with authenticated pages. The request mode to be made through chrome or smart to run. The values for the timeout duration should be in the object shape "/path_or_url": "custom js".
Custom scripts allow you to take control of the browser with events for up to 60 seconds at a time per page.
disable_intercept boolean
Disable request interception when running request as chrome or smart. This may help if you bypass pages that use third-party scripts or external domains. The default setting is false. Note that cost and speed will increase when disabling this feature, as it removes our native Chrome interception.
run_in_background boolean
Run the request in the background. Useful if storing data and wanting to trigger crawls to the dashboard. This has no effect if storageless is set.
skip_config_checks boolean
Skip checking the database for website configuration. This may increase performance for request that are using limit=1. The default is false.
automation_scripts object
Run custom web automated tasks on certain paths. The request mode to be made through chrome or smart to run.
Custom web automation allows you to take control of the browser with events for up to 60 seconds at a time per page.
Below are the available actions for web automation:
Evaluate: Runs custom JavaScript code.
{ "Evaluate": "console.log('Hello, World!');" }
Click: Clicks on an element identified by a CSS selector.
{ "Click": "button#submit" }
Wait: Waits for a specified duration in milliseconds.
{ "Wait": 2000 }
WaitForNavigation: Waits for the next navigation event.
{ "WaitForNavigation": true }
WaitFor: Waits for an element to appear identified by a CSS selector.
{ "WaitFor": "div#content" }
WaitForAndClick: Waits for an element to appear and then clicks on it, identified by a CSS selector.
{ "WaitForAndClick": "button#loadMore" }
ScrollX: Scrolls the screen horizontally by a specified number of pixels.
{ "ScrollX": 100 }
ScrollY: Scrolls the screen vertically by a specified number of pixels.
{ "ScrollY": 200 }
Fill: Fills an input element with a specified value.
{ "Fill": { "selector": "input#name", "value": "John Doe" } }
InfiniteScroll: Scrolls the page until the end for certain duration.
{ "InfiniteScroll": 3000 }
proxy_enabled boolean
Enable premium proxies to prevent detection. Default is false.
Using this configuration can help when request are blocked through the network. This allows more advanced proxies with AI.

Example request

import requests, os

headers = {
    'Authorization': os.getenv("SPIDER_API_KEY"),
    'Content-Type': 'application/json',
}

json_data = {"limit":25,"return_format":"markdown","url":"https://spider.cloud"}

response = requests.post('https://api.spider.cloud/crawl', 
  headers=headers, json=json_data)

print(response.json())

Response

[
  {
    "content": "<html>...",
    "error": null,
    "status": 200,
    "credits_used": 2,
    "url": "https://spider.cloud"
  },
  // more content...
]

Search

Perform a search and gather a list of websites to start crawling and collect resources.

POSThttps://api.spider.cloud/search

Request body

search required string
The search query you want to search for.
Search
search_limit number
The limit amount of urls to fetch or crawl from the search results. Remove the value or set it to 0 to crawl all URLs from the search results.
fetch_page_content boolean
Fetch all the content of the websites by performing crawls. The default is true; if this is disabled, only the search results are returned instead.
country string
The country code to use for the search. It's a two-letter country code. (e.g. us for the United States).
location string
The location from where you want the search to originate.
language string
The language to use for the search. It's a two-letter language code (e.g., en for English).
limit number
The maximum amount of pages allowed to crawl per website. Remove the value or set it to 0 to crawl all pages.
It is better to set a limit upfront on websites where you do not know the size. Re-crawling can effectively use cache to keep costs low as new pages are found.
Crawl Limit
request string
The request type to perform. Possible values are http, chrome, and smart. Use smart to perform HTTP request by default until JavaScript rendering is needed for the HTML.
The request greatly influences how the output is going to look. If the page is server-side rendered, you can stick to the defaults for the most part.
return_format string
The format to return the data in. Possible values are markdown, commonmark, raw, text, and bytes. Use raw to return the default format of the page like HTML etc.
Usually you want to use markdown for LLM processing or text. If you need to store the files without losing any encoding, use the bytes or raw format.
readability boolean
Use readability to pre-process the content for reading. This may drastically improve the content for LLM usage.
This uses the Safari Reader Mode algorithm to extract only important information from the content.
metadata boolean
Collect metadata about the content found like page title, description, keywards and etc. This could help improve AI interoperability. Defaults to false unless you have the website already stored with the configuration enabled.
Using metadata can help extract critical information to use for AI.
anti_bot boolean
Enable anti-bot mode using various techniques to increase the chance of success. Default is false.
This config will attempt to make request resemble a real human. If the request fails on chrome it will retry using a virtual display for the request that is slower difficult to block at the cost of speed.
css_extraction_map object
Use CSS query selectors to scrape contents from the web page. Set the paths and the CSS extraction object map to perform extractions per path or page.
Scrape content using CSS or XPATH selectors to get data. You can scrape using selectors at no extra cost.
store_data boolean
Determine whether to store data. If set this takes precedence over storageless. Defaults to false.
Set to true to collect resources to download and re-use later on.
num number
The maximum number of results to return for the search.
tld boolean
Allow TLD's to be included. Default is false.
page number
The page number for the search results.
depth number
The crawl limit for maximum depth. If 0, no limit will be applied.
Depth allows you to place a distance between the base URL path and subfolders.
Crawl Depth
cache boolean
Use HTTP caching for the crawl to speed up repeated runs. Default is true.
Enabling caching can save costs when you need to perform transformations on different files or handle various events on a website.
delay number
Add a crawl delay of up to 60 seconds, disabling concurrency. The delay needs to be in milliseconds format. The default is 0, which indicates it is disabled.
Using a delay can help with websites that are set on a cron and do not require immediate data retrieval.
budget object
Object that has paths with a counter for limiting the amount of pages example {"*":1} for only crawling the root page. The wildcard matches all routes and you can set child paths preventing a depth level, example of limiting { "/docs/colors": 10, "/docs/": 100 } which only allows a max of 100 pages if the route matches /docs/:pathname and only 10 pages if it matches /docs/colors/:pathname.
The budget explicitly allows you to set paths and limits for the crawl.
Crawl Budget
scroll number
Infinite scroll the page as new content loads, up to a duration in milliseconds. The duration represents the maximum time you would wait to scroll. You may still need to use the wait_for parameters. You also need to ensure that the request is made using chrome.
Use the wait_for configuration to scroll until and disable_intercept to make sure you get data from the network regardless of hostname.
locale string
The locale to use for request, example en-US.
cookies string
Add HTTP cookies to use for request.
Set the cookie value for pages that use SSR authentication.
stealth boolean
Use stealth mode for headless chrome request to help prevent being blocked. The default is false on chrome.
Set to true to almost guarantee not being detected by anything.
headers object
Forward HTTP headers to use for all request. The object is expected to be a map of key value pairs.
Using HTTP headers can help with authenticated pages that use the authorization header field.
sitemap boolean
Include the sitemap results to crawl. The default is false.
The sitemap allows you to include links that may not be exposed in the HTML.
viewport object
Configure the viewport for chrome. Defaults to 800x600.
If you need to get data from a website as a mobile, set the viewport to a phone device's size ex: 375x414.
encoding string
The type of encoding to use like UTF-8, SHIFT_JIS, or etc.
Perform the encoding on the server when you know in advance the type of website.
wait_for object
The wait_for parameter allows you to specify various waiting conditions for a website operation. If provided, it contains the following sub-parameters:
The key idle_network specifies the conditions to wait for the network to be idle. It can include an optional timeout value.
The key selector specifies the conditions to wait for a particular CSS selector to be found on the page. It includes an optional timeout value, and the CSS selector to wait for.
The key delay specifies a delay to wait for, with an optional timeout value.
The key page_navigationsset to true then waiting for all page navigations will be handled.
If wait_for is not provided, the default behavior is to wait for the network to be idle for 500 milliseconds. All of the durations are capped at capped at 60 seconds.
The values for the timeout duration are in the object shape { secs: 10, nanos: 0 }.
blacklist array
Blacklist a set of paths that you do not want to crawl. You can use Regex patterns to help with the list.
whitelist array
Whitelist a set of paths that you want to crawl, ignoring all other routes that do not match the patterns. You can use regex patterns to help with the list.
subdomains boolean
Allow subdomains to be included. Default is false.
user_agent string
Add a custom HTTP user agent to the request. By default this is set to a random agent.
gpt_config object
Use AI to generate actions to perform during the crawl. You can pass an array for the prompt to chain steps.
Perform event-driven browser actions on a web page or extract data using the gpt_config, you need to use your own API key or have at least $100 in credits to use AI.
fingerprint boolean
Use advanced fingerprint detection for chrome.
Set this value to help crawl when websites require a fingerprint.
storageless boolean
Prevent storing any type of data for the request including storage. Defaults to true unless you have the website already stored.
chunking_alg object
Use a chunking algorithm to segment your content output. Possible values are ByWords, ByLines, ByCharacterLength, and BySentence. The following is an example of the object shape for "chunking_alg": { "type": "bysentence", "value": 2 } which splits the text into an array by every 2 sentences found. This works well when used with markdown or text formats.
The chunking algorithm allows you to prepare the content for the AI quick, without needing to add extra code or loaders.
country_code string
Set a ISO country code for premium proxy connections, this only works for HTTP request at the moment. You need to also use proxy_enabled.
The country code allows you to run requests in regions where access to the website is restricted to within that specific region.
root_selector string
The CSS query selector to use when extracting content from the markup.
Test Query Selector
page_insights boolean
Gather page insights like request duration, accessibility, and other vitals. The default is false and the param requires metadata to be used in conjunction.
Page Insights provides a quick, high-level alternative to Google's PageSpeed.
return_headers boolean
Return the HTTP response headers with the results. Defaults to false unless you have the website already stored with the configuration enabled.
Getting the HTTP headers can help setup authentication flows.
return_cookies boolean
Return the HTTP response cookies with the results. Defaults to false unless you have the website already stored with the configuration enabled.
Getting the HTTP cookies can help setup authentication SSR flows.
respect_robots boolean
Respect the robots.txt file for crawling. Default is true.
If you have trouble crawling a website it may be an issue with the robots.txt file. Setting the value to false could help. Make sure to use this config sparingly.
full_resources boolean
Crawl and download all the resources for a website.
Collect all the content from the website, including assets like images, videos, etc.
request_timeout number
The timeout to use for request. Timeouts can be from 5-60. The default is 30 seconds.
The timeout of the request helps prevent long request times from hanging.
external_domains array
A list of external domains to include into the domain results. You can use regex paths to include the domains. Set one of the array values to * to include all domains.
return_page_links boolean
Return the links found on each page. Defaults to false unless you have the website already stored with the configuration enabled.
Getting the links can help index the reference locations found for the resource.
return_embeddings boolean
Include OpenAI embeddings for title and description. The default is false and the param requires metadata to be used in conjunction.
If you are embedding data, you can use these matrices as staples for most vector baseline operations.
execution_scripts object
Run custom JavaScript on certain paths. This helps with authenticated pages. The request mode to be made through chrome or smart to run. The values for the timeout duration should be in the object shape "/path_or_url": "custom js".
Custom scripts allow you to take control of the browser with events for up to 60 seconds at a time per page.
disable_intercept boolean
Disable request interception when running request as chrome or smart. This may help if you bypass pages that use third-party scripts or external domains. The default setting is false. Note that cost and speed will increase when disabling this feature, as it removes our native Chrome interception.
run_in_background boolean
Run the request in the background. Useful if storing data and wanting to trigger crawls to the dashboard. This has no effect if storageless is set.
skip_config_checks boolean
Skip checking the database for website configuration. This may increase performance for request that are using limit=1. The default is false.
automation_scripts object
Run custom web automated tasks on certain paths. The request mode to be made through chrome or smart to run.
Custom web automation allows you to take control of the browser with events for up to 60 seconds at a time per page.
Below are the available actions for web automation:
Evaluate: Runs custom JavaScript code.
{ "Evaluate": "console.log('Hello, World!');" }
Click: Clicks on an element identified by a CSS selector.
{ "Click": "button#submit" }
Wait: Waits for a specified duration in milliseconds.
{ "Wait": 2000 }
WaitForNavigation: Waits for the next navigation event.
{ "WaitForNavigation": true }
WaitFor: Waits for an element to appear identified by a CSS selector.
{ "WaitFor": "div#content" }
WaitForAndClick: Waits for an element to appear and then clicks on it, identified by a CSS selector.
{ "WaitForAndClick": "button#loadMore" }
ScrollX: Scrolls the screen horizontally by a specified number of pixels.
{ "ScrollX": 100 }
ScrollY: Scrolls the screen vertically by a specified number of pixels.
{ "ScrollY": 200 }
Fill: Fills an input element with a specified value.
{ "Fill": { "selector": "input#name", "value": "John Doe" } }
InfiniteScroll: Scrolls the page until the end for certain duration.
{ "InfiniteScroll": 3000 }
proxy_enabled boolean
Enable premium proxies to prevent detection. Default is false.
Using this configuration can help when request are blocked through the network. This allows more advanced proxies with AI.

Example request

import requests, os

headers = {
    'Authorization': os.getenv("SPIDER_API_KEY"),
    'Content-Type': 'application/json',
}

json_data = {"search":"a sports website","search_limit":3,"limit":25,"return_format":"markdown"}

response = requests.post('https://api.spider.cloud/search', 
  headers=headers, json=json_data)

print(response.json())

Response

[
  {
    "content": "<html>...",
    "error": null,
    "status": 200,
    "credits_used": 2,
    "url": "https://spider.cloud"
  },
  // more content...
]

Crawl websites get links

Start crawling a website(s) to collect links found.

POSThttps://api.spider.cloud/links

Request body

url required string
The URI resource to crawl. This can be a comma split list for multiple urls.
To reduce latency, enhance performance, and save on batch multiple URLs into a single call. For large websites with high page limits, it's best to run requests individually.
Test Url
limit number
The maximum amount of pages allowed to crawl per website. Remove the value or set it to 0 to crawl all pages.
It is better to set a limit upfront on websites where you do not know the size. Re-crawling can effectively use cache to keep costs low as new pages are found.
Crawl Limit
request string
The request type to perform. Possible values are http, chrome, and smart. Use smart to perform HTTP request by default until JavaScript rendering is needed for the HTML.
The request greatly influences how the output is going to look. If the page is server-side rendered, you can stick to the defaults for the most part.
return_format string
The format to return the data in. Possible values are markdown, commonmark, raw, text, and bytes. Use raw to return the default format of the page like HTML etc.
Usually you want to use markdown for LLM processing or text. If you need to store the files without losing any encoding, use the bytes or raw format.
readability boolean
Use readability to pre-process the content for reading. This may drastically improve the content for LLM usage.
This uses the Safari Reader Mode algorithm to extract only important information from the content.
metadata boolean
Collect metadata about the content found like page title, description, keywards and etc. This could help improve AI interoperability. Defaults to false unless you have the website already stored with the configuration enabled.
Using metadata can help extract critical information to use for AI.
anti_bot boolean
Enable anti-bot mode using various techniques to increase the chance of success. Default is false.
This config will attempt to make request resemble a real human. If the request fails on chrome it will retry using a virtual display for the request that is slower difficult to block at the cost of speed.
css_extraction_map object
Use CSS query selectors to scrape contents from the web page. Set the paths and the CSS extraction object map to perform extractions per path or page.
Scrape content using CSS or XPATH selectors to get data. You can scrape using selectors at no extra cost.
store_data boolean
Determine whether to store data. If set this takes precedence over storageless. Defaults to false.
Set to true to collect resources to download and re-use later on.
tld boolean
Allow TLD's to be included. Default is false.
depth number
The crawl limit for maximum depth. If 0, no limit will be applied.
Depth allows you to place a distance between the base URL path and subfolders.
Crawl Depth
cache boolean
Use HTTP caching for the crawl to speed up repeated runs. Default is true.
Enabling caching can save costs when you need to perform transformations on different files or handle various events on a website.
delay number
Add a crawl delay of up to 60 seconds, disabling concurrency. The delay needs to be in milliseconds format. The default is 0, which indicates it is disabled.
Using a delay can help with websites that are set on a cron and do not require immediate data retrieval.
budget object
Object that has paths with a counter for limiting the amount of pages example {"*":1} for only crawling the root page. The wildcard matches all routes and you can set child paths preventing a depth level, example of limiting { "/docs/colors": 10, "/docs/": 100 } which only allows a max of 100 pages if the route matches /docs/:pathname and only 10 pages if it matches /docs/colors/:pathname.
The budget explicitly allows you to set paths and limits for the crawl.
Crawl Budget
scroll number
Infinite scroll the page as new content loads, up to a duration in milliseconds. The duration represents the maximum time you would wait to scroll. You may still need to use the wait_for parameters. You also need to ensure that the request is made using chrome.
Use the wait_for configuration to scroll until and disable_intercept to make sure you get data from the network regardless of hostname.
locale string
The locale to use for request, example en-US.
cookies string
Add HTTP cookies to use for request.
Set the cookie value for pages that use SSR authentication.
stealth boolean
Use stealth mode for headless chrome request to help prevent being blocked. The default is false on chrome.
Set to true to almost guarantee not being detected by anything.
headers object
Forward HTTP headers to use for all request. The object is expected to be a map of key value pairs.
Using HTTP headers can help with authenticated pages that use the authorization header field.
sitemap boolean
Include the sitemap results to crawl. The default is false.
The sitemap allows you to include links that may not be exposed in the HTML.
viewport object
Configure the viewport for chrome. Defaults to 800x600.
If you need to get data from a website as a mobile, set the viewport to a phone device's size ex: 375x414.
encoding string
The type of encoding to use like UTF-8, SHIFT_JIS, or etc.
Perform the encoding on the server when you know in advance the type of website.
wait_for object
The wait_for parameter allows you to specify various waiting conditions for a website operation. If provided, it contains the following sub-parameters:
The key idle_network specifies the conditions to wait for the network to be idle. It can include an optional timeout value.
The key selector specifies the conditions to wait for a particular CSS selector to be found on the page. It includes an optional timeout value, and the CSS selector to wait for.
The key delay specifies a delay to wait for, with an optional timeout value.
The key page_navigationsset to true then waiting for all page navigations will be handled.
If wait_for is not provided, the default behavior is to wait for the network to be idle for 500 milliseconds. All of the durations are capped at capped at 60 seconds.
The values for the timeout duration are in the object shape { secs: 10, nanos: 0 }.
blacklist array
Blacklist a set of paths that you do not want to crawl. You can use Regex patterns to help with the list.
whitelist array
Whitelist a set of paths that you want to crawl, ignoring all other routes that do not match the patterns. You can use regex patterns to help with the list.
subdomains boolean
Allow subdomains to be included. Default is false.
user_agent string
Add a custom HTTP user agent to the request. By default this is set to a random agent.
gpt_config object
Use AI to generate actions to perform during the crawl. You can pass an array for the prompt to chain steps.
Perform event-driven browser actions on a web page or extract data using the gpt_config, you need to use your own API key or have at least $100 in credits to use AI.
fingerprint boolean
Use advanced fingerprint detection for chrome.
Set this value to help crawl when websites require a fingerprint.
storageless boolean
Prevent storing any type of data for the request including storage. Defaults to true unless you have the website already stored.
chunking_alg object
Use a chunking algorithm to segment your content output. Possible values are ByWords, ByLines, ByCharacterLength, and BySentence. The following is an example of the object shape for "chunking_alg": { "type": "bysentence", "value": 2 } which splits the text into an array by every 2 sentences found. This works well when used with markdown or text formats.
The chunking algorithm allows you to prepare the content for the AI quick, without needing to add extra code or loaders.
country_code string
Set a ISO country code for premium proxy connections, this only works for HTTP request at the moment. You need to also use proxy_enabled.
The country code allows you to run requests in regions where access to the website is restricted to within that specific region.
root_selector string
The CSS query selector to use when extracting content from the markup.
Test Query Selector
page_insights boolean
Gather page insights like request duration, accessibility, and other vitals. The default is false and the param requires metadata to be used in conjunction.
Page Insights provides a quick, high-level alternative to Google's PageSpeed.
return_headers boolean
Return the HTTP response headers with the results. Defaults to false unless you have the website already stored with the configuration enabled.
Getting the HTTP headers can help setup authentication flows.
return_cookies boolean
Return the HTTP response cookies with the results. Defaults to false unless you have the website already stored with the configuration enabled.
Getting the HTTP cookies can help setup authentication SSR flows.
respect_robots boolean
Respect the robots.txt file for crawling. Default is true.
If you have trouble crawling a website it may be an issue with the robots.txt file. Setting the value to false could help. Make sure to use this config sparingly.
full_resources boolean
Crawl and download all the resources for a website.
Collect all the content from the website, including assets like images, videos, etc.
request_timeout number
The timeout to use for request. Timeouts can be from 5-60. The default is 30 seconds.
The timeout of the request helps prevent long request times from hanging.
external_domains array
A list of external domains to include into the domain results. You can use regex paths to include the domains. Set one of the array values to * to include all domains.
return_page_links boolean
Return the links found on each page. Defaults to false unless you have the website already stored with the configuration enabled.
Getting the links can help index the reference locations found for the resource.
return_embeddings boolean
Include OpenAI embeddings for title and description. The default is false and the param requires metadata to be used in conjunction.
If you are embedding data, you can use these matrices as staples for most vector baseline operations.
execution_scripts object
Run custom JavaScript on certain paths. This helps with authenticated pages. The request mode to be made through chrome or smart to run. The values for the timeout duration should be in the object shape "/path_or_url": "custom js".
Custom scripts allow you to take control of the browser with events for up to 60 seconds at a time per page.
disable_intercept boolean
Disable request interception when running request as chrome or smart. This may help if you bypass pages that use third-party scripts or external domains. The default setting is false. Note that cost and speed will increase when disabling this feature, as it removes our native Chrome interception.
run_in_background boolean
Run the request in the background. Useful if storing data and wanting to trigger crawls to the dashboard. This has no effect if storageless is set.
skip_config_checks boolean
Skip checking the database for website configuration. This may increase performance for request that are using limit=1. The default is false.
automation_scripts object
Run custom web automated tasks on certain paths. The request mode to be made through chrome or smart to run.
Custom web automation allows you to take control of the browser with events for up to 60 seconds at a time per page.
Below are the available actions for web automation:
Evaluate: Runs custom JavaScript code.
{ "Evaluate": "console.log('Hello, World!');" }
Click: Clicks on an element identified by a CSS selector.
{ "Click": "button#submit" }
Wait: Waits for a specified duration in milliseconds.
{ "Wait": 2000 }
WaitForNavigation: Waits for the next navigation event.
{ "WaitForNavigation": true }
WaitFor: Waits for an element to appear identified by a CSS selector.
{ "WaitFor": "div#content" }
WaitForAndClick: Waits for an element to appear and then clicks on it, identified by a CSS selector.
{ "WaitForAndClick": "button#loadMore" }
ScrollX: Scrolls the screen horizontally by a specified number of pixels.
{ "ScrollX": 100 }
ScrollY: Scrolls the screen vertically by a specified number of pixels.
{ "ScrollY": 200 }
Fill: Fills an input element with a specified value.
{ "Fill": { "selector": "input#name", "value": "John Doe" } }
InfiniteScroll: Scrolls the page until the end for certain duration.
{ "InfiniteScroll": 3000 }
proxy_enabled boolean
Enable premium proxies to prevent detection. Default is false.
Using this configuration can help when request are blocked through the network. This allows more advanced proxies with AI.

Example request

import requests, os

headers = {
    'Authorization': os.getenv("SPIDER_API_KEY"),
    'Content-Type': 'application/json',
}

json_data = {"limit":25,"return_format":"markdown","url":"https://spider.cloud"}

response = requests.post('https://api.spider.cloud/links', 
  headers=headers, json=json_data)

print(response.json())

Response

[
  {
    "url": "https://spider.cloud",
    "status": 200,
    "error": null
  },
  // more content...
]

Screenshot websites

Take screenshots to base64 or binary encoding.

POSThttps://api.spider.cloud/screenshot

Request body

url required string
The URI resource to crawl. This can be a comma split list for multiple urls.
To reduce latency, enhance performance, and save on batch multiple URLs into a single call. For large websites with high page limits, it's best to run requests individually.
Test Url
limit number
The maximum amount of pages allowed to crawl per website. Remove the value or set it to 0 to crawl all pages.
It is better to set a limit upfront on websites where you do not know the size. Re-crawling can effectively use cache to keep costs low as new pages are found.
Crawl Limit
metadata boolean
Collect metadata about the content found like page title, description, keywards and etc. This could help improve AI interoperability. Defaults to false unless you have the website already stored with the configuration enabled.
Using metadata can help extract critical information to use for AI.
anti_bot boolean
Enable anti-bot mode using various techniques to increase the chance of success. Default is false.
This config will attempt to make request resemble a real human. If the request fails on chrome it will retry using a virtual display for the request that is slower difficult to block at the cost of speed.
css_extraction_map object
Use CSS query selectors to scrape contents from the web page. Set the paths and the CSS extraction object map to perform extractions per path or page.
Scrape content using CSS or XPATH selectors to get data. You can scrape using selectors at no extra cost.
store_data boolean
Determine whether to store data. If set this takes precedence over storageless. Defaults to false.
Set to true to collect resources to download and re-use later on.
tld boolean
Allow TLD's to be included. Default is false.
depth number
The crawl limit for maximum depth. If 0, no limit will be applied.
Depth allows you to place a distance between the base URL path and subfolders.
Crawl Depth
cache boolean
Use HTTP caching for the crawl to speed up repeated runs. Default is true.
Enabling caching can save costs when you need to perform transformations on different files or handle various events on a website.
delay number
Add a crawl delay of up to 60 seconds, disabling concurrency. The delay needs to be in milliseconds format. The default is 0, which indicates it is disabled.
Using a delay can help with websites that are set on a cron and do not require immediate data retrieval.
budget object
Object that has paths with a counter for limiting the amount of pages example {"*":1} for only crawling the root page. The wildcard matches all routes and you can set child paths preventing a depth level, example of limiting { "/docs/colors": 10, "/docs/": 100 } which only allows a max of 100 pages if the route matches /docs/:pathname and only 10 pages if it matches /docs/colors/:pathname.
The budget explicitly allows you to set paths and limits for the crawl.
Crawl Budget
scroll number
Infinite scroll the page as new content loads, up to a duration in milliseconds. The duration represents the maximum time you would wait to scroll. You may still need to use the wait_for parameters. You also need to ensure that the request is made using chrome.
Use the wait_for configuration to scroll until and disable_intercept to make sure you get data from the network regardless of hostname.
locale string
The locale to use for request, example en-US.
cookies string
Add HTTP cookies to use for request.
Set the cookie value for pages that use SSR authentication.
stealth boolean
Use stealth mode for headless chrome request to help prevent being blocked. The default is false on chrome.
Set to true to almost guarantee not being detected by anything.
headers object
Forward HTTP headers to use for all request. The object is expected to be a map of key value pairs.
Using HTTP headers can help with authenticated pages that use the authorization header field.
sitemap boolean
Include the sitemap results to crawl. The default is false.
The sitemap allows you to include links that may not be exposed in the HTML.
viewport object
Configure the viewport for chrome. Defaults to 800x600.
If you need to get data from a website as a mobile, set the viewport to a phone device's size ex: 375x414.
encoding string
The type of encoding to use like UTF-8, SHIFT_JIS, or etc.
Perform the encoding on the server when you know in advance the type of website.
wait_for object
The wait_for parameter allows you to specify various waiting conditions for a website operation. If provided, it contains the following sub-parameters:
The key idle_network specifies the conditions to wait for the network to be idle. It can include an optional timeout value.
The key selector specifies the conditions to wait for a particular CSS selector to be found on the page. It includes an optional timeout value, and the CSS selector to wait for.
The key delay specifies a delay to wait for, with an optional timeout value.
The key page_navigationsset to true then waiting for all page navigations will be handled.
If wait_for is not provided, the default behavior is to wait for the network to be idle for 500 milliseconds. All of the durations are capped at capped at 60 seconds.
The values for the timeout duration are in the object shape { secs: 10, nanos: 0 }.
blacklist array
Blacklist a set of paths that you do not want to crawl. You can use Regex patterns to help with the list.
whitelist array
Whitelist a set of paths that you want to crawl, ignoring all other routes that do not match the patterns. You can use regex patterns to help with the list.
subdomains boolean
Allow subdomains to be included. Default is false.
user_agent string
Add a custom HTTP user agent to the request. By default this is set to a random agent.
gpt_config object
Use AI to generate actions to perform during the crawl. You can pass an array for the prompt to chain steps.
Perform event-driven browser actions on a web page or extract data using the gpt_config, you need to use your own API key or have at least $100 in credits to use AI.
fingerprint boolean
Use advanced fingerprint detection for chrome.
Set this value to help crawl when websites require a fingerprint.
storageless boolean
Prevent storing any type of data for the request including storage. Defaults to true unless you have the website already stored.
chunking_alg object
Use a chunking algorithm to segment your content output. Possible values are ByWords, ByLines, ByCharacterLength, and BySentence. The following is an example of the object shape for "chunking_alg": { "type": "bysentence", "value": 2 } which splits the text into an array by every 2 sentences found. This works well when used with markdown or text formats.
The chunking algorithm allows you to prepare the content for the AI quick, without needing to add extra code or loaders.
country_code string
Set a ISO country code for premium proxy connections, this only works for HTTP request at the moment. You need to also use proxy_enabled.
The country code allows you to run requests in regions where access to the website is restricted to within that specific region.
root_selector string
The CSS query selector to use when extracting content from the markup.
Test Query Selector
page_insights boolean
Gather page insights like request duration, accessibility, and other vitals. The default is false and the param requires metadata to be used in conjunction.
Page Insights provides a quick, high-level alternative to Google's PageSpeed.
return_headers boolean
Return the HTTP response headers with the results. Defaults to false unless you have the website already stored with the configuration enabled.
Getting the HTTP headers can help setup authentication flows.
return_cookies boolean
Return the HTTP response cookies with the results. Defaults to false unless you have the website already stored with the configuration enabled.
Getting the HTTP cookies can help setup authentication SSR flows.
respect_robots boolean
Respect the robots.txt file for crawling. Default is true.
If you have trouble crawling a website it may be an issue with the robots.txt file. Setting the value to false could help. Make sure to use this config sparingly.
full_resources boolean
Crawl and download all the resources for a website.
Collect all the content from the website, including assets like images, videos, etc.
request_timeout number
The timeout to use for request. Timeouts can be from 5-60. The default is 30 seconds.
The timeout of the request helps prevent long request times from hanging.
external_domains array
A list of external domains to include into the domain results. You can use regex paths to include the domains. Set one of the array values to * to include all domains.
return_page_links boolean
Return the links found on each page. Defaults to false unless you have the website already stored with the configuration enabled.
Getting the links can help index the reference locations found for the resource.
return_embeddings boolean
Include OpenAI embeddings for title and description. The default is false and the param requires metadata to be used in conjunction.
If you are embedding data, you can use these matrices as staples for most vector baseline operations.
execution_scripts object
Run custom JavaScript on certain paths. This helps with authenticated pages. The request mode to be made through chrome or smart to run. The values for the timeout duration should be in the object shape "/path_or_url": "custom js".
Custom scripts allow you to take control of the browser with events for up to 60 seconds at a time per page.
disable_intercept boolean
Disable request interception when running request as chrome or smart. This may help if you bypass pages that use third-party scripts or external domains. The default setting is false. Note that cost and speed will increase when disabling this feature, as it removes our native Chrome interception.
run_in_background boolean
Run the request in the background. Useful if storing data and wanting to trigger crawls to the dashboard. This has no effect if storageless is set.
skip_config_checks boolean
Skip checking the database for website configuration. This may increase performance for request that are using limit=1. The default is false.
automation_scripts object
Run custom web automated tasks on certain paths. The request mode to be made through chrome or smart to run.
Custom web automation allows you to take control of the browser with events for up to 60 seconds at a time per page.
Below are the available actions for web automation:
Evaluate: Runs custom JavaScript code.
{ "Evaluate": "console.log('Hello, World!');" }
Click: Clicks on an element identified by a CSS selector.
{ "Click": "button#submit" }
Wait: Waits for a specified duration in milliseconds.
{ "Wait": 2000 }
WaitForNavigation: Waits for the next navigation event.
{ "WaitForNavigation": true }
WaitFor: Waits for an element to appear identified by a CSS selector.
{ "WaitFor": "div#content" }
WaitForAndClick: Waits for an element to appear and then clicks on it, identified by a CSS selector.
{ "WaitForAndClick": "button#loadMore" }
ScrollX: Scrolls the screen horizontally by a specified number of pixels.
{ "ScrollX": 100 }
ScrollY: Scrolls the screen vertically by a specified number of pixels.
{ "ScrollY": 200 }
Fill: Fills an input element with a specified value.
{ "Fill": { "selector": "input#name", "value": "John Doe" } }
InfiniteScroll: Scrolls the page until the end for certain duration.
{ "InfiniteScroll": 3000 }
proxy_enabled boolean
Enable premium proxies to prevent detection. Default is false.
Using this configuration can help when request are blocked through the network. This allows more advanced proxies with AI.

Example request

import requests, os

headers = {
    'Authorization': os.getenv("SPIDER_API_KEY"),
    'Content-Type': 'application/json',
}

json_data = {"limit":25,"url":"https://spider.cloud"}

response = requests.post('https://api.spider.cloud/screenshot', 
  headers=headers, json=json_data)

print(response.json())

Response

[
  {
    "content": "base64...",
    "error": null,
    "status": 200,
    "credits_used": 2,
    "url": "https://spider.cloud"
  },
  // more content...
]

Transform HTML

Transform HTML to Markdown or text fast. Each HTML transformation costs 1 credit. You can send up to 10MB of data at once. The transform API is also built into the /crawl endpoint by using return_format.

POSThttps://api.spider.cloud/transform

Request body

return_format string
The format to return the data in. Possible values are markdown, commonmark, raw, text, and bytes. Use raw to return the default format of the page like HTML etc.
Usually you want to use markdown for LLM processing or text. If you need to store the files without losing any encoding, use the bytes or raw format.
readability boolean
Use readability to pre-process the content for reading. This may drastically improve the content for LLM usage.
data object
A list of html data to transform. The object list takes the keys html and url. The url key is optional and only used when the readability is enabled.
Data<html> <head> <title>Example Transform</title> </head> <body> <div> <h1>Example Website</h1> <p>This is some example markup to use to test the transform function.</p> <p><a href="https://spider.cloud/guides">Guides</a></p> </div> </body></html>

Example request

import requests, os

headers = {
    'Authorization': os.getenv("SPIDER_API_KEY"),
    'Content-Type': 'application/json',
}

json_data = {"return_format":"markdown","data":[{"html":"<html>\n<head>\n  <title>Example Transform</title>  \n</head>\n<body>\n<div>\n    <h1>Example Website</h1>\n    <p>This is some example markup to use to test the transform function.</p>\n    <p><a href=\"https://spider.cloud/guides\">Guides</a></p>\n</div>\n</body></html>","url":"https://example.com"}]}

response = requests.post('https://api.spider.cloud/transform', 
  headers=headers, json=json_data)

print(response.json())

Response

{
    "content": [
      "Example Domain
Example Domain
==========
This domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission.
[More information...](https://www.iana.org/domains/example)"
    ],
    "credits_used": 2,
    "error": null,
    "status": 200
  }

Query

Query a resource from the global database instead of crawling a website. 1 credit per successful retrieval.

POSThttps://api.spider.cloud/data/query

Request body

url string
The exact path of the url that you want to get.
Test Url
domain string
The website domain you want to query.
pathname string
The website pathname you want to query.

Example request

import requests, os

headers = {
    'Authorization': os.getenv("SPIDER_API_KEY"),
    'Content-Type': 'application/json',
}

json_data = {"url":"https://spider.cloud"}

response = requests.post('https://api.spider.cloud/data/query', 
  headers=headers, json=json_data)

print(response.json())

Response

{
  "content": "<html>
    <body>
      <div>
          <h1>Example Website</h1>
      </div>
    </body>
  </html>",
  "error": null,
  "status": 200
}

Proxy-Mode
Alpha

Spider also offers a proxy front-end to the service. The Spider proxy will then handle requests just like any standard request, with the option to use high-performance residential proxies at 1TB per/s.

**HTTP address**: proxy.spider.cloud:8888**HTTPS address**: proxy.spider.cloud:8889**Username**: YOUR-API-KEY**Password**: PARAMETERS

Example proxy request

import requests, os


# Proxy configuration
proxies = {
    'http': f"http://{os.getenv('SPIDER_API_KEY')}:request=Raw&premium_proxy=False@proxy.spider.cloud:8888",
    'https': f"https://{os.getenv('SPIDER_API_KEY')}:request=Raw&premium_proxy=False@proxy.spider.cloud:8889"
}

# Function to make a request through the proxy
def get_via_proxy(url):
    try:
        response = requests.get(url, proxies=proxies)
        response.raise_for_status()
        print('Response HTTP Status Code: ', response.status_code)
        print('Response HTTP Response Body: ', response.content)
        return response.text
    except requests.exceptions.RequestException as e:
        print(f"Error: {e}")
        return None

# Example usage
if __name__ == "__main__":
     get_via_proxy("https://www.choosealicense.com")
     get_via_proxy("https://www.choosealicense.com/community")

Pipelines

Create powerful workflows with our pipeline API endpoints. Use AI to extract leads from any website or filter links with prompts with ease.

Crawl websites and extract leads

Start crawling a website(s) to collect all leads utilizing AI. A minimum of $25 in credits is necessary for extraction.

POSThttps://api.spider.cloud/pipeline/extract-contacts

Request body

url required string
The URI resource to crawl. This can be a comma split list for multiple urls.
To reduce latency, enhance performance, and save on batch multiple URLs into a single call. For large websites with high page limits, it's best to run requests individually.
Test Url
limit number
The maximum amount of pages allowed to crawl per website. Remove the value or set it to 0 to crawl all pages.
It is better to set a limit upfront on websites where you do not know the size. Re-crawling can effectively use cache to keep costs low as new pages are found.
Crawl Limit
request string
The request type to perform. Possible values are http, chrome, and smart. Use smart to perform HTTP request by default until JavaScript rendering is needed for the HTML.
The request greatly influences how the output is going to look. If the page is server-side rendered, you can stick to the defaults for the most part.
return_format string
The format to return the data in. Possible values are markdown, commonmark, raw, text, and bytes. Use raw to return the default format of the page like HTML etc.
Usually you want to use markdown for LLM processing or text. If you need to store the files without losing any encoding, use the bytes or raw format.
readability boolean
Use readability to pre-process the content for reading. This may drastically improve the content for LLM usage.
This uses the Safari Reader Mode algorithm to extract only important information from the content.
metadata boolean
Collect metadata about the content found like page title, description, keywards and etc. This could help improve AI interoperability. Defaults to false unless you have the website already stored with the configuration enabled.
Using metadata can help extract critical information to use for AI.
anti_bot boolean
Enable anti-bot mode using various techniques to increase the chance of success. Default is false.
This config will attempt to make request resemble a real human. If the request fails on chrome it will retry using a virtual display for the request that is slower difficult to block at the cost of speed.
css_extraction_map object
Use CSS query selectors to scrape contents from the web page. Set the paths and the CSS extraction object map to perform extractions per path or page.
Scrape content using CSS or XPATH selectors to get data. You can scrape using selectors at no extra cost.
store_data boolean
Determine whether to store data. If set this takes precedence over storageless. Defaults to false.
Set to true to collect resources to download and re-use later on.
tld boolean
Allow TLD's to be included. Default is false.
depth number
The crawl limit for maximum depth. If 0, no limit will be applied.
Depth allows you to place a distance between the base URL path and subfolders.
Crawl Depth
cache boolean
Use HTTP caching for the crawl to speed up repeated runs. Default is true.
Enabling caching can save costs when you need to perform transformations on different files or handle various events on a website.
delay number
Add a crawl delay of up to 60 seconds, disabling concurrency. The delay needs to be in milliseconds format. The default is 0, which indicates it is disabled.
Using a delay can help with websites that are set on a cron and do not require immediate data retrieval.
budget object
Object that has paths with a counter for limiting the amount of pages example {"*":1} for only crawling the root page. The wildcard matches all routes and you can set child paths preventing a depth level, example of limiting { "/docs/colors": 10, "/docs/": 100 } which only allows a max of 100 pages if the route matches /docs/:pathname and only 10 pages if it matches /docs/colors/:pathname.
The budget explicitly allows you to set paths and limits for the crawl.
Crawl Budget
scroll number
Infinite scroll the page as new content loads, up to a duration in milliseconds. The duration represents the maximum time you would wait to scroll. You may still need to use the wait_for parameters. You also need to ensure that the request is made using chrome.
Use the wait_for configuration to scroll until and disable_intercept to make sure you get data from the network regardless of hostname.
locale string
The locale to use for request, example en-US.
cookies string
Add HTTP cookies to use for request.
Set the cookie value for pages that use SSR authentication.
stealth boolean
Use stealth mode for headless chrome request to help prevent being blocked. The default is false on chrome.
Set to true to almost guarantee not being detected by anything.
headers object
Forward HTTP headers to use for all request. The object is expected to be a map of key value pairs.
Using HTTP headers can help with authenticated pages that use the authorization header field.
sitemap boolean
Include the sitemap results to crawl. The default is false.
The sitemap allows you to include links that may not be exposed in the HTML.
viewport object
Configure the viewport for chrome. Defaults to 800x600.
If you need to get data from a website as a mobile, set the viewport to a phone device's size ex: 375x414.
encoding string
The type of encoding to use like UTF-8, SHIFT_JIS, or etc.
Perform the encoding on the server when you know in advance the type of website.
wait_for object
The wait_for parameter allows you to specify various waiting conditions for a website operation. If provided, it contains the following sub-parameters:
The key idle_network specifies the conditions to wait for the network to be idle. It can include an optional timeout value.
The key selector specifies the conditions to wait for a particular CSS selector to be found on the page. It includes an optional timeout value, and the CSS selector to wait for.
The key delay specifies a delay to wait for, with an optional timeout value.
The key page_navigationsset to true then waiting for all page navigations will be handled.
If wait_for is not provided, the default behavior is to wait for the network to be idle for 500 milliseconds. All of the durations are capped at capped at 60 seconds.
The values for the timeout duration are in the object shape { secs: 10, nanos: 0 }.
blacklist array
Blacklist a set of paths that you do not want to crawl. You can use Regex patterns to help with the list.
whitelist array
Whitelist a set of paths that you want to crawl, ignoring all other routes that do not match the patterns. You can use regex patterns to help with the list.
subdomains boolean
Allow subdomains to be included. Default is false.
user_agent string
Add a custom HTTP user agent to the request. By default this is set to a random agent.
gpt_config object
Use AI to generate actions to perform during the crawl. You can pass an array for the prompt to chain steps.
Perform event-driven browser actions on a web page or extract data using the gpt_config, you need to use your own API key or have at least $100 in credits to use AI.
fingerprint boolean
Use advanced fingerprint detection for chrome.
Set this value to help crawl when websites require a fingerprint.
storageless boolean
Prevent storing any type of data for the request including storage. Defaults to true unless you have the website already stored.
chunking_alg object
Use a chunking algorithm to segment your content output. Possible values are ByWords, ByLines, ByCharacterLength, and BySentence. The following is an example of the object shape for "chunking_alg": { "type": "bysentence", "value": 2 } which splits the text into an array by every 2 sentences found. This works well when used with markdown or text formats.
The chunking algorithm allows you to prepare the content for the AI quick, without needing to add extra code or loaders.
country_code string
Set a ISO country code for premium proxy connections, this only works for HTTP request at the moment. You need to also use proxy_enabled.
The country code allows you to run requests in regions where access to the website is restricted to within that specific region.
root_selector string
The CSS query selector to use when extracting content from the markup.
Test Query Selector
page_insights boolean
Gather page insights like request duration, accessibility, and other vitals. The default is false and the param requires metadata to be used in conjunction.
Page Insights provides a quick, high-level alternative to Google's PageSpeed.
return_headers boolean
Return the HTTP response headers with the results. Defaults to false unless you have the website already stored with the configuration enabled.
Getting the HTTP headers can help setup authentication flows.
return_cookies boolean
Return the HTTP response cookies with the results. Defaults to false unless you have the website already stored with the configuration enabled.
Getting the HTTP cookies can help setup authentication SSR flows.
respect_robots boolean
Respect the robots.txt file for crawling. Default is true.
If you have trouble crawling a website it may be an issue with the robots.txt file. Setting the value to false could help. Make sure to use this config sparingly.
full_resources boolean
Crawl and download all the resources for a website.
Collect all the content from the website, including assets like images, videos, etc.
request_timeout number
The timeout to use for request. Timeouts can be from 5-60. The default is 30 seconds.
The timeout of the request helps prevent long request times from hanging.
external_domains array
A list of external domains to include into the domain results. You can use regex paths to include the domains. Set one of the array values to * to include all domains.
return_page_links boolean
Return the links found on each page. Defaults to false unless you have the website already stored with the configuration enabled.
Getting the links can help index the reference locations found for the resource.
return_embeddings boolean
Include OpenAI embeddings for title and description. The default is false and the param requires metadata to be used in conjunction.
If you are embedding data, you can use these matrices as staples for most vector baseline operations.
execution_scripts object
Run custom JavaScript on certain paths. This helps with authenticated pages. The request mode to be made through chrome or smart to run. The values for the timeout duration should be in the object shape "/path_or_url": "custom js".
Custom scripts allow you to take control of the browser with events for up to 60 seconds at a time per page.
disable_intercept boolean
Disable request interception when running request as chrome or smart. This may help if you bypass pages that use third-party scripts or external domains. The default setting is false. Note that cost and speed will increase when disabling this feature, as it removes our native Chrome interception.
run_in_background boolean
Run the request in the background. Useful if storing data and wanting to trigger crawls to the dashboard. This has no effect if storageless is set.
skip_config_checks boolean
Skip checking the database for website configuration. This may increase performance for request that are using limit=1. The default is false.
automation_scripts object
Run custom web automated tasks on certain paths. The request mode to be made through chrome or smart to run.
Custom web automation allows you to take control of the browser with events for up to 60 seconds at a time per page.
Below are the available actions for web automation:
Evaluate: Runs custom JavaScript code.
{ "Evaluate": "console.log('Hello, World!');" }
Click: Clicks on an element identified by a CSS selector.
{ "Click": "button#submit" }
Wait: Waits for a specified duration in milliseconds.
{ "Wait": 2000 }
WaitForNavigation: Waits for the next navigation event.
{ "WaitForNavigation": true }
WaitFor: Waits for an element to appear identified by a CSS selector.
{ "WaitFor": "div#content" }
WaitForAndClick: Waits for an element to appear and then clicks on it, identified by a CSS selector.
{ "WaitForAndClick": "button#loadMore" }
ScrollX: Scrolls the screen horizontally by a specified number of pixels.
{ "ScrollX": 100 }
ScrollY: Scrolls the screen vertically by a specified number of pixels.
{ "ScrollY": 200 }
Fill: Fills an input element with a specified value.
{ "Fill": { "selector": "input#name", "value": "John Doe" } }
InfiniteScroll: Scrolls the page until the end for certain duration.
{ "InfiniteScroll": 3000 }
proxy_enabled boolean
Enable premium proxies to prevent detection. Default is false.
Using this configuration can help when request are blocked through the network. This allows more advanced proxies with AI.

Example request

import requests, os

headers = {
    'Authorization': os.getenv("SPIDER_API_KEY"),
    'Content-Type': 'application/jsonl',
}

json_data = {"limit":25,"return_format":"markdown","url":"https://spider.cloud"}

response = requests.post('https://api.spider.cloud/pipeline/extract-contacts', 
  headers=headers, json=json_data)

print(response.json())

Response

[
  {
    "content": [{
      "full_name": "John Doe",
      "email": "johndoe@gmail.com",
      "phone": "555-555-555",
      "title": "Baker"
    }],
    "error": null,
    "status": 200,
    "credits_used": 2,
    "url": "https://spider.cloud"
  },
  // more content...
]

Label website

Crawl a website and accurately categorize it using AI.

POSThttps://api.spider.cloud/pipeline/label

Request body

url required string
The URI resource to crawl. This can be a comma split list for multiple urls.
To reduce latency, enhance performance, and save on batch multiple URLs into a single call. For large websites with high page limits, it's best to run requests individually.
Test Url
limit number
The maximum amount of pages allowed to crawl per website. Remove the value or set it to 0 to crawl all pages.
It is better to set a limit upfront on websites where you do not know the size. Re-crawling can effectively use cache to keep costs low as new pages are found.
Crawl Limit
request string
The request type to perform. Possible values are http, chrome, and smart. Use smart to perform HTTP request by default until JavaScript rendering is needed for the HTML.
The request greatly influences how the output is going to look. If the page is server-side rendered, you can stick to the defaults for the most part.
return_format string
The format to return the data in. Possible values are markdown, commonmark, raw, text, and bytes. Use raw to return the default format of the page like HTML etc.
Usually you want to use markdown for LLM processing or text. If you need to store the files without losing any encoding, use the bytes or raw format.
readability boolean
Use readability to pre-process the content for reading. This may drastically improve the content for LLM usage.
This uses the Safari Reader Mode algorithm to extract only important information from the content.
metadata boolean
Collect metadata about the content found like page title, description, keywards and etc. This could help improve AI interoperability. Defaults to false unless you have the website already stored with the configuration enabled.
Using metadata can help extract critical information to use for AI.
anti_bot boolean
Enable anti-bot mode using various techniques to increase the chance of success. Default is false.
This config will attempt to make request resemble a real human. If the request fails on chrome it will retry using a virtual display for the request that is slower difficult to block at the cost of speed.
css_extraction_map object
Use CSS query selectors to scrape contents from the web page. Set the paths and the CSS extraction object map to perform extractions per path or page.
Scrape content using CSS or XPATH selectors to get data. You can scrape using selectors at no extra cost.
store_data boolean
Determine whether to store data. If set this takes precedence over storageless. Defaults to false.
Set to true to collect resources to download and re-use later on.
tld boolean
Allow TLD's to be included. Default is false.
depth number
The crawl limit for maximum depth. If 0, no limit will be applied.
Depth allows you to place a distance between the base URL path and subfolders.
Crawl Depth
cache boolean
Use HTTP caching for the crawl to speed up repeated runs. Default is true.
Enabling caching can save costs when you need to perform transformations on different files or handle various events on a website.
delay number
Add a crawl delay of up to 60 seconds, disabling concurrency. The delay needs to be in milliseconds format. The default is 0, which indicates it is disabled.
Using a delay can help with websites that are set on a cron and do not require immediate data retrieval.
budget object
Object that has paths with a counter for limiting the amount of pages example {"*":1} for only crawling the root page. The wildcard matches all routes and you can set child paths preventing a depth level, example of limiting { "/docs/colors": 10, "/docs/": 100 } which only allows a max of 100 pages if the route matches /docs/:pathname and only 10 pages if it matches /docs/colors/:pathname.
The budget explicitly allows you to set paths and limits for the crawl.
Crawl Budget
scroll number
Infinite scroll the page as new content loads, up to a duration in milliseconds. The duration represents the maximum time you would wait to scroll. You may still need to use the wait_for parameters. You also need to ensure that the request is made using chrome.
Use the wait_for configuration to scroll until and disable_intercept to make sure you get data from the network regardless of hostname.
locale string
The locale to use for request, example en-US.
cookies string
Add HTTP cookies to use for request.
Set the cookie value for pages that use SSR authentication.
stealth boolean
Use stealth mode for headless chrome request to help prevent being blocked. The default is false on chrome.
Set to true to almost guarantee not being detected by anything.
headers object
Forward HTTP headers to use for all request. The object is expected to be a map of key value pairs.
Using HTTP headers can help with authenticated pages that use the authorization header field.
sitemap boolean
Include the sitemap results to crawl. The default is false.
The sitemap allows you to include links that may not be exposed in the HTML.
viewport object
Configure the viewport for chrome. Defaults to 800x600.
If you need to get data from a website as a mobile, set the viewport to a phone device's size ex: 375x414.
encoding string
The type of encoding to use like UTF-8, SHIFT_JIS, or etc.
Perform the encoding on the server when you know in advance the type of website.
wait_for object
The wait_for parameter allows you to specify various waiting conditions for a website operation. If provided, it contains the following sub-parameters:
The key idle_network specifies the conditions to wait for the network to be idle. It can include an optional timeout value.
The key selector specifies the conditions to wait for a particular CSS selector to be found on the page. It includes an optional timeout value, and the CSS selector to wait for.
The key delay specifies a delay to wait for, with an optional timeout value.
The key page_navigationsset to true then waiting for all page navigations will be handled.
If wait_for is not provided, the default behavior is to wait for the network to be idle for 500 milliseconds. All of the durations are capped at capped at 60 seconds.
The values for the timeout duration are in the object shape { secs: 10, nanos: 0 }.
blacklist array
Blacklist a set of paths that you do not want to crawl. You can use Regex patterns to help with the list.
whitelist array
Whitelist a set of paths that you want to crawl, ignoring all other routes that do not match the patterns. You can use regex patterns to help with the list.
subdomains boolean
Allow subdomains to be included. Default is false.
user_agent string
Add a custom HTTP user agent to the request. By default this is set to a random agent.
gpt_config object
Use AI to generate actions to perform during the crawl. You can pass an array for the prompt to chain steps.
Perform event-driven browser actions on a web page or extract data using the gpt_config, you need to use your own API key or have at least $100 in credits to use AI.
fingerprint boolean
Use advanced fingerprint detection for chrome.
Set this value to help crawl when websites require a fingerprint.
storageless boolean
Prevent storing any type of data for the request including storage. Defaults to true unless you have the website already stored.
chunking_alg object
Use a chunking algorithm to segment your content output. Possible values are ByWords, ByLines, ByCharacterLength, and BySentence. The following is an example of the object shape for "chunking_alg": { "type": "bysentence", "value": 2 } which splits the text into an array by every 2 sentences found. This works well when used with markdown or text formats.
The chunking algorithm allows you to prepare the content for the AI quick, without needing to add extra code or loaders.
country_code string
Set a ISO country code for premium proxy connections, this only works for HTTP request at the moment. You need to also use proxy_enabled.
The country code allows you to run requests in regions where access to the website is restricted to within that specific region.
root_selector string
The CSS query selector to use when extracting content from the markup.
Test Query Selector
page_insights boolean
Gather page insights like request duration, accessibility, and other vitals. The default is false and the param requires metadata to be used in conjunction.
Page Insights provides a quick, high-level alternative to Google's PageSpeed.
return_headers boolean
Return the HTTP response headers with the results. Defaults to false unless you have the website already stored with the configuration enabled.
Getting the HTTP headers can help setup authentication flows.
return_cookies boolean
Return the HTTP response cookies with the results. Defaults to false unless you have the website already stored with the configuration enabled.
Getting the HTTP cookies can help setup authentication SSR flows.
respect_robots boolean
Respect the robots.txt file for crawling. Default is true.
If you have trouble crawling a website it may be an issue with the robots.txt file. Setting the value to false could help. Make sure to use this config sparingly.
full_resources boolean
Crawl and download all the resources for a website.
Collect all the content from the website, including assets like images, videos, etc.
request_timeout number
The timeout to use for request. Timeouts can be from 5-60. The default is 30 seconds.
The timeout of the request helps prevent long request times from hanging.
external_domains array
A list of external domains to include into the domain results. You can use regex paths to include the domains. Set one of the array values to * to include all domains.
return_page_links boolean
Return the links found on each page. Defaults to false unless you have the website already stored with the configuration enabled.
Getting the links can help index the reference locations found for the resource.
return_embeddings boolean
Include OpenAI embeddings for title and description. The default is false and the param requires metadata to be used in conjunction.
If you are embedding data, you can use these matrices as staples for most vector baseline operations.
execution_scripts object
Run custom JavaScript on certain paths. This helps with authenticated pages. The request mode to be made through chrome or smart to run. The values for the timeout duration should be in the object shape "/path_or_url": "custom js".
Custom scripts allow you to take control of the browser with events for up to 60 seconds at a time per page.
disable_intercept boolean
Disable request interception when running request as chrome or smart. This may help if you bypass pages that use third-party scripts or external domains. The default setting is false. Note that cost and speed will increase when disabling this feature, as it removes our native Chrome interception.
run_in_background boolean
Run the request in the background. Useful if storing data and wanting to trigger crawls to the dashboard. This has no effect if storageless is set.
skip_config_checks boolean
Skip checking the database for website configuration. This may increase performance for request that are using limit=1. The default is false.
automation_scripts object
Run custom web automated tasks on certain paths. The request mode to be made through chrome or smart to run.
Custom web automation allows you to take control of the browser with events for up to 60 seconds at a time per page.
Below are the available actions for web automation:
Evaluate: Runs custom JavaScript code.
{ "Evaluate": "console.log('Hello, World!');" }
Click: Clicks on an element identified by a CSS selector.
{ "Click": "button#submit" }
Wait: Waits for a specified duration in milliseconds.
{ "Wait": 2000 }
WaitForNavigation: Waits for the next navigation event.
{ "WaitForNavigation": true }
WaitFor: Waits for an element to appear identified by a CSS selector.
{ "WaitFor": "div#content" }
WaitForAndClick: Waits for an element to appear and then clicks on it, identified by a CSS selector.
{ "WaitForAndClick": "button#loadMore" }
ScrollX: Scrolls the screen horizontally by a specified number of pixels.
{ "ScrollX": 100 }
ScrollY: Scrolls the screen vertically by a specified number of pixels.
{ "ScrollY": 200 }
Fill: Fills an input element with a specified value.
{ "Fill": { "selector": "input#name", "value": "John Doe" } }
InfiniteScroll: Scrolls the page until the end for certain duration.
{ "InfiniteScroll": 3000 }
proxy_enabled boolean
Enable premium proxies to prevent detection. Default is false.
Using this configuration can help when request are blocked through the network. This allows more advanced proxies with AI.

Example request

import requests, os

headers = {
    'Authorization': os.getenv("SPIDER_API_KEY"),
    'Content-Type': 'application/jsonl',
}

json_data = {"limit":25,"return_format":"markdown","url":"https://spider.cloud"}

response = requests.post('https://api.spider.cloud/pipeline/label', 
  headers=headers, json=json_data)

print(response.json())

Response

[
  {
    "content": ["Government"],
    "error": null,
    "status": 200,
    "credits_used": 2,
    "url": "https://spider.cloud"
  },
  // more content...
]

Crawl websites from text

Crawl website(s) found from raw text or markdown.

POSThttps://api.spider.cloud/pipeline/crawl-text

Request body

text required string
The text string to extract urls from. The max limit for the text is 10mb.
limit number
The maximum amount of pages allowed to crawl per website. Remove the value or set it to 0 to crawl all pages.
It is better to set a limit upfront on websites where you do not know the size. Re-crawling can effectively use cache to keep costs low as new pages are found.
Crawl Limit
request string
The request type to perform. Possible values are http, chrome, and smart. Use smart to perform HTTP request by default until JavaScript rendering is needed for the HTML.
The request greatly influences how the output is going to look. If the page is server-side rendered, you can stick to the defaults for the most part.
return_format string
The format to return the data in. Possible values are markdown, commonmark, raw, text, and bytes. Use raw to return the default format of the page like HTML etc.
Usually you want to use markdown for LLM processing or text. If you need to store the files without losing any encoding, use the bytes or raw format.
readability boolean
Use readability to pre-process the content for reading. This may drastically improve the content for LLM usage.
This uses the Safari Reader Mode algorithm to extract only important information from the content.
metadata boolean
Collect metadata about the content found like page title, description, keywards and etc. This could help improve AI interoperability. Defaults to false unless you have the website already stored with the configuration enabled.
Using metadata can help extract critical information to use for AI.
anti_bot boolean
Enable anti-bot mode using various techniques to increase the chance of success. Default is false.
This config will attempt to make request resemble a real human. If the request fails on chrome it will retry using a virtual display for the request that is slower difficult to block at the cost of speed.
css_extraction_map object
Use CSS query selectors to scrape contents from the web page. Set the paths and the CSS extraction object map to perform extractions per path or page.
Scrape content using CSS or XPATH selectors to get data. You can scrape using selectors at no extra cost.
store_data boolean
Determine whether to store data. If set this takes precedence over storageless. Defaults to false.
Set to true to collect resources to download and re-use later on.
tld boolean
Allow TLD's to be included. Default is false.
url string
The URI resource to crawl. This can be a comma split list for multiple urls.
To reduce latency, enhance performance, and save on batch multiple URLs into a single call. For large websites with high page limits, it's best to run requests individually.
Test Url
depth number
The crawl limit for maximum depth. If 0, no limit will be applied.
Depth allows you to place a distance between the base URL path and subfolders.
Crawl Depth
cache boolean
Use HTTP caching for the crawl to speed up repeated runs. Default is true.
Enabling caching can save costs when you need to perform transformations on different files or handle various events on a website.
delay number
Add a crawl delay of up to 60 seconds, disabling concurrency. The delay needs to be in milliseconds format. The default is 0, which indicates it is disabled.
Using a delay can help with websites that are set on a cron and do not require immediate data retrieval.
budget object
Object that has paths with a counter for limiting the amount of pages example {"*":1} for only crawling the root page. The wildcard matches all routes and you can set child paths preventing a depth level, example of limiting { "/docs/colors": 10, "/docs/": 100 } which only allows a max of 100 pages if the route matches /docs/:pathname and only 10 pages if it matches /docs/colors/:pathname.
The budget explicitly allows you to set paths and limits for the crawl.
Crawl Budget
scroll number
Infinite scroll the page as new content loads, up to a duration in milliseconds. The duration represents the maximum time you would wait to scroll. You may still need to use the wait_for parameters. You also need to ensure that the request is made using chrome.
Use the wait_for configuration to scroll until and disable_intercept to make sure you get data from the network regardless of hostname.
locale string
The locale to use for request, example en-US.
cookies string
Add HTTP cookies to use for request.
Set the cookie value for pages that use SSR authentication.
stealth boolean
Use stealth mode for headless chrome request to help prevent being blocked. The default is false on chrome.
Set to true to almost guarantee not being detected by anything.
headers object
Forward HTTP headers to use for all request. The object is expected to be a map of key value pairs.
Using HTTP headers can help with authenticated pages that use the authorization header field.
sitemap boolean
Include the sitemap results to crawl. The default is false.
The sitemap allows you to include links that may not be exposed in the HTML.
viewport object
Configure the viewport for chrome. Defaults to 800x600.
If you need to get data from a website as a mobile, set the viewport to a phone device's size ex: 375x414.
encoding string
The type of encoding to use like UTF-8, SHIFT_JIS, or etc.
Perform the encoding on the server when you know in advance the type of website.
wait_for object
The wait_for parameter allows you to specify various waiting conditions for a website operation. If provided, it contains the following sub-parameters:
The key idle_network specifies the conditions to wait for the network to be idle. It can include an optional timeout value.
The key selector specifies the conditions to wait for a particular CSS selector to be found on the page. It includes an optional timeout value, and the CSS selector to wait for.
The key delay specifies a delay to wait for, with an optional timeout value.
The key page_navigationsset to true then waiting for all page navigations will be handled.
If wait_for is not provided, the default behavior is to wait for the network to be idle for 500 milliseconds. All of the durations are capped at capped at 60 seconds.
The values for the timeout duration are in the object shape { secs: 10, nanos: 0 }.
blacklist array
Blacklist a set of paths that you do not want to crawl. You can use Regex patterns to help with the list.
whitelist array
Whitelist a set of paths that you want to crawl, ignoring all other routes that do not match the patterns. You can use regex patterns to help with the list.
subdomains boolean
Allow subdomains to be included. Default is false.
user_agent string
Add a custom HTTP user agent to the request. By default this is set to a random agent.
gpt_config object
Use AI to generate actions to perform during the crawl. You can pass an array for the prompt to chain steps.
Perform event-driven browser actions on a web page or extract data using the gpt_config, you need to use your own API key or have at least $100 in credits to use AI.
fingerprint boolean
Use advanced fingerprint detection for chrome.
Set this value to help crawl when websites require a fingerprint.
storageless boolean
Prevent storing any type of data for the request including storage. Defaults to true unless you have the website already stored.
chunking_alg object
Use a chunking algorithm to segment your content output. Possible values are ByWords, ByLines, ByCharacterLength, and BySentence. The following is an example of the object shape for "chunking_alg": { "type": "bysentence", "value": 2 } which splits the text into an array by every 2 sentences found. This works well when used with markdown or text formats.
The chunking algorithm allows you to prepare the content for the AI quick, without needing to add extra code or loaders.
country_code string
Set a ISO country code for premium proxy connections, this only works for HTTP request at the moment. You need to also use proxy_enabled.
The country code allows you to run requests in regions where access to the website is restricted to within that specific region.
root_selector string
The CSS query selector to use when extracting content from the markup.
Test Query Selector
page_insights boolean
Gather page insights like request duration, accessibility, and other vitals. The default is false and the param requires metadata to be used in conjunction.
Page Insights provides a quick, high-level alternative to Google's PageSpeed.
return_headers boolean
Return the HTTP response headers with the results. Defaults to false unless you have the website already stored with the configuration enabled.
Getting the HTTP headers can help setup authentication flows.
return_cookies boolean
Return the HTTP response cookies with the results. Defaults to false unless you have the website already stored with the configuration enabled.
Getting the HTTP cookies can help setup authentication SSR flows.
respect_robots boolean
Respect the robots.txt file for crawling. Default is true.
If you have trouble crawling a website it may be an issue with the robots.txt file. Setting the value to false could help. Make sure to use this config sparingly.
full_resources boolean
Crawl and download all the resources for a website.
Collect all the content from the website, including assets like images, videos, etc.
request_timeout number
The timeout to use for request. Timeouts can be from 5-60. The default is 30 seconds.
The timeout of the request helps prevent long request times from hanging.
external_domains array
A list of external domains to include into the domain results. You can use regex paths to include the domains. Set one of the array values to * to include all domains.
return_page_links boolean
Return the links found on each page. Defaults to false unless you have the website already stored with the configuration enabled.
Getting the links can help index the reference locations found for the resource.
return_embeddings boolean
Include OpenAI embeddings for title and description. The default is false and the param requires metadata to be used in conjunction.
If you are embedding data, you can use these matrices as staples for most vector baseline operations.
execution_scripts object
Run custom JavaScript on certain paths. This helps with authenticated pages. The request mode to be made through chrome or smart to run. The values for the timeout duration should be in the object shape "/path_or_url": "custom js".
Custom scripts allow you to take control of the browser with events for up to 60 seconds at a time per page.
disable_intercept boolean
Disable request interception when running request as chrome or smart. This may help if you bypass pages that use third-party scripts or external domains. The default setting is false. Note that cost and speed will increase when disabling this feature, as it removes our native Chrome interception.
run_in_background boolean
Run the request in the background. Useful if storing data and wanting to trigger crawls to the dashboard. This has no effect if storageless is set.
skip_config_checks boolean
Skip checking the database for website configuration. This may increase performance for request that are using limit=1. The default is false.
automation_scripts object
Run custom web automated tasks on certain paths. The request mode to be made through chrome or smart to run.
Custom web automation allows you to take control of the browser with events for up to 60 seconds at a time per page.
Below are the available actions for web automation:
Evaluate: Runs custom JavaScript code.
{ "Evaluate": "console.log('Hello, World!');" }
Click: Clicks on an element identified by a CSS selector.
{ "Click": "button#submit" }
Wait: Waits for a specified duration in milliseconds.
{ "Wait": 2000 }
WaitForNavigation: Waits for the next navigation event.
{ "WaitForNavigation": true }
WaitFor: Waits for an element to appear identified by a CSS selector.
{ "WaitFor": "div#content" }
WaitForAndClick: Waits for an element to appear and then clicks on it, identified by a CSS selector.
{ "WaitForAndClick": "button#loadMore" }
ScrollX: Scrolls the screen horizontally by a specified number of pixels.
{ "ScrollX": 100 }
ScrollY: Scrolls the screen vertically by a specified number of pixels.
{ "ScrollY": 200 }
Fill: Fills an input element with a specified value.
{ "Fill": { "selector": "input#name", "value": "John Doe" } }
InfiniteScroll: Scrolls the page until the end for certain duration.
{ "InfiniteScroll": 3000 }
proxy_enabled boolean
Enable premium proxies to prevent detection. Default is false.
Using this configuration can help when request are blocked through the network. This allows more advanced proxies with AI.

Example request

import requests, os

headers = {
    'Authorization': os.getenv("SPIDER_API_KEY"),
    'Content-Type': 'application/jsonl',
}

json_data = {"text":"Check this link: https://example.com and email to example@email.com","limit":25,"return_format":"markdown"}

response = requests.post('https://api.spider.cloud/pipeline/crawl-text', 
  headers=headers, json=json_data)

print(response.json())

Response

[
  {
    "content": "<html>...",
    "error": null,
    "status": 200,
    "credits_used": 2,
    "url": "https://spider.cloud"
  },
  // more content...
]

Filter links

Filter links using AI and advanced metadata.

POSThttps://api.spider.cloud/pipeline/filter-links

Request body

url required array
The links to filter.
You can pass up to 4k tokens for the links and prompt.
Test Url
model string
The type of AI model to use like gpt-4o, gpt-4o-mini, gpt-4-1106-preview or gpt-3.5-turbo etc. Defaults to gpt-4-mini.
custom_func string
A custom function to run before processing. Contact us for getting a custom function tailored to you.
custom_prompt string
A custom prompt to pass in to the model.

Example request

import requests, os

headers = {
    'Authorization': os.getenv("SPIDER_API_KEY"),
    'Content-Type': 'application/jsonl',
}

json_data = {"limit":25,"return_format":"markdown","url":"https://spider.cloud"}

response = requests.post('https://api.spider.cloud/pipeline/filter-links', 
  headers=headers, json=json_data)

print(response.json())

Response

{
    "content": [
        {
            "relevant_urls": [
                "https://spider.cloud",
                "https://foodnetwork.com"
            ]
        }
    ],
    "credits_used": 52,
    "error": "",
    "status": 200
}

Questions and Answers

Get a question-and-answer list for a website based on any inquiry.

POSThttps://api.spider.cloud/pipeline/extract-qa

Request body

url required string
The URI resource to crawl. This can be a comma split list for multiple urls.
To reduce latency, enhance performance, and save on batch multiple URLs into a single call. For large websites with high page limits, it's best to run requests individually.
Test Url
limit number
The maximum amount of pages allowed to crawl per website. Remove the value or set it to 0 to crawl all pages.
It is better to set a limit upfront on websites where you do not know the size. Re-crawling can effectively use cache to keep costs low as new pages are found.
Crawl Limit
request string
The request type to perform. Possible values are http, chrome, and smart. Use smart to perform HTTP request by default until JavaScript rendering is needed for the HTML.
The request greatly influences how the output is going to look. If the page is server-side rendered, you can stick to the defaults for the most part.
return_format string
The format to return the data in. Possible values are markdown, commonmark, raw, text, and bytes. Use raw to return the default format of the page like HTML etc.
Usually you want to use markdown for LLM processing or text. If you need to store the files without losing any encoding, use the bytes or raw format.
readability boolean
Use readability to pre-process the content for reading. This may drastically improve the content for LLM usage.
This uses the Safari Reader Mode algorithm to extract only important information from the content.
metadata boolean
Collect metadata about the content found like page title, description, keywards and etc. This could help improve AI interoperability. Defaults to false unless you have the website already stored with the configuration enabled.
Using metadata can help extract critical information to use for AI.
anti_bot boolean
Enable anti-bot mode using various techniques to increase the chance of success. Default is false.
This config will attempt to make request resemble a real human. If the request fails on chrome it will retry using a virtual display for the request that is slower difficult to block at the cost of speed.
css_extraction_map object
Use CSS query selectors to scrape contents from the web page. Set the paths and the CSS extraction object map to perform extractions per path or page.
Scrape content using CSS or XPATH selectors to get data. You can scrape using selectors at no extra cost.
store_data boolean
Determine whether to store data. If set this takes precedence over storageless. Defaults to false.
Set to true to collect resources to download and re-use later on.
tld boolean
Allow TLD's to be included. Default is false.
depth number
The crawl limit for maximum depth. If 0, no limit will be applied.
Depth allows you to place a distance between the base URL path and subfolders.
Crawl Depth
cache boolean
Use HTTP caching for the crawl to speed up repeated runs. Default is true.
Enabling caching can save costs when you need to perform transformations on different files or handle various events on a website.
delay number
Add a crawl delay of up to 60 seconds, disabling concurrency. The delay needs to be in milliseconds format. The default is 0, which indicates it is disabled.
Using a delay can help with websites that are set on a cron and do not require immediate data retrieval.
budget object
Object that has paths with a counter for limiting the amount of pages example {"*":1} for only crawling the root page. The wildcard matches all routes and you can set child paths preventing a depth level, example of limiting { "/docs/colors": 10, "/docs/": 100 } which only allows a max of 100 pages if the route matches /docs/:pathname and only 10 pages if it matches /docs/colors/:pathname.
The budget explicitly allows you to set paths and limits for the crawl.
Crawl Budget
scroll number
Infinite scroll the page as new content loads, up to a duration in milliseconds. The duration represents the maximum time you would wait to scroll. You may still need to use the wait_for parameters. You also need to ensure that the request is made using chrome.
Use the wait_for configuration to scroll until and disable_intercept to make sure you get data from the network regardless of hostname.
locale string
The locale to use for request, example en-US.
cookies string
Add HTTP cookies to use for request.
Set the cookie value for pages that use SSR authentication.
stealth boolean
Use stealth mode for headless chrome request to help prevent being blocked. The default is false on chrome.
Set to true to almost guarantee not being detected by anything.
headers object
Forward HTTP headers to use for all request. The object is expected to be a map of key value pairs.
Using HTTP headers can help with authenticated pages that use the authorization header field.
sitemap boolean
Include the sitemap results to crawl. The default is false.
The sitemap allows you to include links that may not be exposed in the HTML.
viewport object
Configure the viewport for chrome. Defaults to 800x600.
If you need to get data from a website as a mobile, set the viewport to a phone device's size ex: 375x414.
encoding string
The type of encoding to use like UTF-8, SHIFT_JIS, or etc.
Perform the encoding on the server when you know in advance the type of website.
wait_for object
The wait_for parameter allows you to specify various waiting conditions for a website operation. If provided, it contains the following sub-parameters:
The key idle_network specifies the conditions to wait for the network to be idle. It can include an optional timeout value.
The key selector specifies the conditions to wait for a particular CSS selector to be found on the page. It includes an optional timeout value, and the CSS selector to wait for.
The key delay specifies a delay to wait for, with an optional timeout value.
The key page_navigationsset to true then waiting for all page navigations will be handled.
If wait_for is not provided, the default behavior is to wait for the network to be idle for 500 milliseconds. All of the durations are capped at capped at 60 seconds.
The values for the timeout duration are in the object shape { secs: 10, nanos: 0 }.
blacklist array
Blacklist a set of paths that you do not want to crawl. You can use Regex patterns to help with the list.
whitelist array
Whitelist a set of paths that you want to crawl, ignoring all other routes that do not match the patterns. You can use regex patterns to help with the list.
subdomains boolean
Allow subdomains to be included. Default is false.
user_agent string
Add a custom HTTP user agent to the request. By default this is set to a random agent.
gpt_config object
Use AI to generate actions to perform during the crawl. You can pass an array for the prompt to chain steps.
Perform event-driven browser actions on a web page or extract data using the gpt_config, you need to use your own API key or have at least $100 in credits to use AI.
fingerprint boolean
Use advanced fingerprint detection for chrome.
Set this value to help crawl when websites require a fingerprint.
storageless boolean
Prevent storing any type of data for the request including storage. Defaults to true unless you have the website already stored.
chunking_alg object
Use a chunking algorithm to segment your content output. Possible values are ByWords, ByLines, ByCharacterLength, and BySentence. The following is an example of the object shape for "chunking_alg": { "type": "bysentence", "value": 2 } which splits the text into an array by every 2 sentences found. This works well when used with markdown or text formats.
The chunking algorithm allows you to prepare the content for the AI quick, without needing to add extra code or loaders.
country_code string
Set a ISO country code for premium proxy connections, this only works for HTTP request at the moment. You need to also use proxy_enabled.
The country code allows you to run requests in regions where access to the website is restricted to within that specific region.
root_selector string
The CSS query selector to use when extracting content from the markup.
Test Query Selector
page_insights boolean
Gather page insights like request duration, accessibility, and other vitals. The default is false and the param requires metadata to be used in conjunction.
Page Insights provides a quick, high-level alternative to Google's PageSpeed.
return_headers boolean
Return the HTTP response headers with the results. Defaults to false unless you have the website already stored with the configuration enabled.
Getting the HTTP headers can help setup authentication flows.
return_cookies boolean
Return the HTTP response cookies with the results. Defaults to false unless you have the website already stored with the configuration enabled.
Getting the HTTP cookies can help setup authentication SSR flows.
respect_robots boolean
Respect the robots.txt file for crawling. Default is true.
If you have trouble crawling a website it may be an issue with the robots.txt file. Setting the value to false could help. Make sure to use this config sparingly.
full_resources boolean
Crawl and download all the resources for a website.
Collect all the content from the website, including assets like images, videos, etc.
request_timeout number
The timeout to use for request. Timeouts can be from 5-60. The default is 30 seconds.
The timeout of the request helps prevent long request times from hanging.
external_domains array
A list of external domains to include into the domain results. You can use regex paths to include the domains. Set one of the array values to * to include all domains.
return_page_links boolean
Return the links found on each page. Defaults to false unless you have the website already stored with the configuration enabled.
Getting the links can help index the reference locations found for the resource.
return_embeddings boolean
Include OpenAI embeddings for title and description. The default is false and the param requires metadata to be used in conjunction.
If you are embedding data, you can use these matrices as staples for most vector baseline operations.
execution_scripts object
Run custom JavaScript on certain paths. This helps with authenticated pages. The request mode to be made through chrome or smart to run. The values for the timeout duration should be in the object shape "/path_or_url": "custom js".
Custom scripts allow you to take control of the browser with events for up to 60 seconds at a time per page.
disable_intercept boolean
Disable request interception when running request as chrome or smart. This may help if you bypass pages that use third-party scripts or external domains. The default setting is false. Note that cost and speed will increase when disabling this feature, as it removes our native Chrome interception.
run_in_background boolean
Run the request in the background. Useful if storing data and wanting to trigger crawls to the dashboard. This has no effect if storageless is set.
skip_config_checks boolean
Skip checking the database for website configuration. This may increase performance for request that are using limit=1. The default is false.
automation_scripts object
Run custom web automated tasks on certain paths. The request mode to be made through chrome or smart to run.
Custom web automation allows you to take control of the browser with events for up to 60 seconds at a time per page.
Below are the available actions for web automation:
Evaluate: Runs custom JavaScript code.
{ "Evaluate": "console.log('Hello, World!');" }
Click: Clicks on an element identified by a CSS selector.
{ "Click": "button#submit" }
Wait: Waits for a specified duration in milliseconds.
{ "Wait": 2000 }
WaitForNavigation: Waits for the next navigation event.
{ "WaitForNavigation": true }
WaitFor: Waits for an element to appear identified by a CSS selector.
{ "WaitFor": "div#content" }
WaitForAndClick: Waits for an element to appear and then clicks on it, identified by a CSS selector.
{ "WaitForAndClick": "button#loadMore" }
ScrollX: Scrolls the screen horizontally by a specified number of pixels.
{ "ScrollX": 100 }
ScrollY: Scrolls the screen vertically by a specified number of pixels.
{ "ScrollY": 200 }
Fill: Fills an input element with a specified value.
{ "Fill": { "selector": "input#name", "value": "John Doe" } }
InfiniteScroll: Scrolls the page until the end for certain duration.
{ "InfiniteScroll": 3000 }
proxy_enabled boolean
Enable premium proxies to prevent detection. Default is false.
Using this configuration can help when request are blocked through the network. This allows more advanced proxies with AI.

Example request

import requests, os

headers = {
    'Authorization': os.getenv("SPIDER_API_KEY"),
    'Content-Type': 'application/jsonl',
}

json_data = {"limit":25,"return_format":"markdown","url":"https://spider.cloud"}

response = requests.post('https://api.spider.cloud/pipeline/extract-qa', 
  headers=headers, json=json_data)

print(response.json())

Response

{
  "content": [
    {
        "answer": "Spider is a data collecting solution designed for web crawling and scraping.",
        "question": "What is the primary function of Spider?"
    },
    {
        "answer": "You can kickstart your data collecting projects by signing up for a free trial or taking advantage of the promotional credits offered.",
        "question": "How can I get started with Spider?"
    },
    {
        "answer": "Spider offers unmatched speed, scalability, and comprehensive data curation, making it trusted by leading tech businesses.",
        "question": "What are the benefits of using Spider for data collection?"
    },
    {
        "answer": "Spider can easily crawl, search, and extract data from various sources, including social media platforms.",
        "question": "What kind of data can Spider extract?"
    },
    {
        "answer": "Spider is built fully in Rust for next-generation scalability.",
        "question": "What programming language is Spider built with?"
    }
  ],
  "credits_used": 13,
  "error": null,
  "status": 200,
  "url": "https://spider.cloud"
}

Queries

Query the data that you collect during crawling and scraping. Add dynamic filters for extracting exactly what is needed.

Websites Collection

Get the websites stored.

GEThttps://api.spider.cloud/data/websites

Request params

limit string
The limit of records to get.
Crawl Limit
page number
The current page to get.
domain string
Filter a single domain record.

Example request

import requests, os

headers = {
    'Authorization': os.getenv("SPIDER_API_KEY"),
    'Content-Type': 'application/jsonl',
}

response = requests.get('https://api.spider.cloud/data/websites?limit%3D25%26return_format%3Dmarkdown', 
  headers=headers)

print(response.json())

Response

{
  "data": [
    {
      "id": "2a503c02-f161-444b-b1fa-03a3914667b6",
      "user_id": "6bd06efa-bb0b-4f1f-a23f-05db0c4b1bfd",
      "url": "6bd06efa-bb0b-4f1f-a29f-05db0c4b1bfd/example.com/index.html",
      "domain": "spider.cloud",
      "created_at": "2024-04-18T15:40:25.667063+00:00",
      "updated_at": "2024-04-18T15:40:25.667063+00:00",
      "pathname": "/",
      "fts": "",
      "scheme": "https:",
      "last_checked_at": "2024-05-10T13:39:32.293017+00:00",
      "screenshot": null
    }
  ]
}

Pages Collection

Get the pages/resources stored.

GEThttps://api.spider.cloud/data/pages

Request params

limit string
The limit of records to get.
Crawl Limit
page number
The current page to get.
domain string
Filter a single domain record.

Example request

import requests, os

headers = {
    'Authorization': os.getenv("SPIDER_API_KEY"),
    'Content-Type': 'application/jsonl',
}

response = requests.get('https://api.spider.cloud/data/pages?limit%3D25%26return_format%3Dmarkdown', 
  headers=headers)

print(response.json())

Response

{
  "data": [
    {
      "id": "733b0d0f-e406-4229-949d-8068ade54752",
      "user_id": "6bd06efa-bb0b-4f1f-a29f-05db0c4b1bfd",
      "url": "https://spider.cloud",
      "domain": "spider.cloud",
      "created_at": "2024-04-17T01:28:15.016975+00:00",
      "updated_at": "2024-04-17T01:28:15.016975+00:00",
      "proxy": true,
      "headless": true,
      "crawl_budget": null,
      "scheme": "https:",
      "last_checked_at": "2024-04-17T01:28:15.016975+00:00",
      "full_resources": false,
      "metadata": true,
      "gpt_config": null,
      "smart_mode": false,
      "fts": "'spider.cloud':1"
    }
  ]
}

Pages Metadata Collection

Get the pages metadata/resources stored.

GEThttps://api.spider.cloud/data/pages_metadata

Request params

limit string
The limit of records to get.
Crawl Limit
page number
The current page to get.
domain string
Filter a single domain record.

Example request

import requests, os

headers = {
    'Authorization': os.getenv("SPIDER_API_KEY"),
    'Content-Type': 'application/jsonl',
}

response = requests.get('https://api.spider.cloud/data/pages_metadata?limit%3D25%26return_format%3Dmarkdown', 
  headers=headers)

print(response.json())

Response

{
  "data": [
    {
      "id": "e27a1995-2abe-4319-acd1-3dd8258f0f49",
      "user_id": "253524cd-3f94-4ed1-83b3-f7fab134c3ff",
      "url": "253524cd-3f94-4ed1-83b3-f7fab134c3ff/www.google.com/search?query=spider.cloud.html",
      "domain": "www.google.com",
      "resource_type": "html",
      "title": "spider.cloud - Google Search",
      "description": "",
      "file_size": 1253960,
      "embedding": null,
      "pathname": "/search",
      "created_at": "2024-05-18T17:40:16.4808+00:00",
      "updated_at": "2024-05-18T17:40:16.4808+00:00",
      "keywords": [
        "Fastest Web Crawler spider",
        "Web scraping"
      ],
      "labels": "Search Engine",
      "extracted_data": null,
      "fts": "'/search':1"
    }
  ]
}

Leads Collection

Get the pages contacts stored.

GEThttps://api.spider.cloud/data/contacts

Request params

limit string
The limit of records to get.
Crawl Limit
page number
The current page to get.
domain string
Filter a single domain record.

Example request

import requests, os

headers = {
    'Authorization': os.getenv("SPIDER_API_KEY"),
    'Content-Type': 'application/jsonl',
}

response = requests.get('https://api.spider.cloud/data/contacts?limit%3D25%26return_format%3Dmarkdown', 
  headers=headers)

print(response.json())

Response

{
  "data": [
    {
      "full_name": "John Doe",
      "email": "johndoe@gmail.com",
      "phone": "555-555-555",
      "title": "Baker"
    }
  ]
}

Crawl State

Get the state of the crawl for the domain.

GEThttps://api.spider.cloud/data/crawl_state

Request params

limit string
The limit of records to get.
Crawl Limit
page number
The current page to get.
domain string
Filter a single domain record.

Example request

import requests, os

headers = {
    'Authorization': os.getenv("SPIDER_API_KEY"),
    'Content-Type': 'application/jsonl',
}

response = requests.get('https://api.spider.cloud/data/crawl_state?limit%3D25%26return_format%3Dmarkdown', 
  headers=headers)

print(response.json())

Response

{
  "data": {
    "id": "195bf2f2-2821-421d-b89c-f27e57ca71fh",
    "user_id": "6bd06efa-bb0a-4f1f-a29f-05db0c4b1bfg",
    "domain": "spider.cloud",
    "url": "https://spider.cloud/",
    "links": 1,
    "credits_used": 3,
    "mode": 2,
    "crawl_duration": 340,
    "message": null,
    "request_user_agent": "Spider",
    "level": "info",
    "status_code": 0,
    "created_at": "2024-04-21T01:21:32.886863+00:00",
    "updated_at": "2024-04-21T01:21:32.886863+00:00"
  },
  "error": null
}

Crawl Logs

Get the last 24 hours of logs.

GEThttps://api.spider.cloud/data/crawl_logs

Request params

limit string
The limit of records to get.
Crawl Limit
page number
The current page to get.
domain string
Filter a single domain record.

Example request

import requests, os

headers = {
    'Authorization': os.getenv("SPIDER_API_KEY"),
    'Content-Type': 'application/jsonl',
}

response = requests.get('https://api.spider.cloud/data/crawl_logs?limit%3D25%26return_format%3Dmarkdown', 
  headers=headers)

print(response.json())

Response

{
  "data": {
    "id": "195bf2f2-2821-421d-b89c-f27e57ca71fh",
    "user_id": "6bd06efa-bb0a-4f1f-a29f-05db0c4b1bfg",
    "domain": "spider.cloud",
    "url": "https://spider.cloud/",
    "links": 1,
    "credits_used": 3,
    "mode": 2,
    "crawl_duration": 340,
    "message": null,
    "request_user_agent": "Spider",
    "level": "info",
    "status_code": 0,
    "created_at": "2024-04-21T01:21:32.886863+00:00",
    "updated_at": "2024-04-21T01:21:32.886863+00:00"
  },
  "error": null
}

Credits

Get the remaining credits available.

GEThttps://api.spider.cloud/data/credits

Example request

import requests, os

headers = {
    'Authorization': os.getenv("SPIDER_API_KEY"),
    'Content-Type': 'application/jsonl',
}

response = requests.get('https://api.spider.cloud/data/credits?limit%3D25%26return_format%3Dmarkdown', 
  headers=headers)

print(response.json())

Response

{
  "data": {
    "id": "8d662167-5a5f-41aa-9cb8-0cbb7d536891",
    "user_id": "6bd06efa-bb0a-4f1f-a29f-05db0c4b1bfg",
    "credits": 53334,
    "created_at": "2024-04-21T01:21:32.886863+00:00",
    "updated_at": "2024-04-21T01:21:32.886863+00:00"
  }
}

Crons

Get the cron jobs that are set to keep data fresh.

GEThttps://api.spider.cloud/data/crons

Request params

limit string
The limit of records to get.
Crawl Limit
page number
The current page to get.
domain string
Filter a single domain record.

Example request

import requests, os

headers = {
    'Authorization': os.getenv("SPIDER_API_KEY"),
    'Content-Type': 'application/jsonl',
}

response = requests.get('https://api.spider.cloud/data/crons?limit%3D25%26return_format%3Dmarkdown', 
  headers=headers)

print(response.json())

Response

User Profile

Get the profile of the user. This returns data such as approved limits and usage for the month.

GEThttps://api.spider.cloud/data/profile

Example request

import requests, os

headers = {
    'Authorization': os.getenv("SPIDER_API_KEY"),
    'Content-Type': 'application/jsonl',
}

response = requests.get('https://api.spider.cloud/crawl?limit%3D25%26return_format%3Dmarkdown', 
  headers=headers)

print(response.json())

Response

{
  "data": {
    "id": "6bd06efa-bb0b-4f1f-a29f-05db0c4b1bfd",
    "email": "user@gmail.com",
    "stripe_id": "cus_OYO2rAhSQaYqHT",
    "is_deleted": null,
    "proxy": null,
    "headless": false,
    "billing_limit": 50,
    "billing_limit_soft": 120,
    "approved_usage": 0,
    "crawl_budget": {
      "*": 200
    },
    "usage": null,
    "has_subscription": false,
    "depth": null,
    "full_resources": false,
    "meta_data": true,
    "billing_allowed": false,
    "initial_promo": false
  }
}

User-Agents

Get a real user agent to use for crawling.

GEThttps://api.spider.cloud/data/user_agents

Request params

limit string
The limit of records to get.
Crawl Limit
os string
Filter a by a device ex: Android, Mac OS, Android, Windows, Linux and more.
page number
The current page to get.
platform string
Filter a by a platform ex: Chrome, Edge, Safari, Firefox and more.

Example request

import requests, os

headers = {
    'Authorization': os.getenv("SPIDER_API_KEY"),
    'Content-Type': 'application/jsonl',
}

response = requests.get('https://api.spider.cloud/data/user_agents?limit%3D25%26return_format%3Dmarkdown', 
  headers=headers)

print(response.json())

Response

{
  "data": {
    "agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36",
    "platform": "Chrome",
    "platform_version": "123.0.0.0",
    "device": "Macintosh",
    "os": "Mac OS",
    "os_version": "10.15.7",
    "cpu_architecture": "",
    "mobile": false,
    "device_type": "desktop"
  }
}

Download file

Download a resource from storage.

GEThttps://api.spider.cloud/data/download

Request body

url string
The exact path of the url that you want to get.
Test Url
domain string
The website domain you want to query.
pathname string
The website pathname you want to query.

Example request

import requests, os

headers = {
    'Authorization': os.getenv("SPIDER_API_KEY"),
    'Content-Type': 'application/jsonl',
}

response = requests.get('https://api.spider.cloud/data/download?', 
  headers=headers)

print(response.json())

Response

{
  "data": "<file>"
}

Manage

Configure data to enhance crawl efficiency: create, update, and delete records.

Website

Create or update a website by configuration.

POSThttps://api.spider.cloud/data/websites

Request body

url required string
The URI resource to crawl. This can be a comma split list for multiple urls.
To reduce latency, enhance performance, and save on batch multiple URLs into a single call. For large websites with high page limits, it's best to run requests individually.
Test Url
limit number
The maximum amount of pages allowed to crawl per website. Remove the value or set it to 0 to crawl all pages.
It is better to set a limit upfront on websites where you do not know the size. Re-crawling can effectively use cache to keep costs low as new pages are found.
Crawl Limit
request string
The request type to perform. Possible values are http, chrome, and smart. Use smart to perform HTTP request by default until JavaScript rendering is needed for the HTML.
The request greatly influences how the output is going to look. If the page is server-side rendered, you can stick to the defaults for the most part.
return_format string
The format to return the data in. Possible values are markdown, commonmark, raw, text, and bytes. Use raw to return the default format of the page like HTML etc.
Usually you want to use markdown for LLM processing or text. If you need to store the files without losing any encoding, use the bytes or raw format.
readability boolean
Use readability to pre-process the content for reading. This may drastically improve the content for LLM usage.
This uses the Safari Reader Mode algorithm to extract only important information from the content.
metadata boolean
Collect metadata about the content found like page title, description, keywards and etc. This could help improve AI interoperability. Defaults to false unless you have the website already stored with the configuration enabled.
Using metadata can help extract critical information to use for AI.
anti_bot boolean
Enable anti-bot mode using various techniques to increase the chance of success. Default is false.
This config will attempt to make request resemble a real human. If the request fails on chrome it will retry using a virtual display for the request that is slower difficult to block at the cost of speed.
css_extraction_map object
Use CSS query selectors to scrape contents from the web page. Set the paths and the CSS extraction object map to perform extractions per path or page.
Scrape content using CSS or XPATH selectors to get data. You can scrape using selectors at no extra cost.
store_data boolean
Determine whether to store data. If set this takes precedence over storageless. Defaults to false.
Set to true to collect resources to download and re-use later on.
tld boolean
Allow TLD's to be included. Default is false.
cron string
Set a cron period to run the website crawls automatically. Possible values are daily, weekly, and monthly.
depth number
The crawl limit for maximum depth. If 0, no limit will be applied.
Depth allows you to place a distance between the base URL path and subfolders.
Crawl Depth
cache boolean
Use HTTP caching for the crawl to speed up repeated runs. Default is true.
Enabling caching can save costs when you need to perform transformations on different files or handle various events on a website.
delay number
Add a crawl delay of up to 60 seconds, disabling concurrency. The delay needs to be in milliseconds format. The default is 0, which indicates it is disabled.
Using a delay can help with websites that are set on a cron and do not require immediate data retrieval.
budget object
Object that has paths with a counter for limiting the amount of pages example {"*":1} for only crawling the root page. The wildcard matches all routes and you can set child paths preventing a depth level, example of limiting { "/docs/colors": 10, "/docs/": 100 } which only allows a max of 100 pages if the route matches /docs/:pathname and only 10 pages if it matches /docs/colors/:pathname.
The budget explicitly allows you to set paths and limits for the crawl.
Crawl Budget
scroll number
Infinite scroll the page as new content loads, up to a duration in milliseconds. The duration represents the maximum time you would wait to scroll. You may still need to use the wait_for parameters. You also need to ensure that the request is made using chrome.
Use the wait_for configuration to scroll until and disable_intercept to make sure you get data from the network regardless of hostname.
locale string
The locale to use for request, example en-US.
cookies string
Add HTTP cookies to use for request.
Set the cookie value for pages that use SSR authentication.
stealth boolean
Use stealth mode for headless chrome request to help prevent being blocked. The default is false on chrome.
Set to true to almost guarantee not being detected by anything.
headers object
Forward HTTP headers to use for all request. The object is expected to be a map of key value pairs.
Using HTTP headers can help with authenticated pages that use the authorization header field.
sitemap boolean
Include the sitemap results to crawl. The default is false.
The sitemap allows you to include links that may not be exposed in the HTML.
viewport object
Configure the viewport for chrome. Defaults to 800x600.
If you need to get data from a website as a mobile, set the viewport to a phone device's size ex: 375x414.
encoding string
The type of encoding to use like UTF-8, SHIFT_JIS, or etc.
Perform the encoding on the server when you know in advance the type of website.
wait_for object
The wait_for parameter allows you to specify various waiting conditions for a website operation. If provided, it contains the following sub-parameters:
The key idle_network specifies the conditions to wait for the network to be idle. It can include an optional timeout value.
The key selector specifies the conditions to wait for a particular CSS selector to be found on the page. It includes an optional timeout value, and the CSS selector to wait for.
The key delay specifies a delay to wait for, with an optional timeout value.
The key page_navigationsset to true then waiting for all page navigations will be handled.
If wait_for is not provided, the default behavior is to wait for the network to be idle for 500 milliseconds. All of the durations are capped at capped at 60 seconds.
The values for the timeout duration are in the object shape { secs: 10, nanos: 0 }.
blacklist array
Blacklist a set of paths that you do not want to crawl. You can use Regex patterns to help with the list.
whitelist array
Whitelist a set of paths that you want to crawl, ignoring all other routes that do not match the patterns. You can use regex patterns to help with the list.
subdomains boolean
Allow subdomains to be included. Default is false.
user_agent string
Add a custom HTTP user agent to the request. By default this is set to a random agent.
gpt_config object
Use AI to generate actions to perform during the crawl. You can pass an array for the prompt to chain steps.
Perform event-driven browser actions on a web page or extract data using the gpt_config, you need to use your own API key or have at least $100 in credits to use AI.
fingerprint boolean
Use advanced fingerprint detection for chrome.
Set this value to help crawl when websites require a fingerprint.
storageless boolean
Prevent storing any type of data for the request including storage. Defaults to true unless you have the website already stored.
chunking_alg object
Use a chunking algorithm to segment your content output. Possible values are ByWords, ByLines, ByCharacterLength, and BySentence. The following is an example of the object shape for "chunking_alg": { "type": "bysentence", "value": 2 } which splits the text into an array by every 2 sentences found. This works well when used with markdown or text formats.
The chunking algorithm allows you to prepare the content for the AI quick, without needing to add extra code or loaders.
country_code string
Set a ISO country code for premium proxy connections, this only works for HTTP request at the moment. You need to also use proxy_enabled.
The country code allows you to run requests in regions where access to the website is restricted to within that specific region.
root_selector string
The CSS query selector to use when extracting content from the markup.
Test Query Selector
page_insights boolean
Gather page insights like request duration, accessibility, and other vitals. The default is false and the param requires metadata to be used in conjunction.
Page Insights provides a quick, high-level alternative to Google's PageSpeed.
return_headers boolean
Return the HTTP response headers with the results. Defaults to false unless you have the website already stored with the configuration enabled.
Getting the HTTP headers can help setup authentication flows.
return_cookies boolean
Return the HTTP response cookies with the results. Defaults to false unless you have the website already stored with the configuration enabled.
Getting the HTTP cookies can help setup authentication SSR flows.
respect_robots boolean
Respect the robots.txt file for crawling. Default is true.
If you have trouble crawling a website it may be an issue with the robots.txt file. Setting the value to false could help. Make sure to use this config sparingly.
full_resources boolean
Crawl and download all the resources for a website.
Collect all the content from the website, including assets like images, videos, etc.
request_timeout number
The timeout to use for request. Timeouts can be from 5-60. The default is 30 seconds.
The timeout of the request helps prevent long request times from hanging.
external_domains array
A list of external domains to include into the domain results. You can use regex paths to include the domains. Set one of the array values to * to include all domains.
return_page_links boolean
Return the links found on each page. Defaults to false unless you have the website already stored with the configuration enabled.
Getting the links can help index the reference locations found for the resource.
return_embeddings boolean
Include OpenAI embeddings for title and description. The default is false and the param requires metadata to be used in conjunction.
If you are embedding data, you can use these matrices as staples for most vector baseline operations.
execution_scripts object
Run custom JavaScript on certain paths. This helps with authenticated pages. The request mode to be made through chrome or smart to run. The values for the timeout duration should be in the object shape "/path_or_url": "custom js".
Custom scripts allow you to take control of the browser with events for up to 60 seconds at a time per page.
disable_intercept boolean
Disable request interception when running request as chrome or smart. This may help if you bypass pages that use third-party scripts or external domains. The default setting is false. Note that cost and speed will increase when disabling this feature, as it removes our native Chrome interception.
run_in_background boolean
Run the request in the background. Useful if storing data and wanting to trigger crawls to the dashboard. This has no effect if storageless is set.
skip_config_checks boolean
Skip checking the database for website configuration. This may increase performance for request that are using limit=1. The default is false.
automation_scripts object
Run custom web automated tasks on certain paths. The request mode to be made through chrome or smart to run.
Custom web automation allows you to take control of the browser with events for up to 60 seconds at a time per page.
Below are the available actions for web automation:
Evaluate: Runs custom JavaScript code.
{ "Evaluate": "console.log('Hello, World!');" }
Click: Clicks on an element identified by a CSS selector.
{ "Click": "button#submit" }
Wait: Waits for a specified duration in milliseconds.
{ "Wait": 2000 }
WaitForNavigation: Waits for the next navigation event.
{ "WaitForNavigation": true }
WaitFor: Waits for an element to appear identified by a CSS selector.
{ "WaitFor": "div#content" }
WaitForAndClick: Waits for an element to appear and then clicks on it, identified by a CSS selector.
{ "WaitForAndClick": "button#loadMore" }
ScrollX: Scrolls the screen horizontally by a specified number of pixels.
{ "ScrollX": 100 }
ScrollY: Scrolls the screen vertically by a specified number of pixels.
{ "ScrollY": 200 }
Fill: Fills an input element with a specified value.
{ "Fill": { "selector": "input#name", "value": "John Doe" } }
InfiniteScroll: Scrolls the page until the end for certain duration.
{ "InfiniteScroll": 3000 }
proxy_enabled boolean
Enable premium proxies to prevent detection. Default is false.
Using this configuration can help when request are blocked through the network. This allows more advanced proxies with AI.

Example request

import requests, os

headers = {
    'Authorization': os.getenv("SPIDER_API_KEY"),
    'Content-Type': 'application/jsonl',
}

json_data = {"limit":25,"return_format":"markdown","url":"https://spider.cloud"}

response = requests.post('https://api.spider.cloud/crawl', 
  headers=headers, json=json_data)

print(response.json())

Response

{
  "data": null
}

Website

Delete a website from your collection. Remove the url body to delete all websites.

DELETEhttps://api.spider.cloud/data/websites

Request body

url required string
The URI resource to crawl. This can be a comma split list for multiple urls.
To reduce latency, enhance performance, and save on batch multiple URLs into a single call. For large websites with high page limits, it's best to run requests individually.
Test Url

Example request

import requests, os

headers = {
    'Authorization': os.getenv("SPIDER_API_KEY"),
    'Content-Type': 'application/jsonl',
}

json_data = {"limit":25,"return_format":"markdown","url":"https://spider.cloud"}

response = requests.post('https://api.spider.cloud/data/websites', 
  headers=headers, json=json_data)

print(response.json())

Response

{
  "data": null
}

Pages

Delete a web page from your collection. Remove the url body to delete all pages.

DELETEhttps://api.spider.cloud/data/pages

Request body

url required string
The URI resource to crawl. This can be a comma split list for multiple urls.
To reduce latency, enhance performance, and save on batch multiple URLs into a single call. For large websites with high page limits, it's best to run requests individually.
Test Url

Example request

import requests, os

headers = {
    'Authorization': os.getenv("SPIDER_API_KEY"),
    'Content-Type': 'application/jsonl',
}

json_data = {"limit":25,"return_format":"markdown","url":"https://spider.cloud"}

response = requests.post('https://api.spider.cloud/data/pages', 
  headers=headers, json=json_data)

print(response.json())

Response

{
  "data": null
}

Pages Metadata

Delete a web page metadata from your collection. Remove the url body to delete all pages metadata.

DELETEhttps://api.spider.cloud/data/pages_metadata

Request body

url required string
The URI resource to crawl. This can be a comma split list for multiple urls.
To reduce latency, enhance performance, and save on batch multiple URLs into a single call. For large websites with high page limits, it's best to run requests individually.
Test Url

Example request

import requests, os

headers = {
    'Authorization': os.getenv("SPIDER_API_KEY"),
    'Content-Type': 'application/jsonl',
}

json_data = {"limit":25,"return_format":"markdown","url":"https://spider.cloud"}

response = requests.post('https://api.spider.cloud/data/pages_metadata', 
  headers=headers, json=json_data)

print(response.json())

Response

{
  "data": null
}

Leads

Delete a contact or lead from your collection. Remove the url body to delete all contacts.

DELETEhttps://api.spider.cloud/data/contacts

Request body

url required string
The URI resource to crawl. This can be a comma split list for multiple urls.
To reduce latency, enhance performance, and save on batch multiple URLs into a single call. For large websites with high page limits, it's best to run requests individually.
Test Url

Example request

import requests, os

headers = {
    'Authorization': os.getenv("SPIDER_API_KEY"),
    'Content-Type': 'application/jsonl',
}

json_data = {"limit":25,"return_format":"markdown","url":"https://spider.cloud"}

response = requests.post('https://api.spider.cloud/data/contacts', 
  headers=headers, json=json_data)

print(response.json())

Response

{
  "data": null
}

Comprehensive Data APIs

Access JSON-ready data for various financial metrics, including currency exchange rates, commodity prices, and more.

Exchange Rates

Get access to real-time exchange rates for up to 150 global currency pairs with our comprehensive API, available for just 1 credit per call. With rates refreshed daily, you can ensure that your financial transactions are based on the most current data. This reliable endpoint seamlessly integrates with Stripe checkout, making it ideal for universal payments and enabling precise currency conversions to optimize your financial operations. Maximize your transactional efficiency and accuracy with our currency rate service.

GEThttps://api.spider.cloud/data/exchange-rates

Example request

import requests, os

headers = {
    'Authorization': os.getenv("SPIDER_API_KEY"),
    'Content-Type': 'application/jsonl',
}

response = requests.post('https://api.spider.cloud/data/exchange-rates', 
  headers=headers)

print(response.json())

Response

{
    "content": {
        "usd": 1,
        "eur": 0.9266,
        "aud": 1.5374,
    },
    "error": null
}

Commodity Rates

Unlock the power of market intelligence with our service! For just 1 credit per call, you can access comprehensive commodity prices updated daily. Whether you’re involved in agriculture, energy, metals, or other sectors, our data-driven insights ensure you’re always equipped with the latest, most accurate information to make informed decisions and stay ahead in the fiercely competitive market.

GEThttps://api.spider.cloud/data/commodity-rates

Example request

import requests, os

headers = {
    'Authorization': os.getenv("SPIDER_API_KEY"),
    'Content-Type': 'application/jsonl',
}

response = requests.post('https://api.spider.cloud/data/commodity-rates', 
  headers=headers)

print(response.json())

Response

{
  "content": {
    "energy": {
      "Crude Oil": 76.98,
      "Brent": 79.74,
    },
    "metals": {
      "Gold": 2430.58,
      "Silver": 27.449,
    },
    "agricultural": {
      "Soybeans": 1004.82,
      "Wheat": 542,
    },
    "industrial": {
      "Bitumen": 3590,
      "Cobalt": 26500,
    },
    "livestock": {
      "Feeder Cattle": 246.1513,
      "Live Cattle": 184.1172,
    },
    "index": {
      "CRB Index": 325.77,
      "LME Index": 3793.2,
    },
    "electricity": {
      "United Kingdom": 81.15,
      "Germany": 75.16,
    },
    "other": {
      "NVDA": 104.75,
      "TSLA": 200,
    },
  },
  "error": null
}

API Reference

Just getting started?

Not a developer?

Crawl websites

url required string

limit number

request string

return_format string

readability boolean

metadata boolean

anti_bot boolean

css_extraction_map object

store_data boolean

tld boolean

depth number

cache boolean

delay number

budget object

scroll number

locale string

cookies string

stealth boolean

headers object

sitemap boolean

viewport object

encoding string

wait_for object

blacklist array

whitelist array

subdomains boolean

user_agent string

gpt_config object

fingerprint boolean

storageless boolean

chunking_alg object

country_code string

root_selector string

page_insights boolean

return_headers boolean

return_cookies boolean

respect_robots boolean

full_resources boolean

request_timeout number

external_domains array

return_page_links boolean

return_embeddings boolean

execution_scripts object

disable_intercept boolean

run_in_background boolean

skip_config_checks boolean

automation_scripts object

proxy_enabled boolean

Search

search required string

search_limit number

fetch_page_content boolean

country string

location string

language string

limit number

request string

return_format string

readability boolean

metadata boolean

anti_bot boolean

css_extraction_map object

store_data boolean

num number

tld boolean

page number

depth number

cache boolean

delay number

budget object

scroll number

locale string

cookies string

stealth boolean

headers object

sitemap boolean