Scraping and crawling

/scrape fetches one URL and returns its content. /crawl starts from a URL, follows internal links, and returns content for each page. Both share the same output formats, proxy settings, and request modes.

Scrape one URL

Send a URL to /scrape and pick a return_format: markdown for LLMs, raw for the original HTML. Call the API directly or through any of the SDK libraries.

import requests, os

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

params = {
    "url": "https://www.example.com",
    "request": "smart",            # auto-pick HTTP vs Chrome
    "return_format": "markdown",   # LLM-friendly format
}

response = requests.post(
    'https://api.spider.cloud/scrape',
    headers=headers,
    json=params,
)

print(response.json())

Tip

Try a few different URLs to see how each site responds.

Crawl from a URL

/crawl starts from a URL and follows internal links. limit caps how many pages come back; depth caps how many link hops the crawler takes. Set a small limit when testing. Most sites have thousands of pages.

import requests, os

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

params = {
    "url": "https://www.example.com",
    "limit": 25,                   # max pages
    "depth": 3,                    # link-hops from the seed
    "return_format": "markdown",
}

response = requests.post(
    'https://api.spider.cloud/crawl',
    headers=headers,
    json=params,
)

print(response.json())

# Iterate over each page in the response
for result in response.json():
    print(result['content'])       # body in the requested format
    print(result['url'])           # final URL
    print(result['status'])        # HTTP status
    print(result['error'])         # error message if any
    print(result['costs'])         # cost breakdown for this page

Request types

The request parameter controls how Spider fetches each page.

smartDefault

Picks HTTP or Chrome per page, based on what the page needs.

httpFast

Static HTML only. Fastest and cheapest.

browserJS / SPA

Full browser rendering. Use for SPAs or protected sites.

Stream the response

With streaming you handle each page the moment it finishes instead of waiting for the whole crawl. Set Content-Type: application/jsonl and read the response line by line.

Response fields

Each page in the response array carries these fields.

Field	Type	Description
url	string	The URL that was crawled or scraped.
status	number	HTTP status code from the target page (200, 404, 500 and so on).
content	string	Page content in the requested format: HTML, markdown, text, or base64 for screenshots.
error	string \| null	Error message if the page failed to load. null on success.
costs	object	Cost breakdown for this request in USD.
costs.total_cost	number	Total cost of this request.
costs.total_cost_formatted	string	The total as a formatted string.
costs.ai_cost	number	AI processing cost: extraction, labeling, and so on.
costs.bytes_transferred_cost	number	Cost based on data transferred.
costs.compute_cost	number	Cost of compute resources: browser rendering and similar.

[
  {
    "url": "https://example.com",
    "status": 200,
    "content": "<html>...</html>",
    "error": null,
    "costs": {
      "ai_cost": 0.0,
      "ai_cost_formatted": "0",
      "bytes_transferred_cost": 3.165e-9,
      "bytes_transferred_cost_formatted": "0.0000000032",
      "compute_cost": 0.0,
      "compute_cost_formatted": "0",
      "file_cost": 0.000029,
      "file_cost_formatted": "0.0000290000",
      "total_cost": 0.000029,
      "total_cost_formatted": "0.0000290000",
      "transform_cost": 0.0,
      "transform_cost_formatted": "0"
    }
  }
]

Handle errors

Some pages in a crawl will fail: 404s, timeouts, bot blocks. The API still returns an HTTP 200, but a page in the array may carry a non-200 status or an error field. Check each page's status before reading its content. The error codes page lists every value.

import requests, os

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

response = requests.post(
    'https://api.spider.cloud/crawl',
    headers=headers,
    json={"url": "https://example.com", "limit": 20, "return_format": "markdown"},
)

# Check the Spider API request itself
if response.status_code != 200:
    print(f"API error: {response.status_code} - {response.text}")
else:
    data = response.json()
    succeeded = [p for p in data if p.get('status') == 200]
    failed = [p for p in data if p.get('status') != 200]

    print(f"Crawled {len(data)} pages, {len(succeeded)} ok, {len(failed)} failed")

    for page in failed:
        print(f"  {page['url']}: status {page.get('status')}, {page.get('error', 'no details')}")

    for page in succeeded:
        content = page.get('content', '')
        if len(content) > 50:
            print(f"  {page['url']}: {len(content)} chars")

Tip

Multi-step browser automation runs through the automation_scripts parameter. See the API reference.