Scraping and crawling

/scrape fetches a single URL and returns its content. /crawl starts from a URL, follows internal links, and returns content for each page. Both share the same output formats, proxy settings, and request modes.

Single page

Scrape one URL

Send a URL to /scrape and pick a return_format (markdown for LLMs, raw for the original HTML). Call the API directly or use any of our SDK libraries.

import requests, os

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

params = {
    "url": "https://www.example.com",
    "request": "smart",            # auto-pick HTTP vs Chrome
    "return_format": "markdown",   # LLM-friendly format
    "proxy_enabled": True,
}

response = requests.post(
    'https://api.spider.cloud/scrape',
    headers=headers,
    json=params,
)

print(response.json())

Tip

Test with a few different URLs to see how each site responds.

Multiple pages

Crawl from a URL

/crawl starts from a URL and follows internal links. limit caps how many pages come back; depth controls how many link-levels deep the crawler goes. Set a reasonable limit when testing. Most sites have thousands of pages.

import requests, os

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

params = {
    "url": "https://www.example.com",
    "limit": 30,                   # max pages
    "depth": 3,                    # link-hops from the seed
    "request": "smart",
    "return_format": "markdown",
}

response = requests.post(
    'https://api.spider.cloud/crawl',
    headers=headers,
    json=params,
)

print(response.json())

# Iterate over each page in the response
for result in response.json():
    print(result['content'])       # body in the requested format
    print(result['url'])           # final URL
    print(result['status'])        # HTTP status
    print(result['error'])         # error message if any
    print(result['costs'])         # cost breakdown for this page

Reference

Request types

The request parameter controls how Spider fetches each page.

smartDefault

Auto-picks between HTTP and Chrome based on what the page actually needs.

httpFast

Static HTML only. Fastest and cheapest.

browserJS / SPA

Full browser rendering. Use for SPAs or bot-protected sites.

Streaming responses

Use streaming to process pages the moment they finish crawling instead of waiting for the entire result set. Set Content-Type: application/jsonl and read the response line by line.

Response fields

Each page in the response array carries these fields.

Field	Type	Description
`url`	string	The URL that was crawled or scraped.
`status`	number	HTTP status code from the target page (200, 404, 500, …).
`content`	string	Page content in the requested format: HTML, markdown, text, or base64 for screenshots.
`error`	string \| null	Error message if the page failed to load. null on success.
`costs`	object	Cost breakdown for this request in USD.
`costs.total_cost`	number	Total cost of this request.
`costs.total_cost_formatted`	string	Human-readable formatted total.
`costs.ai_cost`	number	AI processing cost: extraction, labeling, and so on.
`costs.bytes_transferred_cost`	number	Cost based on data transferred.
`costs.compute_cost`	number	Cost of compute resources: browser rendering and similar.

[
  {
    "url": "https://example.com",
    "status": 200,
    "content": "<html>...</html>",
    "error": null,
    "costs": {
      "ai_cost": 0.0,
      "ai_cost_formatted": "0",
      "bytes_transferred_cost": 3.165e-9,
      "bytes_transferred_cost_formatted": "0.0000000032",
      "compute_cost": 0.0,
      "compute_cost_formatted": "0",
      "file_cost": 0.000029,
      "file_cost_formatted": "0.0000290000",
      "total_cost": 0.000029,
      "total_cost_formatted": "0.0000290000",
      "transform_cost": 0.0,
      "transform_cost_formatted": "0"
    }
  }
]

Handling errors

Some pages in a crawl will fail: 404s, timeouts, bot blocks. The API still returns an HTTP 200, but individual pages in the array may carry non-200 status values or an error field. Check per-page status before reading content. See Error Codes for the full reference.

import requests, os

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

response = requests.post(
    'https://api.spider.cloud/crawl',
    headers=headers,
    json={"url": "https://example.com", "limit": 20, "return_format": "markdown"},
)

# Check the Spider API request itself
if response.status_code != 200:
    print(f"API error: {response.status_code} - {response.text}")
else:
    data = response.json()
    succeeded = [p for p in data if p.get('status') == 200]
    failed = [p for p in data if p.get('status') != 200]

    print(f"Crawled {len(data)} pages, {len(succeeded)} ok, {len(failed)} failed")

    for page in failed:
        print(f"  {page['url']}: status {page.get('status')}, {page.get('error', 'no details')}")

    for page in succeeded:
        content = page.get('content', '')
        if len(content) > 50:
            print(f"  {page['url']}: {len(content)} chars")

Tip

Spider supports multi-step browser automation. See the API reference for the automation_scripts parameter.