Spider Developer Platform

Collect web data at scale. Spider handles crawling, rendering, proxy rotation, and anti-bot evasion, you get clean data back through a single API.

Developer quickstartInstall the SDK, export your API key, and make your first request.3 mins
Request
import requests, json

headers = {
    'Authorization': 'Bearer $SPIDER_API_KEY',
    'Content-Type': 'application/json',
}

json_data = {"limit":5,"url":"https://example.com"}

response = requests.post('https://api.spider.cloud/crawl', 
  headers=headers, json=json_data, stream=True)

with response as r:
    r.raise_for_status()
    
    buffer = b""

    for chunk in response.iter_content(chunk_size=8192):
        if chunk:
            buffer += chunk
            try:
                data = json.loads(buffer.decode('utf-8'))
                print(data)
                buffer = b""
            except json.JSONDecodeError:
                continue
CrawlFollow links across entire sites. Set depth, limit, and domain scope.
ScrapeFetch a single page as HTML, markdown, text, or structured JSON.
SearchSearch the web and scrape the results in one request.
ScreenshotCapture full-page screenshots with Chrome rendering.
StreamingProcess pages as they finish instead of waiting for the full result.
AI ExtractionExtract structured data from any page using AI or CSS selectors.
Data ConnectorsStream results directly to S3, Google Cloud, Azure Blob, Sheets, or Supabase.
Anti-Bot BypassRotate proxies across 100+ countries with automatic fingerprinting and stealth mode.
Fetch API (Alpha)AI-configured per-website scrapers. Discovers selectors automatically, caches and reuses.

How It Works

Every request goes through three stages: fetch (retrieve the page using HTTP or headless Chrome), process (render JavaScript, rotate proxies, handle anti-bot challenges), and deliver (convert to your chosen format and return). Spider's Rust-based engine runs all stages concurrently, so a 500-page crawl takes seconds, not hours.

API Endpoints

All endpoints accept JSON and return JSON. Authenticate with a Bearer token.

MethodPathDescription
POST/crawlStart from a URL and follow links to discover and fetch multiple pages.
POST/scrapeFetch a single page and return its content in any format.
POST/searchSearch the web and optionally scrape the results.
POST/screenshotCapture a full-page screenshot as base64 PNG.
POST/fetch/{domain}/{path}AI-configured per-website scraper with cached configs. (Alpha)
GET/data/scraper-directoryBrowse optimized scraper configs for popular websites.

Request Modes

Choose how Spider fetches each page. smart (default) automatically picks between HTTP and Chrome based on the page. Use http for static HTML, it is the fastest and cheapest. Use chrome when you need JavaScript rendering, SPA support, or real browser fingerprints for bot-protected sites. See Concepts for details.

Credits

Usage is measured in credits at $1 / 10,000 credits. Each page costs a base amount, with additional credits for Chrome rendering, proxy usage, and AI extraction. Every response includes a costs object with a per-request breakdown. Monitor your balance on the usage page.

Build now

JavaScript
npm install
Python
pip install
Rust
cargo add
CLI
terminal

Explore our guides

  • A practical walkthrough for collecting web data with Spider, from your first crawl to production pipelines.

    Jeff Mendez ·

  • An overview of Spider's API capabilities, endpoints, request modes, output formats, and how to get started.

    Jeff Mendez ·