API Features
A complete toolkit for web data collection. Crawl, scrape, search, screenshot, transform, and extract — all from a single API.
Crawl
Recursively crawl entire websites and collect every page. Set depth limits, respect robots.txt, and get structured output in markdown, HTML, or plain text.
Scrape
Extract content from individual pages with precision. Optimized for single-page extraction with CSS selectors and metadata.
Search
Perform search engine queries and automatically crawl the results. Discovery + extraction in one step.
Screenshot
Capture high-quality screenshots of any web page. Full-page or viewport, as base64 or binary.
Transform
Convert raw HTML into clean markdown, plain text, or sanitized HTML. Process content offline without re-fetching pages from the web.
Links
Collect all links from a website without extracting page content. Optimized for sitemap generation and URL discovery at lower cost.
Unblocker
Access content behind anti-bot protections and challenging security measures. Advanced fingerprinting and session management.
Fetch
Per-website fetch APIs with AI-discovered configurations. Configs are discovered once, validated, cached, and reused.
Shared Across All Endpoints
// every endpoint includes these
Proxy Support
Residential, mobile, and ISP proxies with geo-routing to 100+ countries
Streaming
JSONL streaming so you can process results as they arrive
Caching
Built-in HTTP caching with configurable TTL
Webhooks
Async delivery when crawls complete
Rate Limiting
Automatic concurrency management
Auth Header
Simple Bearer token on every request
Multi-Format
JSON, XML, CSV, or JSONL output
Client Libraries
SDKs for Python, JavaScript, Rust, Go, and CLI
Transparent Pricing
// bandwidth + compute, no credit multipliers
Pricing Calculator
Estimate your monthly cost based on volume and configuration.
Auto-detects when a page needs a browser
Cost at 100K pages/mo
Competitor costs estimated from published pricing pages. Some require monthly commitments shown as minimum.
How Spider Compares
Estimated monthly cost at production volumes. Spider charges bandwidth + compute with no credit multipliers.
| Provider | 10K pages | 100K pages | 1M pages | Multipliers | Credits expire |
|---|---|---|---|---|---|
| Spider | ~$0.80 | ~$8 | ~$80 | None | Never |
| Firecrawl | $16 min | $83 | $599 | None | Monthly |
| ScrapingBee | $49+ | $99–$249 | $599+ | 1–75x | Monthly |
| Bright Data | $40+ | $270+ | $2,700+ | Per-feature | Monthly |
| ScrapFly | $30+ | $250–$500 | $500+ | 1–30x | Monthly |
Costs assume production workloads with anti-bot bypass. Competitor costs based on published pricing as of Feb 2026. Spider costs based on average production workload of ~$0.08/1K pages.