Give your agents the whole web, as data.
Crawl, scrape and search the web with one API. Get clean markdown or JSON back for your agents.
{
"url": "https://example.com",
"return_format": "markdown"
}# Example Domain This domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission. [More information...](https://www.iana.org/domains/example)
Three products. One key.
Use the API, the browser or the proxy network on their own. This example shows one crawl passing through all three.
- api.spider.cloud 200 · 30 pages
Example trace, step one: the API receives POST /crawl for news.ycombinator.com, hands the job to the browser, waits, and returns 30 pages of markdown with status 200.
Crawl, scrape, search
Crawl whole sites or scrape single pages. Return markdown, JSON, links or screenshots, and search the web from the same key.
See the API - browser.spider.cloud session · closed
Step two: the browser waits for the API, takes the request as session demo-01, routes it through a German ISP exit on the network, loads each page, and hands the result back to the API.
A browser we wrote
Headless and written from scratch in Rust. Stealth, proxy rotation and CAPTCHA solving are on by default, with up to 100 concurrent sessions.
See the browser - proxy.spider.cloud DE · ISP
Step three: the network waits for the browser, takes its request, routes demo-01 through an ISP exit in Germany rotating per request, fetches each page, and returns them. 215M+ IPs in 199 countries.
The network underneath
215M+ residential and ISP exits in 199 countries, rotated per request. Use it with our browser or your own.
See proxy locations
Use the whole web like a dataset.
Most websites have no API. Spider renders their pages and extracts the content for your RAG pipeline or agent.
Renders like a user
JavaScript, infinite scroll, lazy media. The page as a person sees it.
request: "browser"Handles anti-bot checks
CAPTCHAs and geo walls, handled by the browser and the network together.
stealth: trueExtracts the content
Clean markdown or structured JSON from the rendered page.
return_format: "markdown"Streams, in order
Whole sites arrive page by page while the crawl runs. Nothing waits for the last URL.
application/jsonl
Check the numbers
- pages per second
- 100K+
- Whole sites crawled and streamed back as markdown, in order.
- spider-rs on the benchmark machine, not a per-account limit. BENCHMARKS.md
- exits
- 215M+
- Residential and ISP proxies in 199 countries, rotated per request.
- counted from the proxy pool at build time. Proxy locations
Pricing
pay as you go
no minimumPlus $0.001 per CPU minute.
- Bandwidth
- $1 per GB
- CPU time
- $0.001 per minute
- Blocked, timed out, failed
- $0
- Minimum
- none
Example: 1 GB plus 10 CPU minutes comes to $1.01.
Start freeunlimited
flat ratePriced by concurrency units. View unit pricing
- Requests
- unlimited
- Starting allocation
- 2 concurrency units
- Per concurrency unit
- $20 down to $13
- Failed requests
- $0
Make your first request.
01Get a key
Create API keySign up and copy the key. New accounts start with a free balance.
export SPIDER_API_KEY="sk-..."
SDKs, the MCP server and every command read it.
02 Connect Claude Code Client configs in the docs
claude mcp add spider --transport http \ https://mcp.spider.cloud/mcp \ -H "Authorization: Bearer $SPIDER_API_KEY"
Read and follow https://spider.cloud/agent-skill/SKILL.md
Or skip MCP and let the agent onboard itself
02 Connect Codex Client configs in the docs
Read and follow https://spider.cloud/agent-skill/SKILL.md
Paste into a Codex session. The skill covers the whole API.
02 Connect your MCP client Client configs in the docs
Endpoint: https://mcp.spider.cloud/mcp Transport: Streamable HTTP Auth: Authorization: Bearer sk-...
Works in Claude Desktop, Cursor, Windsurf, or any MCP client.
02 Make the first request Full example in the docs
curl -X POST https://api.spider.cloud/scrape \
-H "Authorization: Bearer $SPIDER_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com", "return_format": "markdown"}'02 Install the SDK and run it Full example in the docs
pip install spider_client
from spider import Spider
app = Spider() # reads SPIDER_API_KEY
page = app.scrape_url("https://example.com")02 Install the SDK and run it Full example in the docs
npm install @spider-cloud/spider-client
import { Spider } from "@spider-cloud/spider-client";
const app = new Spider(); // reads SPIDER_API_KEY
const page = await app.scrapeUrl("https://example.com");03Read the data
example output[{
"url": "https://example.com",
"status": 200,
"content": "# Example Domain\n\n..."
}]Markdown or JSON, streamed. Drop it into your RAG store, agent context or database.
Start with one URL.
Create a key. Make your first request.