Skip to main content

Give your agents the whole web, as data.

Crawl, scrape and search the web with one API. Get clean markdown or JSON back for your agents.

Get started free
request POST /scrape
{
  "url": "https://doc.rust-lang.org/book/",
  "return_format": "markdown"
}
example response 200 · 1 page · markdown
# The Rust Programming Language

*by Steve Klabnik, Carol Nichols, and Chris Krycho, with contributions from the Rust Community* This version of the text assumes you’re using Rust 1.90.0 (released 2025-09-18) or later with `edition = "2024"` in the *Cargo.toml* file of all projects to configure them to use Rust 2024 Edition idioms.
Captured example. The playground runs the real call. Run in the playground
Framework integrations

Three products. One key.

Use the API, the browser or the proxy network on their own. This example shows one crawl passing through all three.

authentication

export SPIDER_API_KEY="<YOUR_API_KEY>"

Get a key
example trace
  • api.spider.cloud

    Example trace, step one: the API receives POST /crawl for doc.rust-lang.org/book, hands the job to the browser, waits, and returns 30 pages of markdown with status 200. Captured 2026-09-05.

    Crawl, scrape, search

    Whole sites or single pages, returned as markdown, JSON, links or screenshots. Search the web from the same key.

    See the API
  • browser.spider.cloud

    Step two: the browser waits for the API, takes the request as session demo-01, routes it through a German ISP exit on the network, loads each page, and hands the result back to the API.

    A browser we wrote

    Headless, written from scratch in Rust. Stealth and CAPTCHA solving are on by default, up to 100 sessions at once.

    See the browser
  • proxy.spider.cloud

    Step three: the network waits for the browser, takes its request, routes demo-01 through an ISP exit in Germany rotating per request, fetches each page, and returns them. 215M+ IPs in 199 countries.

    The network underneath

    215M+ residential and ISP exits in 199 countries, rotated per request. Works with our browser or yours.

    See proxy locations

Use the whole web like a dataset.

Most websites have no API. Spider renders their pages and extracts the content for your RAG pipeline or agent.

Two figures, each with its source.

pages per second
100K+
spider-rs on the benchmark machine, not a per-account limit. BENCHMARKS.md
exits
215M+
counted from the proxy pool at build time. Proxy locations

  1. Renders like a user

    JavaScript, infinite scroll, lazy media. The page as a person sees it.

    request: "browser"

  2. Handles anti-bot checks

    CAPTCHAs and geo walls, handled by the browser and the network together.

    stealth: true

  3. Extracts the content

    Clean markdown or structured JSON from the rendered page.

    return_format: "markdown"

  4. Streams in order

    Whole sites arrive page by page while the crawl runs. Nothing waits for the last URL.

    application/jsonl

Pay for usage, or for concurrency.

pay as you go

no minimum
$1 per GB

Plus $0.001 per CPU minute.

Bandwidth
$1 per GB
CPU time
$0.001 per minute
Blocked, timed out, failed
$0
Minimum
none

Example: 1 GB plus 10 CPU minutes comes to $1.01.

Start free

unlimited

flat rate
from $40 a month

Priced by concurrency units. View unit pricing

Requests
unlimited
Starting allocation
2 concurrency units
One request in flight per unit.
Per concurrency unit
$20 down to $13
Failed requests
$0
See unlimited plans

Make your first request.

quickstart

01Get a key

Create API key

Sign up and copy the key. New accounts start with a free balance.

export SPIDER_API_KEY="sk-..."

SDKs, the MCP server and every command read it.

02 Connect Claude Code Client configs in the docs 

claude mcp add spider --transport http \
  https://mcp.spider.cloud/mcp \
  -H "Authorization: Bearer $SPIDER_API_KEY"
Read and follow https://spider.cloud/agent-skill/SKILL.md

Or skip MCP and let the agent onboard itself

03Read the data

example output
[{
  "url": "https://example.com",
  "status": 200,
  "content": "# Example Domain\n\n..."
}]

Markdown or JSON, streamed. Drop it into your RAG store, agent context or database.

Start with one URL.

Create a key. Make your first request.

Get started free