Skip to main content

Give your agents the whole web, as data.

Crawl, scrape and search the web with one API. Get clean markdown or JSON back for your agents.

Get started free

Sign up and run requests before you buy. Or copy the setup prompt into your coding agent.

request POST /scrape
{
  "url": "https://doc.rust-lang.org/book/",
  "return_format": "markdown"
}
example response 200 · 1 page · markdown
# The Rust Programming Language

*by Steve Klabnik, Carol Nichols, and Chris Krycho, with contributions from the Rust Community* This version of the text assumes you’re using Rust 1.90.0 (released 2025-09-18) or later with `edition = "2024"` in the *Cargo.toml* file of all projects to configure them to use Rust 2024 Edition idioms.
Captured example. The playground runs the real call. Run in the playground
Framework integrations

Three products. One key.

Start with the API. The browser and the proxy network also work on their own. The trace below is one crawl through all three.

authentication

export SPIDER_API_KEY="<YOUR_API_KEY>"

Get a key
example trace
  • api.spider.cloud

    Step one of an example trace, captured 2026-09-05. The API receives POST /crawl for doc.rust-lang.org/book with a limit of 30, hands the job to the browser, waits, and returns 30 pages of markdown totalling 331 kilobytes with status 200. Every page came back 200. demo-01 is an illustrative correlation id, not an API field.

  • browser.spider.cloud

    Step two. The browser opens a session, accepts the request from the API, routes it through an ISP exit in Germany, loads each of the 30 pages from doc.rust-lang.org/book, and hands the markdown back to the API before closing the session. demo-01 is an illustrative correlation id, not an API field.

  • proxy.spider.cloud

    Step three. The network accepts the browser's request and routes it through an ISP exit in Germany, rotating the exit per request, then fetches the 30 pages from doc.rust-lang.org/book and returns them to the browser. The pool is 215M+ exits in 199 countries. demo-01 is an illustrative correlation id, not an API field.

Crawl, scrape, search

Whole sites or single pages, returned as markdown, JSON, links or screenshots. Search the web from the same key.

See the API

A browser we wrote

Headless, written from scratch in Rust. Stealth and CAPTCHA solving are on by default, up to 100 sessions at once.

See the browser

The network underneath

215M+ residential and ISP exits in 199 countries, rotated per request. Works with our browser or yours.

See proxy locations

Use the whole web like a dataset.

Most websites have no API. Spider renders their pages and extracts the content for your RAG pipeline or agent.

pages per second
100K+
exits
215M+

  1. Renders like a user

    Engraving of an eighteenth century printing shop, the pressman at the press with a sheet on the tympan. From the Encyclopedie of Diderot and d'Alembert.

    Scripts run, lazy images load, and infinite scroll reaches the end. You get the page a person sees, not the one the server sends.

    See the browser

  2. Passes the checkpoint

    Engraving of the Porta Maggiore in Rome, two arches under an inscribed attic with the road passing through. From Piranesi's Vedute di Roma.

    The request handles bot walls, CAPTCHAs and geo checks on its own. What comes back is the page, with a 200.

    See the unblocker

  3. Extracts the content

    Woodcut of a papermaker lifting a fresh sheet from the vat on a mould, a boy carrying finished sheets beside him. From Jost Amman's Book of Trades.

    Navigation, ads and footers drop away. What is left is markdown or JSON that reads the way the page was written.

    See the formats

  4. Streams in order

    Engraving of the Marly machine in elevation, a water wheel driving a row of pumps on pilings. From the Encyclopedie of Diderot and d'Alembert.

    Each page in a crawl arrives as one JSONL line the moment it is done, in order. Nothing waits for the last URL.

    See streaming

Pay for usage, or for concurrency.

pay as you go

no minimum
$1 per GB

Plus $0.001 per CPU minute.

Bandwidth
$1 per GB
CPU time
$0.001 per minute
Blocked, timed out, failed
$0
Start free

unlimited

flat rate
from $40 a month

Priced by concurrency units. View unit pricing

Requests
unlimited
Starting allocation
2 concurrency units
Per concurrency unit
$20 down to $13
See unlimited plans

Make your first request.

Three steps, the same ones the dashboard shows after you sign up. Pick your client and paste.

01 Get a key

Create API key

Sign up and copy the key. New accounts start with a free balance.

export SPIDER_API_KEY="sk-..."

SDKs, the MCP server and every command read it.

02 Connect Claude Code

claude mcp add spider --transport http \
  https://mcp.spider.cloud/mcp \
  -H "Authorization: Bearer $SPIDER_API_KEY"
Read and follow https://spider.cloud/agent-skill/SKILL.md

Or skip MCP and let the agent onboard itself

Client configs in the docs

03 Read the data

example output example data after a scrape
[{
  "url": "https://example.com",
  "status": 200,
  "content": "# Example Domain\n\n..."
}]

The connection comes first. Data like this arrives once the client makes a request.

Markdown or JSON, streamed. Drop it into your RAG store, agent context or database.

Start with one URL.

Create a key. Make your first request.

Get started free