---
name: spider
description: >-
  Set up Spider Cloud for fast web crawling, scraping, and search. Starts with a
  keyless request that needs no signup, then covers Spider Cloud (hosted REST
  API + Python/JS/Go/Rust SDKs and CLI) and the open-source Spider engine for
  self-hosting, including a hosted MCP server so AI tools can call Spider
  directly. Use it to pick the right path and get a working request in
  minutes.
---

# Spider Cloud

Spider Cloud is a web crawler and scraper built in Rust. It turns any site into
clean Markdown/HTML/text for LLMs and pipelines, runs a headless Browser when
a page needs JavaScript, rotates proxies, and streams results so large crawls
stay fast and cheap.

## Try it now (no key)

`POST /scrape` works with no API key and no signup. Run this first to confirm
you can reach Spider before setting anything up:

```bash
curl -X POST https://api.spider.cloud/scrape \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "return_format": "markdown"}'
```

The keyless tier is rate limited and covers `/scrape` (and its alias
`/v1/scrape`) only. Going over the limit returns `429`; honor `Retry-After`.

Every other route needs a key and refuses a keyless call with `402`. The
one exception is `/unlimited/*`, which returns `401`. Get a key below when
you need those endpoints, higher limits, or concurrency.

---

There are two ways to run it:

- **Spider Cloud**, the hosted API at `https://api.spider.cloud`. Nothing to
  run. Bring an API key. This is the right choice for almost everyone.
- **Open-source engine**, the `spider` Rust crate you host yourself. Choose
  this only if you need to run the crawler inside your own infrastructure.

## Get a key (Spider Cloud)

1. Create a key at https://spider.cloud (Account → API Keys).
2. Export it. The SDKs and the CLI read this variable; the REST API expects
   it as a bearer token.

```bash
export SPIDER_API_KEY="sk-..."
```

Check it works:

```bash
curl https://api.spider.cloud/data/credits \
  -H "Authorization: Bearer $SPIDER_API_KEY"
```

## Pick a path

| You want to…                                   | Go to |
| ---------------------------------------------- | ----- |
| Just scrape one page right now, no signup      | **Try it now (no key)**, above |
| Call the API directly (any language, curl)     | **A. REST API** |
| Use an SDK in Python / JS / Go / Rust          | **B. Client SDK** |
| Run the crawler in your own infrastructure     | **C. Self-host** |
| Let Claude / Cursor / an agent call Spider     | **D. MCP server** |

Most integrations are **A** or **B**. Reach for **C/D** only when you have a
specific reason (data residency, an MCP-native agent).

---

## A. REST API

Base URL `https://api.spider.cloud`. Send JSON, authenticate with the bearer
token. The content endpoints take a `url` plus optional parameters; `/search`
takes `search`, `/transform` takes the HTML you already have, and the `/ai/*`
routes are driven by a `prompt`.

```bash
curl -X POST https://api.spider.cloud/scrape \
  -H "Authorization: Bearer $SPIDER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "return_format": "markdown"
  }'
```

Endpoints:

| Endpoint | Purpose |
|---|---|
| `POST /scrape` | One page, returned as Markdown/HTML/text. Works keyless. |
| `POST /crawl` | A whole site. Set `limit` and `request` (see below). |
| `POST /links` | Collect links from a site without returning content. |
| `POST /search` | Web search. The query field is named `search`, not `query`. |
| `POST /screenshot` | Render the page and return an image. |
| `POST /transform` | Convert raw HTML you already have into clean Markdown. Fetches nothing; takes `data: [{ "html": "...", "url": "..." }]`. |
| `POST /unblocker` | Fetch one page through Spider's anti-bot bypass (proxy rotation, fingerprinting, JS challenge solving). Use it for sites that block normal `/scrape` requests. |
| `GET /data/credits` | Remaining account credits. There is no `/credits`. |

Every path above also accepts a `/v1/` prefix (`/v1/crawl` is `/crawl`).

`/search` returns ranked result URLs and, with `fetch_page_content: true`,
also fetches each result's content. Cap results with `search_limit` or `num`,
page through with `page` or `auto_pagination`, and localize with
`country`/`location`/`language`, or pin exact coordinates with `latitude` +
`longitude` (Google engine; optional `radius` in meters).

### What comes back

Content routes return a JSON **array**, one object per page. That includes
`/scrape`: its single page arrives as a one-element array, so
`response[0]["content"]`, not `response["content"]`. `/transform` is the
exception: it returns a single `{ "content": ... }` object.

```json
[{
  "url": "https://example.com",
  "content": "# Example Domain\n\n...",
  "status": 200,
  "error": null,
  "duration_elasped_ms": 122,
  "costs": { "total_cost": 0.00004, "compute_cost": 0.00001,
             "file_cost": 0.00002, "bytes_transferred_cost": 0.00002 }
}]
```

`status` is the page's own HTTP status, separate from the API call's status.
A `200` API response routinely contains pages with `"status": 404`, so check
both, and read `error` per object rather than assuming success. Fields that
appear only when requested: `metadata`, `css_extracted`, `links`, `headers`,
`cookies`, `json_data`.

Two things about that object that bite. `duration_elasped_ms` is misspelled
in the API itself, and that spelling is the wire key, so match it exactly.
And `costs` is `null`, not an object of zeros, whenever nothing was billed,
so guard before reading `costs.total_cost`. Its values are USD, and each one
has a `_formatted` string twin carrying full precision.

### Errors

| Code | Meaning |
|---|---|
| 400 | Bad parameters or malformed JSON. On `/unlimited/*`, an AI parameter was sent; the error names it. |
| 401 | Missing, invalid, or expired key. Also keyless calls to `/unlimited/*`. |
| 402 | Out of credits, or a keyless call to any route other than `/scrape` (`free_tier_route_blocked`). On `/ai/*`, no active AI plan gives `ai_subscription_required`. |
| 403 | `/unlimited/*` without an active plan (`unlimited_plan_required` / `unlimited_plan_inactive`). |
| 404 | The route does not exist. |
| 413 | Payload too large. |
| 429 | Rate limited; honor `Retry-After`. On `/unlimited/*`, all concurrency seats are busy. |
| 500 / 503 | Server error / temporarily unavailable. `/unlimited/*` sends `503` with `Retry-After: 2` while your plan snapshot loads. |

Successful and rate-limited responses carry `RateLimit-Limit`,
`RateLimit-Remaining`, and `RateLimit-Reset`, plus `Retry-After` on `429`.
A successful `/unlimited/*` call and its seat-busy `429` also carry
`X-Concurrency-Limit` and `X-Concurrency-Active`.

### AI (prompt-guided) endpoints

Each takes a natural-language `prompt` (and a `url`, except `ai/search`) and
returns extracted/structured results instead of raw pages. Use these when you
want an answer, not a document. They require an active AI plan, billed
separately from credits: https://spider.cloud/ai/pricing. Without one the
call is refused with
`{"error":"AI Studio subscription required","code":"ai_subscription_required"}`.

| Endpoint | Body and result |
|---|---|
| `POST /ai/scrape` | `{ url, prompt }`. Extract exactly what the prompt asks from one page. |
| `POST /ai/crawl` | `{ url, prompt }`. Crawl a site and extract per the prompt. |
| `POST /ai/links` | `{ url, prompt }`. Return only the links the prompt is after. |
| `POST /ai/search` | `{ prompt }`. Search the web and return an extracted answer. |
| `POST /ai/browser` | `{ url, prompt }`. Drive a headless Browser to complete the prompt. |
| `POST /ai/unblocker` | `{ url, prompt }`. Extract per the prompt from a bot-walled page. |

```bash
curl -X POST https://api.spider.cloud/ai/scrape \
  -H "Authorization: Bearer $SPIDER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://example.com/pricing", "prompt": "Return each plan name and monthly price as JSON" }'
```

### Unlimited endpoints (Unlimited plan required)

Flat monthly rate billed by purchased concurrency seats instead of
per-request credits. Requires an active Unlimited subscription
(https://spider.cloud/pricing?plan=unlimited).

| Endpoint | Purpose |
|---|---|
| `POST /unlimited/scrape` | Same parameters as `/scrape`. |
| `POST /unlimited/crawl` | Same parameters as `/crawl`. |
| `POST /unlimited/links` | Same parameters as `/links`. |

There is no queueing: when all purchased seats are in flight, requests get
an immediate `429` with a `Retry-After` header. Retry with backoff. AI
parameters are a hard `400` naming the offending field; blocked: `prompt`,
`custom_prompt`, `extraction_prompt`, `extraction_schema`, `model`,
`openrouter_model`, `openrouter_fallback_model`, `vision_model`,
`return_embeddings`, `custom_function`, and `stealth` or `external_chrome`
set to `true`, plus `country_code` and a body-level `authorization`.
Use `/ai/*` for extraction. The refusal codes are in the error table above.
Docs: https://spider.cloud/docs/api/unlimited

Unlimited pairs with the default pay-as-you-go plan rather than replacing
it. The same API key uses both; keep AI extraction and one-off calls on the
standard credit-billed endpoints.

### Key parameters

These matter on most calls:

- `request`: `"http"` (fastest, no JS), `"browser"` (a headless Browser, runs
  JS), or `"smart"` (HTTP first, escalating to a browser only when the page
  needs it). `"smart"` is the default on the main crawl path; `/screenshot`
  defaults to `"http"`, so send `request: "smart"` there to escalate.
  `"chrome"` and `"headless"` are legacy aliases for `"browser"`;
  `"smart_mode"` and `"smartmode"` are aliases for `"smart"`. Spellings are
  matched exactly, in lowercase, Capitalized, or UPPERCASE. Any other casing
  and any unrecognized value silently falls back to `"http"`, so a typo here
  looks like a page that failed to render.
- `limit`: max pages for `/crawl`. Leave it unset for no page cap, in which
  case the crawl runs until your credits stop it.
- `return_format`: `"raw"` (the actual default), `"markdown"` (use this for
  LLM input), `"commonmark"`, `"text"`, `"html2text"`, `"xml"`, `"bytes"`,
  `"screenshot"`, `"empty"`. Pass an array to get several at once. There is
  no `"html"` value, and an unrecognized one silently becomes `"raw"`.
- `remote_proxy`: route the request through your own proxy, e.g.
  `"http://user:pass@host:port"`. Leave it unset to use Spider's network.
- `css_extraction_map`: pull specific fields from each page. Top-level keys
  are URL-path patterns (`"/"` matches everything). Each value is an array
  of `{ "name": "<field>", "selectors": ["<css>", ...] }` entries; list
  multiple selectors as fallbacks, and XPath works alongside CSS. Results
  come back under `css_extracted`.
- `wait_for`: when the browser considers a page "ready". Object with any of
  `selector`, `idle_network`, `idle_network0`, `almost_idle_network0`,
  `dom`, `delay`, plus the boolean `page_navigations`. The timeout fields
  are Rust `Duration`s, `{ "secs": <n>, "nanos": <n> }`; `selector` and
  `dom` also take a `selector` string. `page_navigations` is `true`/`false`,
  and sending it as an object is a `400`. Only applies when `request` is
  `"browser"` or `"smart"`.

Also worth knowing, one line each:

- `metadata`: return title, description, and other page metadata.
- `readability`: boilerplate removal before formatting.
- `depth`: max crawl depth for `/crawl`.
- `proxy` / `proxy_enabled` / `country_code`: pick Spider's proxy tier and
  exit country.
- `stealth`: handle bot walls, CAPTCHAs and geo checks inside the request
  and return the page. Rejected on `/unlimited/*`.
- `headers` / `cookies`: send your own request headers and cookies.
- `blacklist` / `whitelist`: path patterns to skip or allow (regex works).
- `root_selector` / `exclude_selector`: trim each page to, or strip, a
  selector before formatting.
- `filter_output_main_only`: drop nav, header, and footer chrome.
- `return_page_links` / `return_headers` / `return_cookies` /
  `return_json_data`: opt in to the matching response fields.
- `cache`: reuse a recent copy of a page. On by default.
- `webhooks`: push results to your own endpoint as they land.

Full parameter tables: https://spider.cloud/llms-full.txt

```bash
curl -X POST https://api.spider.cloud/scrape \
  -H "Authorization: Bearer $SPIDER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://store.example.com/product/123",
    "request": "browser",
    "wait_for": {
      "selector": { "selector": "h1.product-title", "timeout": { "secs": 10, "nanos": 0 } },
      "idle_network": { "timeout": { "secs": 5, "nanos": 0 } }
    },
    "css_extraction_map": {
      "/": [
        { "name": "title",  "selectors": ["h1.product-title"] },
        { "name": "price",  "selectors": [".price-value"] },
        { "name": "images", "selectors": ["img.hero-image", "img.gallery"] }
      ]
    }
  }'
```

Routing a scrape through your own proxy:

```bash
curl -X POST https://api.spider.cloud/scrape \
  -H "Authorization: Bearer $SPIDER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "remote_proxy": "http://user:pass@proxy.example.com:8080"
  }'
```

Full reference: https://spider.cloud/docs/api

---

## B. Client SDK

The SDKs mirror the endpoints above: `scrape`, `crawl`, `links`, `search`,
`screenshot`, `transform`. Python and JS read `SPIDER_API_KEY` when
constructed with no arguments; Go wants `spider.New("")` and Rust
`Spider::new(None)` to pull it from the environment. All of them also accept
the key directly in the constructor.

**Python** `pip install spider_client`

```python
from spider import Spider

app = Spider()  # reads SPIDER_API_KEY

# One page
page = app.scrape_url("https://example.com")

# A whole site, escalating to Chrome only when needed
result = app.crawl_url(
    "https://example.com",
    params={"limit": 200, "request": "smart"},
)

# Prompt-guided extraction. Returns structured results, not raw pages.
# Also: ai_crawl(url, prompt), ai_links(url, prompt), ai_browser(url, prompt),
# and ai_search(prompt), which takes no url.
data = app.ai_scrape(
    "https://example.com/pricing",
    "Return each plan name and monthly price as JSON",
)
```

For large crawls, stream instead of buffering. Pass `stream=True` **and** a
`callback`; the callback is what switches the request to `application/jsonl`
and hands you each page as it arrives. `stream=True` alone returns the raw
streamed `requests` response for you to parse yourself.

```python
def on_page(page: dict) -> None:
    print(page["url"], page["status"])

app.crawl_url("https://example.com", params={"limit": 200},
              stream=True, callback=on_page)
```

The same AI methods exist in the JS client as `aiScrape`, `aiCrawl`,
`aiLinks`, `aiBrowser`, and `aiSearch(prompt)`.

**JavaScript / TypeScript** `npm install @spider-cloud/spider-client`

```ts
import { Spider } from "@spider-cloud/spider-client";

const app = new Spider(); // reads SPIDER_API_KEY
const page = await app.scrapeUrl("https://example.com");
const site = await app.crawlUrl("https://example.com", { limit: 200, request: "smart" });
```

**Go** `go get github.com/spider-rs/spider-clients/go`

**Rust** add `spider-client` to `Cargo.toml` (`cargo add spider-client`).

**CLI** `cargo install spider-cloud-cli`

```bash
spider-cloud-cli auth --api-key $SPIDER_API_KEY
spider-cloud-cli scrape --url https://example.com
spider-cloud-cli crawl  --url https://example.com --limit 50
spider-cloud-cli search --query "rust web crawler"
```

SDK docs and examples: https://spider.cloud/docs/libraries

---

## C. Self-host (open-source engine)

Use the `spider` crate when the crawl must run inside your own
infrastructure. This is the engine itself, not the Cloud client. No API key,
no `api.spider.cloud`.

```toml
[dependencies]
spider = "2"
```

```rust
use spider::{tokio, website::Website};

#[tokio::main]
async fn main() {
    let mut website = Website::new("https://example.com");
    let mut rx = website.subscribe(16);

    tokio::spawn(async move {
        while let Ok(page) = rx.recv().await {
            println!("{}  {}", page.status_code, page.get_url());
        }
    });

    website.crawl().await;
    website.unsubscribe();
}
```

Pages stream as they arrive; the crawl stops when nothing is left to fetch.
Prefer a command-line tool instead of a library? `cargo install spider_cli`.

Source and full docs: https://github.com/spider-rs/spider

---

## D. MCP server (Claude, Cursor, agents)

Spider runs a hosted Model Context Protocol server so an AI tool can call the
crawler directly as a tool. Nothing to install.

Endpoint (streamable HTTP):

```
https://mcp.spider.cloud/mcp
```

On first connect the server runs an OAuth flow that mints a scoped API key for
you, so you do not need to create one by hand. A bearer key works too if you
already have one.

**Claude Code**

```bash
claude mcp add spider --transport http https://mcp.spider.cloud/mcp \
  -H "Authorization: Bearer your-key"
```

**Cursor, Windsurf, VS Code, Claude Desktop**: add the endpoint to your
client's MCP config:

```json
{
  "mcpServers": {
    "spider": {
      "type": "url",
      "url": "https://mcp.spider.cloud/mcp",
      "headers": {
        "Authorization": "Bearer your-api-key"
      }
    }
  }
}
```

Per-client walkthroughs, including one-click install links for Cursor and VS
Code: https://spider.cloud/mcp/connect

**Fallbacks.** Use these only if you cannot reach the hosted server.

- Local process, talking to Spider Cloud: `npx spider-cloud-mcp`
  (https://www.npmjs.com/package/spider-cloud-mcp)
- Fully self-hosted, on top of your own open-source engine from section C:
  `cargo install spider_mcp`
  (https://github.com/spider-rs/spider/tree/main/spider_mcp)

---

## Choosing an endpoint

- A single known page → `scrape`.
- An entire site / many pages → `crawl` with a `limit`.
- Just the URLs, not content → `links`.
- "Find pages about X on the web" → `search`, or `ai/search` to get an
  extracted answer.
- HTML you already fetched, want clean Markdown → `transform`.
- Site renders content with JavaScript → set `request: "browser"`, or
  leave it on `"smart"` and let Spider decide.
- Page blocks normal requests (bot wall, CAPTCHA, fingerprinting) →
  `unblocker`.
- You need specific fields, not the whole page → `css_extraction_map`
  with selectors you already know. For natural-language extraction
  instead, use the `ai/*` endpoints (`ai_scrape`, `ai_crawl`, `ai_links`,
  `ai_search`, `ai_browser`): pass a `prompt` and get structured results
  back.
- Chrome page isn't ready when it returns → add `wait_for` with a
  `selector` or `idle_network`.
- High, steady volume at a flat monthly rate → the `/unlimited/*`
  variants of `scrape`/`crawl`/`links` (requires an Unlimited plan; pairs
  with pay-as-you-go, so AI and one-off calls stay on the standard
  credit-billed endpoints).

## Links

- Full parameter reference (machine-readable, read this next):
  https://spider.cloud/llms-full.txt
- OpenAPI spec: https://spider.cloud/openapi.yaml
- API reference: https://spider.cloud/docs/api
- SDKs & quickstart: https://spider.cloud/docs/libraries
- MCP setup: https://spider.cloud/mcp/connect
- Open source: https://github.com/spider-rs/spider
- Clients: https://github.com/spider-rs/spider-clients