# AI Web Crawler, Scraping API and Cloud Browser | Spider

The web has an API now. Scrape, crawl and automate any website on our own cloud browser and proxy network. Markdown or JSON back, priced per page.

Canonical: https://spider.cloud/

## The web has an API now.

Scrape, crawl and search the web from your agent or RAG pipeline. [One call](/api/) returns markdown or JSON, and Spider runs the [browsers](/browser/) and [proxies](/proxy-locations/).

[Get an API key](https://spider.cloud/register/)

No card required.

Request

POST api.spider.cloud/scrape

url

This is what your agent gets back.

Sample response, captured November 18, 2026.

\## Introduction

Go is an open-source programming language that focuses on simplicity, reliability,

and efficiency, specifically designed to make it easy to build software at scale.

Although it borrows ideas from existing languages,

it has unusual properties that make effective Go programs

different in character from programs written in its relatives.

A straightforward translation of a C++ or Java program into Go

- Time

  Run to measure

- Cost

  Run to measure

- Status and size

  - 200
  - 100.2 KB

From $0.16 per 1,000 pages. Cost varies by site. An attempt bills only the bandwidth it used and the time it ran. [Get an API key](/register/)

Try

- [api.spider.cloud ](/api/)POST

  10,000 requests per minute

  Scrape, crawl and search on one host. The limit starts here and rises to 80,000.

- [browser.spider.cloud ](/browser/)WS

  1,000 sessions open at once

  A browser we run for you. Your code drives it over a WebSocket.

- [proxy.spider.cloud ](/proxy-locations/)HTTP

  199 countries

  Residential IP addresses, rotated per request.

One key opens all three.

- [999 of 1,000 URLs returned content, February 2026](/benchmarks/#api-1000-urls-2026-02-11)
- [Open source under the MIT license](https://github.com/spider-rs/spider)
- 2.9k GitHub stars

[Integrations](/docs/integrations/overview/)

## Use the whole web like a dataset.

Spider is a research lab for web data quality. Most websites have no API, so we ship one for your [RAG pipeline](/use-cases/rag/) or [agent](/use-cases/ai-agents/). Run the [web scraper](/web-scraper/) before you sign up, or read our [web scraping](/web-scraping/) guide.

1. ### Renders like a user

   Scripts run before we read the page, so you get what a person sees.

   `request: "browser"` [See the browser](/browser/)

2. ### Loads the protected page

   Some pages refuse a plain fetch. This loads them in a full browser.

   `stealth: true` [See the scrape endpoint](/api/scrape/)

3. ### Extracts the content

   The page comes back as markdown or plain text instead of raw HTML.

   `return_format: "markdown"` [See the formats](/docs/api/scrape/)

4. ### Streams as it crawls

   In a crawl, each page arrives as one JSON line when it finishes.

   `application/jsonl` [See streaming](/docs/core/concurrent-streaming/)

## Pay for what you use, or a flat month.

From $0.16 per 1,000 pages. Cost varies by site.

### pay as you go

from $1 per GB

No subscription. Top up from $5.

- Time a request runs

  $0.0001 per minute

- Balance

  Never expires

[Get an API key](/register/)

### unlimited

$40 a month

For 2 requests at once in http mode.

- Requests at once

  from 2

- Requests

  unlimited

- Per request at once

  $20 down to $13

[See unlimited plans](/pricing/#unlimited)

## Make your first request.

Pick your client and paste.

### 01 Get a key

[Get an API key](/register/)

Your new account gets 2,500 free credits, worth $0.25, if your signup passes our checks.

```
export SPIDER_API_KEY="sk-..."
```

### 02 Connect Claude Code

```
claude mcp add spider --transport http \
  https://mcp.spider.cloud/mcp \
  -H "Authorization: Bearer $SPIDER_API_KEY"
```

CLAUDE.md

```
For web pages use spider-agent. Read https://spider.cloud/agent-skill/spider-agent/SKILL.md first.
```

Or the CLI. It picks HTTP or a browser per page, stops at your budget, and returns 187 bytes for named fields where asking for everything returns 9021.

[Client configs in the docs](/docs/integrations/mcp/)

### 02 Connect Codex

```
Read and follow https://spider.cloud/agent-skill/SKILL.md
```

Paste into a Codex session. The skill covers the whole API.

AGENTS.md

```
For web pages use spider-agent. Read https://spider.cloud/agent-skill/spider-agent/SKILL.md first.
```

Or the CLI. It picks HTTP or a browser per page, stops at your budget, and returns 187 bytes for named fields where asking for everything returns 9021.

[Every command in the docs](/docs/integrations/agent-cli/)

### 02 Connect your MCP client

```
Endpoint:  https://mcp.spider.cloud/mcp
Transport: Streamable HTTP
Auth:      Authorization: Bearer sk-...
```

Works in Claude Desktop, Cursor, Windsurf, or any MCP client.

[Client configs in the docs](/docs/integrations/mcp/)

### 02 Install the CLI and run it

```
curl -fsSL https://spider.cloud/install/spider-agent.sh | sh
```

Downloads the release build, checks its SHA256, and puts it in \~/.local/bin. With a Rust toolchain, cargo install spider-agent-cli works too.

```
spider-agent scrape https://example.com --goal markdown --json
```

Picks HTTP or a browser per page, escalates on a refusal, stops at the budget you set.

[Every command in the docs](/docs/integrations/agent-cli/)

### 02 Make the first request

```
curl -X POST https://api.spider.cloud/scrape \
  -H "Authorization: Bearer $SPIDER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "return_format": "markdown"}'
```

[Full example in the docs](/docs/quickstart/)

### 02 Install the SDK and run it

```
pip install spider_client
```

```
from spider import Spider
app = Spider()  # reads SPIDER_API_KEY
page = app.scrape_url("https://example.com")
```

[Full example in the docs](/docs/libraries/)

### 02 Install the SDK and run it

```
npm install @spider-cloud/spider-client
```

```
import { Spider } from "@spider-cloud/spider-client";
const app = new Spider(); // reads SPIDER_API_KEY
const page = await app.scrapeUrl("https://example.com");
```

[Full example in the docs](/docs/libraries/)

### 03 Read the data

example output example data after a scrape

```
[{
  "url": "https://example.com",
  "status": 200,
  "content": "# Example Domain\n\n..."
}]
```

[Quickstart in the docs](/docs/quickstart/)

## Start with one URL.

[Get an API key](/register/)

No card required.
