# Spider API overview: endpoints, plans and credits | Spider

Every Spider API endpoint in one page, with the Unlimited and AI Studio plans and how credits are billed.

Canonical: https://spider.cloud/docs/overview/

One API for crawling, scraping, search, screenshots, and AI extraction. Bearer auth, JSON in, JSON out, optional streaming. Pay per request with credits, or run unlimited requests on an Unlimited concurrency plan.

quickstart [Developer quickstart Install the SDK, export your API key, and make your first request. 3 mins](https://spider.cloud/docs/quickstart/)

Request

```
import requests, os, json

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

json_data = {"limit":5,"url":"https://example.com"}

response = requests.post('https://api.spider.cloud/crawl', 
  headers=headers, json=json_data, stream=True)

with response as r:
    r.raise_for_status()
    
    buffer = b""

    for chunk in response.iter_content(chunk_size=8192):
        if chunk:
            buffer += chunk
            try:
                data = json.loads(buffer.decode('utf-8'))
                print(data)
                buffer = b""
            except json.JSONDecodeError:
                continue
```

Crawl Follow links across a whole site. Set depth, limit, and domain scope.

Scrape Fetch a single page as HTML, markdown, text, or structured JSON.

Search Search the web and scrape the results in one request.

Screenshot Capture full-page screenshots with Chrome rendering.

Streaming Process pages as they finish instead of waiting for the full result.

AI extraction Extract structured data from any page with AI or CSS selectors.

Data connectors Stream results straight to S3, Google Cloud, Azure Blob, Sheets, or Supabase.

Protected pages Scrape loads demanding pages through a native browser, proxies, and retries.

Proxy mode Geo-routing across residential proxies in 199+ countries.

Cloud browser Cloud browsers over a CDP WebSocket. Playwright and Puppeteer compatible, with a native browser engine, proxies, and recording.

Fetch API (alpha) AI-configured per-website scrapers. Finds selectors, then caches and reuses them.

## How it works

Every request runs three stages concurrently. A 500-page crawl typically completes in under 20 seconds. Every response includes a `costs` object and timing data so you can benchmark it yourself.

Fetch Retrieve the page with HTTP or a headless browser session.

Process Render JavaScript, rotate proxies, complete challenge pages.

Deliver Convert to markdown, HTML, JSON, or bytes and stream back.

## API endpoints

All endpoints accept JSON and return JSON. Authenticate with a Bearer token. For one page at a time, the [scraper API](https://spider.cloud/api/scrape/) page shows the request, the response and what it costs.

| Method | Path                                                                        | Description                                                           |
| ------ | --------------------------------------------------------------------------- | --------------------------------------------------------------------- |
| POST   | [/crawl](https://spider.cloud/docs/core/scraping-crawling/)                 | Start from a URL and follow links to fetch many pages.                |
| POST   | [/scrape](https://spider.cloud/docs/core/scraping-crawling/)                | Fetch a single page and return its content in any format.             |
| POST   | [/search](https://spider.cloud/docs/core/realtime-search/)                  | Search the web and, if you ask, scrape the results.                   |
| POST   | [/screenshot](https://spider.cloud/docs/core/scraping-crawling/)            | Capture a full-page screenshot as base64 PNG.                         |
| POST   | [/unlimited/\*](https://spider.cloud/docs/api/unlimited/)                   | Scrape, crawl, and links for one monthly price. Unlimited             |
| POST   | [/ai/\*](https://spider.cloud/docs/ai-studio/)                              | Prompt-guided crawl, scrape, search, browser, and links. AI Studio    |
| POST   | [/fetch/{domain}/{path}](https://spider.cloud/api/fetch/)                   | AI-configured per-website scraper with cached configs. Alpha.         |
| GET    | [/data/scraper-directory](https://spider.cloud/docs/api/#scraper-directory) | Browse scraper configs for popular websites.                          |
| HTTP   | [proxy.spider.cloud](https://spider.cloud/docs/api/proxy/)                  | Route requests through residential proxies.                           |
| WS     | [browser.spider.cloud](https://spider.cloud/docs/api/browser/)              | Connect a Playwright or Puppeteer client to a cloud browser over CDP. |

## Unlimited plan

Buy requests at once instead of paying per request. A request counts as 1 in http mode and 2 in browser mode. Smart mode, the default, counts as 1, and as 2 once a page needs a browser. Everything you run through it is covered by the monthly price. The plan starts at $40 a month for 2 requests at once in http mode. It scales to any whole number you need, and the price for each drops from $20 to $13 as you add more. When the plan is full, a new request gets a 429 at once and adds nothing to the bill. The `429` carries a `Retry-After` header, and nothing queues. See the [Unlimited reference](https://spider.cloud/docs/api/unlimited/), [pricing](https://spider.cloud/pricing/?plan=unlimited), or [manage your plan](https://spider.cloud/account/plan/).

`/unlimited/scrape` One page, same parameters as /scrape.

`/unlimited/crawl` Follow links across a site, same as /crawl.

`/unlimited/links` Collect links only, same as /links.

Requests and responses match the standard endpoints, so moving a job onto Unlimited means changing the path and nothing else. Every route also answers at `/v1/unlimited/...`. AI endpoints stay metered in credits and are billed separately.

## AI endpoints

Describe the result you want in a prompt and Spider works out the selectors, pagination, and clicks. These routes need an AI Studio plan on the account, and the calls are metered in credits on top of it rather than covered by Unlimited. See [AI Studio](https://spider.cloud/docs/ai-studio/) for parameters and [AI pricing](https://spider.cloud/ai/pricing/) for per-call rates.

`/ai/scrape` Pull structured fields off a single page from a prompt.

`/ai/crawl` Crawl a site and extract the same shape from every page.

`/ai/search` Search the web and extract from the results.

`/ai/links` Find the links that match what you asked for.

`/ai/browser` Drive a real browser through steps you describe.

AI Studio plans

Starter $6/mo 1 req/s

Lite $30/mo 5 req/s

Standard $125/mo 10 req/s

Scale $600/mo 25 req/s

Enterprise $2,000/mo Custom

Every plan includes all five AI routes. The plan sets your rate limit, and credits for each call are billed on top. Start or change one on the [AI Studio pricing page](https://spider.cloud/ai/pricing/).

## Request modes

Pick how Spider fetches each page. See [Concepts](https://spider.cloud/docs/concepts/) for the trade-offs.

`smart` Default

Picks between HTTP and browser for each page.

`http` Fast

Static HTML only. The fastest and cheapest mode.

`browser` JS / SPA

Full browser rendering. Use for SPAs or protected sites.

## Proxy mode

Route any Spider request through `proxy.spider.cloud` and Spider sends it through residential proxies, rotates IPs, and handles geo-routing. Use `country_code` to target a country. Works with [Crawl](https://spider.cloud/docs/api/crawl/), [Scrape](https://spider.cloud/docs/api/scrape/), [Screenshot](https://spider.cloud/docs/api/screenshot/), [Search](https://spider.cloud/docs/api/search/), and [Links](https://spider.cloud/docs/api/links/). See the [Proxy API reference](https://spider.cloud/docs/api/proxy/) for pricing.

## Cloud browser

Cloud browsers over a CDP WebSocket at `wss://browser.spider.cloud/v1/browser?token=YOUR-API-KEY`. Connect Playwright or Puppeteer with `connectOverCDP()`. Sessions run on our native browser engine, with proxy rotation and optional [session recording](https://spider.cloud/playground/). An account holds up to 1,000 open sessions and opens up to 100 new ones a minute. See the [cloud browser reference](https://spider.cloud/docs/api/browser/) or grab the [spider-browser](https://www.npmjs.com/package/spider-browser) npm package.

## Credits

Pay as you go usage is measured in credits at `$1 per 10,000 credits`. Each page has a base cost; Chrome rendering, proxy usage, and AI extraction add on top. An attempt bills for the bandwidth and compute it used, whatever status the page returns. A failed page is not free. Every response includes a `costs` object with a per-request breakdown, and the live balance is on the [usage page](https://spider.cloud/account/plan/). Zero data retention, an optional account setting, bills requests at 2.5x credits on self-serve accounts. Requests on Unlimited routes meter no credits at all.

[agent setup](/agent-skill/SKILL.md)

[SKILL.md](/agent-skill/SKILL.md)

[Point Claude, Cursor, or any agent at this file to bootstrap a Spider integration.](/agent-skill/SKILL.md)

[/agent-skill/SKILL.md ↗](/agent-skill/SKILL.md)

SDK clients

[JavaScript](https://github.com/spider-rs/spider-clients/tree/main/javascript)

[npm install](https://github.com/spider-rs/spider-clients/tree/main/javascript)

[ ](https://github.com/spider-rs/spider-clients/tree/main/javascript)

[Python](https://github.com/spider-rs/spider-clients/tree/main/python)

[pip install](https://github.com/spider-rs/spider-clients/tree/main/python)

[ ](https://github.com/spider-rs/spider-clients/tree/main/python)

[Rust](https://github.com/spider-rs/spider-clients/tree/main/rust)

[cargo add](https://github.com/spider-rs/spider-clients/tree/main/rust)

[ ](https://github.com/spider-rs/spider-clients/tree/main/rust)

[CLI](https://github.com/spider-rs/spider-clients/tree/main/cli)

[terminal](https://github.com/spider-rs/spider-clients/tree/main/cli)

Guides

- [developers web-scraping tutorials Extract structured JSON from a web page with one call Send a URL and a map of CSS selectors to /scrape and get named fields back as JSON. One call returned 250 rows of four fields in about 1.5 seconds on two days. 5 min Nov 11, 2026](https://spider.cloud/guides/structured-json-extraction-from-pages/)
- [AI Give a Microsoft Agent Framework agent the web Two ways to let a Microsoft Agent Framework agent crawl, scrape and search: point it at the hosted Spider MCP server, or wrap the Python SDK in a @tool function. 3 min Sep 22, 2026](https://spider.cloud/guides/microsoft-agent-framework/)
- [developers pricing Spider pricing and plans: what you pay and why A breakdown of Spider's pay-as-you-go pricing, per-endpoint costs, volume discounts, and how billing works for web scraping and browser automation. 7 min Mar 8, 2026](https://spider.cloud/guides/pricing-and-plans/)
- [developers self-hosting rust Self-hosting Spider: using the open source Rust crate Build your own web crawler with the open source spider Rust crate. Quick start, Docker setup, configuration, and when to upgrade to the cloud API. 5 min Feb 20, 2026](https://spider.cloud/guides/self-hosting-spider/)
- [developers open-source rust The Spider open source ecosystem A guide to all open source Spider projects: the core crawler, browser client, HTML transformer, TLS fingerprinting, and more. Quick-start examples for each. 5 min Feb 20, 2026](https://spider.cloud/guides/spider-oss-ecosystem/)
- [SERP web-scraping Web search API (SERP) to search and scrape A web search API that returns SERP results and can scrape every page in the same call. Parameters, geo and time filters, pagination, Python examples. 6 min Mar 22, 2025](https://spider.cloud/guides/search/)
- [web-scraping headless-browser technology Scaling headless Chrome for high-performance applications Practical strategies for scaling headless Chrome, from container orchestration to Rust-based CDP handlers and ALB configuration. 3 min Feb 21, 2025](https://spider.cloud/guides/scaling-headless-chrome/)
- [discord AI Discord real-time data retrieval Set up Spider Bot on your Discord server to fetch and analyze web data using slash commands. 1 min Aug 25, 2024](https://spider.cloud/guides/discord-chat-spider/)
- [AI Build an AI agent from scratch in Python Build an AI agent from scratch in Python, no framework, in about 100 lines. It writes its own search queries, searches the web, and refines until it answers. 8 min Jul 9, 2024](https://spider.cloud/guides/ai-agent-from-scratch/)
- [development Static site search with Pagefind and the Spider CLI Crawl your own site with the Spider CLI, index the output with Pagefind, and ship full-text search as static files. No search service, no monthly bill. 5 min Jul 2, 2024](https://spider.cloud/guides/automated-free-website-static-search/)
- [AI Speedy resilient web scraper for RAG AI: part 1 Choosing your scraper, cleaning HTML for RAG, deduplicating content, and testing on a single site before scaling up. 10 min Jun 22, 2024](https://spider.cloud/guides/building-a-speedy-resilient-web-scraper-for-rag-ai-part1-preparing/)
- [AI Scaling a resilient web scraper for RAG: part 2 How one RAG project scraped 1.2 million pages: retry logic with backoff, hard timeouts, a fleet of t2.nano workers, deduplication and data quality checks. 16 min Jun 22, 2024](https://spider.cloud/guides/building-a-speedy-resilient-web-scraper-for-rag-ai-part2-scaling-up/)
- [web-scraping Crawling authenticated pages with cookies or scripts Scrape pages behind a login with Spider. Pass a session cookie, or run an execution script that fills the login form, then crawl the authenticated pages. 3 min Jun 14, 2024](https://spider.cloud/guides/crawling-authenticated-pages/)
- [web-scraping Spider proxy: rotating residential IPs over HTTP or SOCKS5 Route any HTTP, HTTPS or SOCKS5 client through proxy.spider.cloud with your API key as the username. Rotating residential IPs, from $1 per GB transferred. 2 min Jun 6, 2024](https://spider.cloud/guides/proxy-mode/)
- [AI Scrape & crawl agent with Microsoft's Autogen Set up an Autogen agent that scrapes and crawls websites using the Spider API. 5 min May 20, 2024](https://spider.cloud/guides/crawl-agent-with-autogen/)
- [outreach Automated cold email outreach using Spider Extract company info from inbound emails, scrape their website with Spider, and generate personalized replies with RAG. 6 min May 17, 2024](https://spider.cloud/guides/auto-email-response-outreach/)
- [AI Stock research assistant using crewAI and Spider Build a crewAI research pipeline that uses Spider to scrape financial data and write stock analysis reports. 9 min May 12, 2024](https://spider.cloud/guides/crewai-spider-research-agent/)
- [AI LangChain + Groq + Spider integration guide Crawl multiple URLs with Spider's LangChain loader, then summarize the results with Groq and Llama 3. 5 min May 4, 2024](https://spider.cloud/guides/langchain-groq/)
- [developers web-scraping Website archiving Archive web pages with Spider. Capture full page resources, automate regular crawls, and store content for long-term access. 2 min Feb 7, 2024](https://spider.cloud/guides/website-archiving/)
- [outreach Extract leadership contact information from websites Extract leadership contact information from websites with Spider's API or dashboard. Crawl team and about pages, then get names, titles, emails, and phones. 2 min Feb 1, 2024](https://spider.cloud/guides/pipelines-extract-contacts/)
- [developers web-scraping Spider API: endpoints, quick start and SDKs The Spider API turns any URL into markdown, HTML, text or JSON. Crawl, scrape and search endpoints, a Python quick start, SDKs, and $1 per 10,000 credits. 3 min Jan 3, 2024](https://spider.cloud/guides/spider-api/)
- [developers web-scraping Web crawling and scraping API guide Crawl and scrape any website with the Spider API. Create a key, run your first crawl, then tune request modes, proxies, streaming, and markdown for LLMs. 6 min Jan 2, 2024](https://spider.cloud/guides/spider/)
