Web scraping API. Known URLs in, named fields out.

POST the pages you already have. Spider fetches each one, renders JavaScript when the page calls for it, and returns markdown, HTML, or text, plus the exact fields you mapped with selectors. Pay as you go, from $1 per GB.

Start scraping Scrape API docs

Response 200

[
  {
    "url": "https://store.example.com/products/aria-lamp",
    "status": 200,
    "content": "# Aria Desk Lamp...",
    "css_extracted": {
      "price": ["$89.00"],
      "stock": ["In stock"]
    }
  }
]

Scrape fetches the URLs you name and never follows a link. When the job is finding those URLs in the first place, that is Crawl . The two share parameters and response shape, so nothing you build here is throwaway.

The selector map is the parser.

Name a field, point it at a selector, and the response comes back already parsed. The extraction code you would write and maintain lives in the request instead.

Each field in the selector map and the value it returns
Field	Selector	Comes back as
name	h1.product-title	["Aria Desk Lamp"]
price	.price .value	["$89.00"]
stock	.stock-status	["In stock"]

POST /scrape

{
  "url": "https://store.example.com/products/aria-lamp",
  "return_format": "markdown",
  "css_extraction_map": {
    "/products": [
      { "name": "name",  "selectors": ["h1.product-title"] },
      { "name": "price", "selectors": [".price .value"] },
      { "name": "stock", "selectors": [".stock-status"] }
    ]
  }
}

Response 200

[
  {
    "url": "https://store.example.com/products/aria-lamp",
    "status": 200,
    "css_extracted": {
      "name":  ["Aria Desk Lamp"],
      "price": ["$89.00"],
      "stock": ["In stock"]
    },
    "content": "# Aria Desk Lamp..."
  }
]

Map keys are URL paths. One request can carry a map for /products and another for /blog, and each page gets the one that matches. The price and stock map above is the shape behind price monitoring .

Fields take CSS or XPath selectors, and more than one per field, so a single map survives template changes.

Does the page need a browser?

The request parameter decides how Spider fetches, and it sets most of what a request costs and how long it takes. Pick by what you already know about the page.

"request": "http"

You know the page renders on the server.

A plain fetch with no browser involved. It is the fastest and cheapest way to scrape, and the right call for docs, feeds, and any page that already works with curl.

Speed: Fastest
Cost: Lowest
JS rendering: No

"request": "smart"

Default

You are not sure, or the batch is mixed.

Inspects each page and picks HTTP or Chrome per URL. You pay for rendering only on the pages that turn out to need it.

Speed: Fast
Cost: Low to medium
JS rendering: When detected

"request": "browser"

You know it is an SPA or bot protected.

Full Chrome rendering without a fleet of your own to babysit. Also the mode that runs evaluate_on_new_document, your JavaScript on the page before extraction.

Speed: Slower
Cost: Higher
JS rendering: Always

Every response carries a costs object with compute and data transfer split out, so you can see what each mode choice cost. The rates are on the pricing page .

When a selector is not the right tool.

Embedded JSON you never mapped, article text with the chrome stripped, and pages where your own script has to run first.

Embedded JSON

Set return_json_data and JSON-LD, product schema, and the state objects frameworks ship inside the page come back under json_data. Often richer than anything visible on the page.

Article text

Readability mode strips navigation, sidebars, and ads down to the main body. Pair it with return_format: markdown for text an LLM can use directly.

Batches

Comma-separate URLs or send an array of objects. The whole list goes out in one request and comes back as one array.

Your own JavaScript

Pass a script to evaluate_on_new_document and it runs before extraction. Dismiss a cookie modal, open a tab, reshape the DOM. Needs a browser request.

Your first request.

Official Python and JavaScript clients, or plain HTTP with your API key.

Run this

curl -X POST https://api.spider.cloud/scrape \
  -H "Authorization: Bearer $SPIDER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://store.example.com/products/aria-lamp",
    "return_format": "markdown",
    "css_extraction_map": {
      "/products": [
        { "name": "price", "selectors": [".price .value"] },
        { "name": "stock", "selectors": [".stock-status"] }
      ]
    }
  }'

from spider import Spider

client = Spider()

result = client.scrape(
    "https://store.example.com/products/aria-lamp",
    params={
        "return_format": "markdown",
        "css_extraction_map": {
            "/products": [
                {"name": "price", "selectors": [".price .value"]},
                {"name": "stock", "selectors": [".stock-status"]},
            ]
        },
    },
)

print(result[0]["css_extracted"])

import Spider from "@spider-cloud/spider-client";

const client = new Spider();

// A comma-separated list is one batch request
const pages = await client.scrape(
  "https://example.com/pricing,https://example.com/docs",
  {
    return_format: "markdown",
    return_json_data: true,
  },
);

One object per URL.

The response is an array with one entry per page, whether you sent one URL or fifty.

url Final URL after redirects
status HTTP status of the fetch
content The page in your return_format: markdown, raw HTML, text, or bytes
css_extracted Named fields from your selector map
json_data JSON-LD and embedded objects, when return_json_data is on
metadata Title, description, keywords
links Every link found, when return_page_links is on
headers Response headers, when return_headers is on
cookies Cookies the page set, when return_cookies is on
costs What this request cost, compute and transfer split out
error Set per page, so one bad URL does not fail the batch

More from the API.

Crawl API

Discover and fetch every page under a domain

Transform API

Markdown from HTML or PDF you already have

JSON scraping guide

Patterns for return_json_data and embedded state

Stop maintaining parsers.

Selector maps, embedded JSON, readability, and full browser rendering behind one POST request. Pay as you go from $1 per GB with no subscription.

Create your API key Read the scrape docs