Skip to main content

Known URLs in. Named fields out.

POST the pages you already have. Spider fetches each one, renders JavaScript when the page calls for it, and returns markdown, HTML, or text, plus the exact fields you mapped with selectors. Pay as you go, from $1 per GB.

Start scraping Scrape API docs
Response 200
[
  {
    "url": "https://store.example.com/products/aria-lamp",
    "status": 200,
    "content": "# Aria Desk Lamp...",
    "css_extracted": {
      "price": ["$89.00"],
      "stock": ["In stock"]
    }
  }
]
Scrape fetches the URLs you name and never follows a link. When the job is finding those URLs in the first place, that is Crawl . The two share parameters and response shape, so nothing you build here is throwaway.

The selector map is the parser.

Name a field, point it at a selector, and the response comes back already parsed. The extraction code you would write and maintain lives in the request instead.

Each field in the selector map and the value it returns
FieldSelectorComes back as
nameh1.product-title["Aria Desk Lamp"]
price.price .value["$89.00"]
stock.stock-status["In stock"]
POST /scrape
{
  "url": "https://store.example.com/products/aria-lamp",
  "return_format": "markdown",
  "css_extraction_map": {
    "/products": [
      { "name": "name",  "selectors": ["h1.product-title"] },
      { "name": "price", "selectors": [".price .value"] },
      { "name": "stock", "selectors": [".stock-status"] }
    ]
  }
}
Response 200
[
  {
    "url": "https://store.example.com/products/aria-lamp",
    "status": 200,
    "css_extracted": {
      "name":  ["Aria Desk Lamp"],
      "price": ["$89.00"],
      "stock": ["In stock"]
    },
    "content": "# Aria Desk Lamp..."
  }
]

Map keys are URL paths. One request can carry a map for /products and another for /blog, and each page gets the one that matches.

Fields take CSS or XPath selectors, and more than one per field, so a single map survives template changes.

Does the page need a browser?

The request parameter decides how Spider fetches, and it is the main speed and cost lever on this endpoint. Pick by what you already know about the page.

"request": "http"

You know the page renders on the server.

A plain fetch, no browser involved. The fastest and cheapest way to scrape, and the right call for docs, feeds, and any page that already works with curl.

Speed
Fastest
Cost
Lowest
JS rendering
No

"request": "smart"

Default

You are not sure, or the batch is mixed.

Inspects each page and picks HTTP or Chrome per URL. You pay for rendering only on the pages that turn out to need it.

Speed
Fast
Cost
Low to medium
JS rendering
When detected

"request": "browser"

You know it is an SPA or bot protected.

Full Chrome rendering without a fleet of your own to babysit. Also the mode that runs evaluate_on_new_document, your JavaScript on the page before extraction.

Speed
Slower
Cost
Higher
JS rendering
Always

No guessing what a mode choice cost you. Every response carries a costs object with compute and data transfer split out, per request.

When a selector is not the right tool.

Structured data you never mapped, clean article text, and pages that need convincing first.

Embedded JSON

Set return_json_data and JSON-LD, product schema, and the state objects frameworks ship inside the page come back under json_data. Often richer than anything visible on the page.

Article text

Readability mode strips navigation, sidebars, and ads down to the main body. Pair it with return_format: markdown for text an LLM can use directly.

Batches

Comma-separate URLs or send an array of objects. One request, one response array, one round trip for the whole list.

Your own JavaScript

Pass a script to evaluate_on_new_document and it runs before extraction. Dismiss a cookie modal, open a tab, reshape the DOM. Needs a browser request.

Your first request.

Official Python and JavaScript clients, or plain HTTP with your API key.

curl -X POST https://api.spider.cloud/scrape \
  -H "Authorization: Bearer $SPIDER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://store.example.com/products/aria-lamp",
    "return_format": "markdown",
    "css_extraction_map": {
      "/products": [
        { "name": "price", "selectors": [".price .value"] },
        { "name": "stock", "selectors": [".stock-status"] }
      ]
    }
  }'

One object per URL.

The response is an array with one entry per page, whether you sent one URL or fifty.

  • url Final URL after redirects
  • status HTTP status of the fetch
  • content The page in your return_format: markdown, raw HTML, text, or bytes
  • css_extracted Named fields from your selector map
  • json_data JSON-LD and embedded objects, when return_json_data is on
  • metadata Title, description, keywords
  • links Every link found, when return_page_links is on
  • headers Response headers, when return_headers is on
  • cookies Cookies the page set, when return_cookies is on
  • costs What this request cost, compute and transfer split out
  • error Set per page, so one bad URL does not fail the batch

More from the API.

Crawl API

Discover and fetch every page under a domain

Transform API

Markdown from HTML or PDF you already have

JSON scraping guide

Patterns for return_json_data and embedded state

Stop maintaining parsers.

Selector maps, embedded JSON, readability, and full browser rendering behind one POST request. Pay as you go from $1 per GB, no subscription.

Create your API key Read the scrape docs