# Web scraping API: URL in, markdown or fields out | Spider

Spider's scraper API turns a URL into markdown, HTML or text, plus the fields you name. From $0.16 per 1,000 pages; cost varies by site. Try it with no key.

Canonical: https://spider.cloud/api/scrape/

## Web scraping API. Page in, fields out.

Your agent sends a URL and gets the page back as markdown, HTML or text, plus any fields you name.

Request

```
curl https://api.spider.cloud/scrape \  -H "Content-Type: application/json" \  -d '{"url": "https://books.toscrape.com/catalogue/its-only-the-himalayas_981/index.html", "return_format": "markdown"}'
```

Response 200, content

1. It's Only the Himalayas | Books to Scrape - Sandbox
2. \* \[Home]\(../../index.html)
3. \* \[Books]\(../category/books_1/index.html)
4. \* \[Travel]\(../category/books/travel_2/index.html)
5. \* It's Only the Himalayas
6. \# It's Only the Himalayas
7. £45.17
8. In stock (19 available)

This request runs as written, with no key. The response is the one recorded November 17, 2026 against books.toscrape.com, a public sandbox site.

In [our run of 1,000 URLs](https://spider.cloud/benchmarks/#api-1000-urls-2026-02-11) on February 11, 2026, the API returned content for 999 of them and for 249 of the 250 protected ones.

From $0.16 per 1,000 pages. Cost varies by site. [Unlimited](https://spider.cloud/pricing/#unlimited) is $40 a month for 2 requests at once in http mode.

[Get an API key](https://spider.cloud/register/)

Keyless access is /scrape only. It is rate limited. On a 429, honor Retry-After. Get a key for anything sustained.

POST /scrape 2026-11-17

return_format, with the characters of content each returned

raw 19,320 markdown 3,988 text 2,256

1. \<div id="product_gallery" class="carousel">
2. \<div class="thumbnail">
3. \<div class="carousel-inner">
4. \<div class="item active">
5. \<img src="../../media/cache/6d/41/6d418a73cc7d4ecfd75ca11d854041db.jpg" alt="It\&#39;s Only the Himalayas" />
6. \</div>
7. \</div>
8. \</div>
9. \</div>
10. \</div>
11. \<div class="col-sm-6 product_main">
12. \<h1>It\&#39;s Only the Himalayas\</h1> returned as title
13. \<p class="price_color">£45.17\</p> returned as price
14. \<p class="instock availability">
15. \<i class="icon-ok">\</i>
16. In stock (19 available) returned as stock
17. \</p>
18. ...

content, HTML as fetched lines 132 to 175 of 820, no blank lines lines 153 to 175 of 820, no blank lines

1. It's Only the Himalayas | Books to Scrape - Sandbox
2. \* \[Home]\(../../index.html)
3. \* \[Books]\(../category/books_1/index.html)
4. \* \[Travel]\(../category/books/travel_2/index.html)
5. \* It's Only the Himalayas
6. !\[It\&#39;s Only the Himalayas]\(../../media/cache/6d/41/6d418a73cc7d4ecfd75ca11d854041db.jpg)
7. \# It's Only the Himalayas returned as title
8. £45.17 returned as price
9. \*\*
10. In stock (19 available) returned as stock
11. \*\*
12. ...

content, markdown lines 1 to 11 of 75

1. It\&#39;s Only the Himalayas | Books to Scrape - Sandbox
2. Home Books Travel It\&#39;s Only the Himalayas It\&#39;s Only the Himalayas £45.17 In stock (19 available) returned as title, price, stock
3. \&nbsp;
4. Warning! This is a demo website for web scraping purposes. Prices and ratings here were randomly assigned and have no real meaning. Product Description ...
5. Add to basket Mesaerion: The Best Science ... £37.59 In stock
6. Add to basket Olio £23.88 In stock
7. Add to basket Our Band Could Be ... £57.25 In stock
8. Add to basket Rip it Up and ... £35.02 In stock
9. Add to basket Scott Pilgrim\&#39;s Precious Little ... £52.29 In stock
10. Add to basket

content, plain text all 10 lines, line 4 cut short

`css_extracted` 3 fields from 3 selectors

- title .product_main h1

  It's Only the Himalayas

- price .product_main .price_color

  £45.17

- stock .product_main .availability

  In stock (19 available)

* characters of content

  19,320 return_format: raw

* wall time

  1.486 s HTTP 200, from a laptop

* response size

  22,817 bytes the whole JSON body

costs, USD [Rates](https://spider.cloud/pricing/)

- compute_cost

  0.00000247

- bytes_transferred_cost

  0.00000080

- file_cost

  0.00005199

- transform_cost

  0.00000000

- total_cost

  0.00005525

* characters of content

  3,988 return_format: markdown

* wall time

  1.217 s HTTP 200, from a laptop

* response size

  6,279 bytes the whole JSON body

costs, USD [Rates](https://spider.cloud/pricing/)

- compute_cost

  0.00000144

- bytes_transferred_cost

  0.00000080

- file_cost

  0.00005199

- transform_cost

  0.00001790

- total_cost

  0.00007213

* characters of content

  2,256 return_format: text

* wall time

  1.074 s HTTP 200, from a laptop

* response size

  4,520 bytes the whole JSON body

costs, USD [Rates](https://spider.cloud/pricing/)

- compute_cost

  0.00000170

- bytes_transferred_cost

  0.00000080

- file_cost

  0.00005199

- transform_cost

  0.00001790

- total_cost

  0.00007239

Recorded November 17, 2026 against books.toscrape.com, a public sandbox site. One URL and one selector map, sent 3 times with a different return_format.

## Named fields from CSS selectors

Name a field, point it at a selector, and the response comes back already parsed.

- "/"

  A map key is a URL path, and this one runs its fields on every page.

- "/any/path"

  Any other key runs only on the page whose path is exactly that key, so a prefix matches nothing.

- selectors

  A field takes CSS or XPath and more than one selector, so one map can outlast a template change.

[Try a page with no key](https://spider.cloud/web-scraper/)

[Run this](https://spider.cloud/playground/?mode=scrape\&url=https%3A%2F%2Fbooks.toscrape.com%2Fcatalogue%2Fits-only-the-himalayas_981%2Findex.html\&format=markdown\&src=api_scrape_run)

```
curl https://api.spider.cloud/scrape \  -H "Authorization: Bearer $SPIDER_API_KEY" \  -H "Content-Type: application/json" \  -d '{    "url": "https://books.toscrape.com/catalogue/its-only-the-himalayas_981/index.html",    "return_format": "markdown",    "css_extraction_map": {      "/": [        { "name": "title", "selectors": [".product_main h1"] },        { "name": "price", "selectors": [".product_main .price_color"] },        { "name": "stock", "selectors": [".product_main .availability"] }      ]    }  }'
```

```
from spider import Spider client = Spider() url = (    "https://books.toscrape.com/catalogue/"    "its-only-the-himalayas_981/index.html")fields = {    "title": ".product_main h1",    "price": ".product_main .price_color",    "stock": ".product_main .availability",}selector_map = [    {"name": name, "selectors": [css]}    for name, css in fields.items()] pages = client.scrape_url(    url,    params={        "return_format": "markdown",        "css_extraction_map": {            "/": selector_map,        },    },) print(pages[0]["css_extracted"])
```

```
import {  Spider,} from "@spider-cloud/spider-client"; const client = new Spider(); const url =  "https://books.toscrape.com/catalogue/" +  "its-only-the-himalayas_981/index.html";const fields = {  title: ".product_main h1",  price: ".product_main .price_color",  stock: ".product_main .availability",}; const pages = await client.scrapeUrl(url, {  return_format: "markdown",  css_extraction_map: {    "/": Object.entries(fields).map(      ([name, css]) => ({        name,        selectors: [css],      }),    ),  },}); console.log(pages[0].css_extracted);
```

## Does the page need a browser?

The request parameter sets how Spider fetches a page, which decides most of what the request costs and how long it takes.

"request"

- HTTP fetch Chrome: when detected

  ### `smart` default

  Most pages, and any batch you are not sure about.

  Spider inspects each page and picks HTTP or Chrome per URL, so you pay for rendering only on the pages that need it.

- HTTP fetch Chrome: no

  ### `http`

  You know the page renders on the server.

  A plain fetch with no browser, for docs, feeds and any page that already works with curl.

- HTTP fetch Chrome: always

  ### `browser`

  The request sends scripts, waits or scrolling.

  Chrome renders every page. automation_scripts, evaluate_on_new_document, wait_for and scroll run only here, and a smart request can finish on HTTP and skip them.

[Web scraping guide](https://spider.cloud/web-scraping/)

## What one call returns

The response is an array with one object per URL, and each option adds a field to that object.

Every object also carries url, status, costs and error, and error is set per page so one bad URL does not fail the batch.

Scrape fetches the URLs you name and never follows a link, and [Crawl](https://spider.cloud/api/crawl/) finds those URLs for you with the same parameters and response shape.

[Every parameter in the scrape reference](https://spider.cloud/docs/api/scrape/)

send get back

- `return_format` returns `content`

  The page as markdown, raw HTML, text or bytes.

- `css_extraction_map` returns `css_extracted`

  Named fields from your selector map.

- `return_json_data` returns `json_data`

  JSON-LD, product schema and the state objects a framework ships inside the page.

- `readability` returns `content`

  The main body with navigation, sidebars and ads removed.

- `evaluate_on_new_document`

  Your script runs before extraction, on a browser request.

- `url`

  Comma-separate URLs or send an array of objects, and the whole list comes back as one array.

- `metadata` returns `metadata`

  Title, description and keywords.

- `return_page_links` returns `links`

  Every link found on the page.

- `return_headers` returns `headers`

  The response headers.

- `return_cookies` returns `cookies`

  The cookies the page set.

## What the scraper API costs

From $0.16 per 1,000 pages. Cost varies by site, because it follows page weight.

Pay as you go from $1 per GB of bandwidth plus $0.0001 per minute a request runs. No subscription. Top up from $5.

Each row is a kind of page we measured, not a plan you choose. Every response reports its own cost in costs.total_cost, so a sample of your own pages shows what yours will cost.

For steady volume, Unlimited is a flat $40 a month for 2 requests at once.

- Light static pages

  - $0.16 per 1,000 pages
  - Measured November 17, 2026 on 25 catalogue pages of books.toscrape.com, the site the request above fetches.

- Documentation and articles

  - $0.27 to $0.33 per 1,000 pages
  - Measured November 6, 2026 on 25 pages of each kind.

- Store product pages and reference articles

  - $1.38 to $1.48 per 1,000 pages
  - Measured in the same November 6 run.

## Common questions

How much does the web scraping API cost?

From $0.16 per 1,000 pages on light static pages. Cost varies by site. The heaviest pages we measured, store product pages and reference articles, ran $1.38 to $1.48 per 1,000. The table above has each kind of page.

Do failed requests cost anything?

An attempt bills for the bandwidth and compute it used, whatever status the page returns. A failed page is not free. A blocked or empty page pays the 0.01 credit page minimum plus the compute it used, usually a fraction of a cent. Spider does not bill bandwidth for bytes it never received.

Can I try the scraper API without an API key?

Yes. Send the request at the top of this page with no Authorization header. Only /scrape answers without a key, and it is rate limited, so get a key before you run anything steady.

Does the scraper API render JavaScript?

Only when the page needs it. The default mode, smart, sends a plain HTTP request first and opens a browser only when that is not enough. Set request to http to never render, or to browser to render every page.

What formats does it return?

The page as markdown, raw HTML, text or bytes, set with return_format. Add a css_extraction_map and each result also carries css_extracted, with the fields you named.

What are the rate limits?

Pay as you go starts at 10,000 requests per minute and rises through four levels to 80,000 as the account uses more. Past the limit the API answers 429. Wait the number of seconds in Retry-After, then send the request again.

### Keep reading

[Transform API](https://spider.cloud/api/transform/)

[Markdown from HTML or PDF you already have.](https://spider.cloud/api/transform/)

[JSON scraping guide](https://spider.cloud/docs/advanced/json-scraping/)

[Patterns for return_json_data and embedded state.](https://spider.cloud/docs/advanced/json-scraping/)

[Price monitoring](https://spider.cloud/use-cases/price-monitoring/)

[A price and stock map like the one above, run on a schedule.](https://spider.cloud/use-cases/price-monitoring/)

## Stop maintaining parsers

Pay as you go from $0.16 per 1,000 pages, with no subscription. Cost varies by site.

[Get an API key](https://spider.cloud/register/)
