# Web crawling API: one URL in, a site's pages out | Spider

Spider's crawl API follows a site's links from one URL and returns each page as markdown, HTML or text. From $0.16 per 1,000 pages; cost varies by site.

Canonical: https://spider.cloud/api/crawl/

## Web crawling API. One URL in, the site back.

Send one URL and Spider follows the site's links and returns each page as markdown, HTML or text, up to the caps you set.

Request

```
curl https://api.spider.cloud/crawl \  -H "Authorization: Bearer $SPIDER_API_KEY" \  -H "Content-Type: application/json" \  -d '{    "url": "https://books.toscrape.com/",    "limit": 25,    "depth": 2,    "return_format": "markdown",    "request": "http",    "return_page_links": true  }'
```

Response 200, one object per page

1. 200 / 6,250 bytes
2. 200 /index.html 6,250 bytes
3. 200 /catalogue/page-2.html 6,162 bytes
4. 200 /catalogue/page-3.html 6,562 bytes
5. 200 /catalogue/page-1.html 5,903 bytes

Recorded November 19, 2026. These are the first 3 of the 25 pages, in the order they came back. Recorded November 19, 2026. These are the first 5 of the 25 pages, in the order they came back.

In [our run of November 16, 2026](https://spider.cloud/benchmarks/#crawl-vs-scrape-2026-11-16), one crawl call returned the 74 pages of a documentation site in 5.1 s.

From $0.16 per 1,000 pages. Cost varies by site. [Unlimited](https://spider.cloud/pricing/#unlimited) is $40 a month for 2 requests at once in http mode.

[Get an API key](https://spider.cloud/register/)

POST /crawl 2026-11-19

- depth 2

  25 pages 25 of 25 slots filled

* wall time

  5.52 s

* total cost

  $ 0.0039

- depth 1

  2 pages 2 of 25 slots filled

* total cost

  $ 0.0003

1. / 6,250 bytes of markdown
2. /index.html 6,250 bytes of markdown
3. /catalogue/page-2.html 6,162 bytes of markdown
4. /catalogue/page-3.html 6,562 bytes of markdown
5. /catalogue/page-1.html 5,903 bytes of markdown
6. /catalogue/page-4.html 7,415 bytes of markdown
7. /catalogue/page-5.html 6,905 bytes of markdown
8. /catalogue/page-6.html 6,721 bytes of markdown
9. /catalogue/page-7.html 6,395 bytes of markdown
10. /catalogue/page-8.html 6,377 bytes of markdown
11. /catalogue/page-9.html 6,120 bytes of markdown
12. /catalogue/page-14.html 6,079 bytes of markdown
13. /catalogue/page-15.html 6,456 bytes of markdown
14. /catalogue/page-10.html 6,374 bytes of markdown
15. /catalogue/page-12.html 5,960 bytes of markdown
16. /catalogue/page-11.html 5,914 bytes of markdown
17. /catalogue/page-16.html 6,119 bytes of markdown
18. /catalogue/page-17.html 6,121 bytes of markdown
19. /catalogue/page-13.html 6,302 bytes of markdown
20. /catalogue/page-18.html 6,166 bytes of markdown
21. /catalogue/page-19.html 6,370 bytes of markdown
22. /catalogue/page-20.html 6,351 bytes of markdown
23. /catalogue/page-21.html 5,751 bytes of markdown
24. /catalogue/page-22.html 6,015 bytes of markdown
25. /catalogue/page-23.html 6,690 bytes of markdown

and 17 more pages, listed on a wider screen

We sent one request to books.toscrape.com, a public sandbox, twice on November 19, 2026, 43 seconds apart. The second call changed depth from 2 to 1 and kept every other field.

## The request that returned 25 pages

Six fields went over the wire. Two of them cap the crawl. The other four say where it starts and what comes back for each page.

[Run this](https://spider.cloud/playground/?mode=crawl\&url=https%3A%2F%2Fbooks.toscrape.com%2F\&format=markdown\&p.limit=25\&p.depth=2\&src=api_crawl_run)

```
curl https://api.spider.cloud/crawl \  -H "Authorization: Bearer $SPIDER_API_KEY" \  -H "Content-Type: application/json" \  -d '{    "url": "https://books.toscrape.com/",    "limit": 25,    "depth": 2,    "return_format": "markdown",    "request": "http",    "return_page_links": true  }'
```

```
from spider import Spider client = Spider() pages = client.crawl_url(    "https://books.toscrape.com/",    params={        "limit": 25,        "depth": 2,        "return_format": "markdown",        "request": "http",        "return_page_links": True,    },) for page in pages:    print(page["status"], page["url"])
```

```
import {  Spider,} from "@spider-cloud/spider-client"; const client = new Spider(); const pages = await client.crawlUrl(  "https://books.toscrape.com/",  {    limit: 25,    depth: 2,    return_format: "markdown",    request: "http",    return_page_links: true,  },); for (const page of pages) {  console.log(page.status, page.url);}
```

- url

  Where the crawl starts. Spider follows links from it.

- limit

  The most pages the crawl may return, 25 here.

- depth

  How deep the crawl may go. 2 here, 1 in the second run.

- return_format

  The form of each page's content, markdown here.

- request

  How Spider fetches a page. http is a plain fetch without a browser.

- return_page_links

  Adds a links array to each page, 73 or 74 hrefs here.

## Which caps should a crawl set?

Set limit first on any site you have not crawled before. In the recorded pair, limit stopped one run and depth stopped the other.

caps

- `limit`

  The most pages one crawl may return. Both runs sent 25, and the depth 2 run ended on it with 25 pages.

- `depth`

  Depth 1 ended the other run at 2 pages, 23 short of its limit. The default is 25 and 0 removes the cap, so try a value on a small limit before you rely on it.

Four more caps from the reference. Neither recorded run sent them.

- `budget`

  A page count for each path you name, to cap one part of a site apart from the rest.

- `max_credits_allowed`

  The most credits the whole run may use.

- `max_credits_per_page`

  The most credits any one page may use.

- `crawl_timeout`

  The longest the crawl may run, sent as an object of secs and nanos.

[Caps in the crawl reference](https://spider.cloud/docs/api/crawl/)

## What one crawl returns

The response is one JSON array with an object for each page fetched, in the order the pages came back.

one page object page 3 of 25

- `url`

  https\://books.toscrape.com /catalogue/page-2.html

  The address of the page. In both recorded runs the seed came first.

- `status`

  200

  The HTTP status of the fetch. All 25 were 200.

- `content`

  1. All products | Books to Scrape - Sandbox
  2. \* \[Home]\(../index.html)
  3. \* All products
  4. \*\*1000\*\* results - showing \*\*21\*\* to \*\*40\*\*.
  5. \### \[In Her Wake]\(in-her-wake_980/index.html)
  6. £12.84
  7. In stock

  7 of 169 lines

  The page in the format you asked for. Here that was markdown, 157,728 bytes across the 25 pages.

- `links`

  74 hrefs

  The hrefs found on the page, present because the request sent return_page_links. Together the 25 pages named 537 distinct URLs.

- `costs`

  - compute_cost

    0.00000041

  - bytes_transferred_cost

    0.00000073

  - file_cost

    0.00013660

  - transform_cost

    0.00001790

  - total_cost

    0.00015565

  What the page cost in dollars, split into compute_cost, bytes_transferred_cost, file_cost and transform_cost, with their sum in total_cost.

- `error`

  Holds the failure for that page when a fetch goes wrong.

This request named one URL and 25 pages came back, 24 of them found by following links. When you already have the URLs, [Scrape](https://spider.cloud/api/scrape/) fetches the ones you name.

The recorded run sent Content-Type: application/json and the pages arrived as one array. Send application/jsonl and the API answers with one JSON line per page as each one finishes, per the reference.

[All crawl parameters](https://spider.cloud/docs/api/crawl/)

## What a crawl costs

From $0.16 per 1,000 pages on light static pages. Cost varies by site, and with a crawl it also depends on how many pages the caps let through.

- Light static pages

  $0.16 per 1,000 pages

- Documentation and articles

  $0.27 to $0.33 per 1,000 pages

- Store product pages and reference articles

  $1.38 to $1.48 per 1,000 pages

Measured by kind of page for the pricing guide, most recently on 2026-11-17. [Pricing guide](https://spider.cloud/guides/pricing-and-plans/)

| costs key              | 25 pages, USD |
| ---------------------- | ------------- |
| compute_cost           | 0.00001332    |
| bytes_transferred_cost | 0.00001815    |
| file_cost              | 0.00342876    |
| transform_cost         | 0.00044750    |
| total_cost             | 0.00390773    |

Recorded November 19, 2026 against books.toscrape.com. US dollars, each key summed over the pages of the run.

Pay as you go from $1 per GB of bandwidth plus $0.0001 per minute a request runs. No subscription. Top up from $5.

The receipt adds up the costs object on each of the 25 recorded pages. They came to $0.0039, about $0.16 per 1,000 on light static pages. $0.0034 of that was file_cost, the page bodies as fetched, so a site with heavier pages costs more.

Every page reports its own cost in costs.total_cost. Crawl 25 pages of your own site and the response tells you what the next thousand will cost.

If you crawl on a schedule, Unlimited is $40 a month for 2 requests at once in http mode.

## Common questions

How much does the web crawling API cost?

From $0.16 per 1,000 pages on light static pages. Cost varies by site. In our recorded run of November 19, 2026, a crawl of 25 pages of books.toscrape.com cost $0.0039 in total, about $0.16 per 1,000 pages. Each page in a response carries its own cost in costs.total_cost.

How do I keep a crawl from running too long or costing too much?

Set limit for the most pages the crawl may return and depth for how deep it may go. max_credits_allowed sets the most credits the run may use, and crawl_timeout sets the longest it may run. In our recorded run of November 19, 2026, the same request returned 25 pages at depth 2 and 2 pages at depth 1.

What does the depth parameter do in a crawl?

It caps how deep a crawl goes from the URL you send. In our recorded run of November 19, 2026 against books.toscrape.com, one request returned 25 pages at depth 2 and 2 pages at depth 1 with every other field the same. The default is 25 and 0 removes the cap. Try a value on a small limit before you rely on it, and see the crawl reference at spider.cloud/docs/api/crawl.

What is the difference between crawl and scrape?

Crawl takes one URL and follows links to find the rest. Scrape fetches the URLs you name. Both return the same response shape. In our recorded run of November 19, 2026, a crawl request that named one URL returned 25 pages.

Does a page that fails during a crawl still bill?

An attempt bills for the bandwidth and compute it used, whatever status the page returns. A failed page is not free. A blocked page usually costs a fraction of a cent.

Does a crawl follow robots.txt?

Yes, by default. respect_robots is true unless you change it, so a crawl skips the paths a site disallows.

### Keep reading

[Links API](https://spider.cloud/api/links/)

[The links on each page without the content, for choosing URLs before you crawl.](https://spider.cloud/api/links/)

[Scrape API](https://spider.cloud/api/scrape/)

[The same response for URLs you already have.](https://spider.cloud/api/scrape/)

[Crawling and scraping guide](https://spider.cloud/web-crawling-and-scraping/)

[One crawl call measured against a scrape call per URL.](https://spider.cloud/web-crawling-and-scraping/)

## Start with 25 pages

Send the request on this page at your own site and the response shows each page and what it cost. From $0.16 per 1,000 pages on light static pages. Cost varies by site.

[Get an API key](https://spider.cloud/register/)
