Crawl a whole site with the bill capped up front.

POST one seed URL to the crawl API and Spider walks the domain. Every page streams back as clean markdown, HTML, or text the moment it completes. You set the page cap, the depth, and the spend ceiling before anything runs.

Start crawling Crawl API docs

The whole contract

POST https://api.spider.cloud/crawl
{
  "url": "https://example.com",
  "return_format": "markdown",
  "limit": 200,
  "depth": 3,
  "max_credits_allowed": 1000
}

limit never more than 200 pages
depth never more than 3 hops from the seed
max_credits_allowed never more than 1,000 credits, which is $0.10

Bandwidth: $1/GB
Compute: $0.0001/min
Failed page: $0
Credits per $1: 10,000

Depth sets the shape. You set the bill.

/crawl is the one endpoint where you hand Spider an open-ended job. Depth and limit bound the shape. The spend is a ceiling you authorize before the run starts, so there is nothing to guess.

Crawl shape by link depth, the parameter that bounds each row, and the spend ceiling you can authorize for it
Hops from seed	Pages	The cap that stops it	max_credits_allowed	Bill tops out at
0	1	`budget: {"*": 1}` seed page only	100	$0.01
1	8	`depth: 1` seed plus its links	500	$0.05
2	47	`depth: 2` most of a docs site	1,000	$0.10
3	200	`depth: 3` full discovery	2,500	$0.25
any	2,000	`limit: 2000` page cap wins	10,000	$1.00

Billing is two line items per page, bandwidth at $1 per GB of content before transformation and compute at $0.0001 per minute of CPU. Most pages land at a fraction of a cent, heavy ones cost more, and failed, blocked, or timed-out pages bill $0. Page counts here are from a typical documentation site.

The last two columns are not a forecast. max_credits_allowed is a ceiling you set in the same credits your balance holds, 10,000 to the dollar, so 1000 means the crawl cannot bill past $0.10 whatever the pages weigh. The costs object on every page response shows the exact charge as it happens.

You cannot accidentally crawl the internet.

Four independent caps bound every run. The crawler stops at whichever it hits first.

limit Page cap

A hard count. limit: 500 ends the crawl at page 500 no matter how many links remain. Set it first on any site you have not measured.

depth Hop ceiling

Distance from the seed URL. It defaults to 25, so a crawl with no caps set at all still has a ceiling.

budget Per-path counts

Page counts by route. {"*": 1} fetches only the root, and {"/docs/": 100} lets docs run to 100 pages while the wildcard holds everything else.

max_credits_allowed Spend ceiling

A credit cap for the whole run, in the same units your balance holds at 10,000 credits to the dollar. max_credits_per_page keeps any single page from eating it.

Everything draws from a prepaid balance, so a crawl can never spend money you have not loaded, and the balance never expires. Top-ups of $500 or more earn a bonus, 5% at $500 and 12% at $2,000.

Running the same large crawls every day? The Unlimited plan on the pricing page swaps credits for flat-rate concurrency from $40 a month.

The machinery you stop owning.

A production crawler is six systems pretending to be one. All six ship inside the endpoint.

Frontier and dedup

The crawler dedupes every discovered link against one queue before it fetches, so the same page never downloads or bills twice.

Politeness per host

Spider honors robots.txt by default, and delay and concurrency_limit slow it further for sites that need gentler pacing.

Browser pool

Smart mode fetches static pages over plain HTTP and sends only JS-rendered ones to a headless Chrome pool you never run.

Proxy rotation

Standard rotation is included in the price. Residential and ISP pools are there for targets that fight back.

Failures and re-runs

A page that fails comes back as an error entry billed at $0. The cache is on by default, so a re-crawl does not pay again for pages it already has.

Content pipeline

HTML becomes markdown with nav, ads, and boilerplate stripped, chunked for embeddings if you ask.

A bounded crawl, end to end.

The request sets three caps. The stream carries each page's exact charge, so the bill is the sum of what you can already see.

Run this

from spider import Spider

client = Spider()

# 200 pages, 3 hops, never more than $0.10
pages = client.crawl(
    "https://example.com",
    params={
        "return_format": "markdown",
        "limit": 200,
        "depth": 3,
        "max_credits_allowed": 1000,
    },
)

for page in pages:
    print(page["url"], page["costs"]["total_cost"])

curl -X POST https://api.spider.cloud/crawl \
  -H "Authorization: Bearer $SPIDER_API_KEY" \
  -H "Content-Type: application/jsonl" \
  -d '{
    "url": "https://example.com",
    "return_format": "markdown",
    "limit": 200,
    "depth": 3,
    "max_credits_allowed": 1000
  }'

import Spider from "@spider-cloud/spider-client";

const client = new Spider();

const pages = await client.crawl("https://example.com", {
  return_format: "markdown",
  limit: 200,
  depth: 3,
  max_credits_allowed: 1000,
});

const spent = pages.reduce((t, p) => t + p.costs.total_cost, 0);
console.log(pages.length, "pages for $" + spent.toFixed(4));

Streamed response application/jsonl

[
  {
    "url": "https://example.com/",
    "status": 200,
    "content": "# Example Domain\n\nThis domain is...",
    "error": null,
    "costs": {
      "compute_cost": 0.00001,
      "file_cost": 0.00002,
      "bytes_transferred_cost": 0.00002,
      "total_cost": 0.00004
    }
  },
  // 199 more pages, streamed as each completes
]

total_cost is in dollars. Sum it across the stream and that is the whole bill. Pages with an error contribute $0.

The full parameter reference covers proxies, caching, webhooks, and network filtering. For a crawl wired in as an agent tool, see agent data collection .

Output and reach.

Once the crawl is bounded, these decide what comes back and how far past the seed domain it reaches.

return_format

markdown, html, text, or bytes. Markdown is the one AI pipelines want.

request

http for speed, browser for JS-heavy sites, or smart to decide per page.

css_extraction_map

Apply CSS and XPath selectors to every page and get back only the matched fields.

chunking_alg

Split output by words, sentences, or lines to fit embedding context windows.

subdomains

Follow links onto subdomains like docs.example.com. tld does the same for sister domains.

external_domains

Treat listed domains as part of the same crawl. Exact matches and regex.

metadata

Titles, descriptions, and keywords per page. return_headers and return_cookies add the transport layer.

sitemap

Seed the queue from the sitemap, or crawl the sitemap alone with sitemap_only.

What it gets pointed at.

RAG knowledge bases

Docs sites and help centers crawled to markdown on a schedule, chunked and ready to embed.

Training corpora

Whole domains of text collected with one request per site instead of one scraper per site.

Site migrations

Every page with its metadata intact, so nothing goes missing on the way to a new CMS.

Competitive catalogs

A competitor's products, prices, and pages indexed across the whole domain.

More from the API.

Scrape API

One page at a time, same formats

Links API

URLs only, fewer credits per page

Crawling guide

The long-form technical docs

Cap it at a dime and let it run.

1,000 credits is a dime, and a crawl capped there cannot bill past it however heavy the pages run. Sign up, load a balance, and set your caps. Everything the stream sends back is yours to keep.

Start crawling Crawl API docs