Crawl a whole site with the bill capped up front.

One POST with a seed URL and Spider walks the domain, streaming every page back as clean markdown, HTML, or text the moment it completes. The page cap, the depth, and the spend ceiling are yours to set before anything runs.

Start crawling Crawl API docs

The whole contract

POST https://api.spider.cloud/crawl
{
  "url": "https://example.com",
  "return_format": "markdown",
  "limit": 200,
  "depth": 3,
  "max_credits_allowed": 1000
}

limit never more than 200 pages
depth never more than 3 hops from the seed
max_credits_allowed never more than 1,000 credits, which is $0.10

Bandwidth: $1/GB
Compute: $0.001/min
Failed page: $0
Credits per $1: 10,000

Depth sets the shape. You set the bill.

Crawling is the one endpoint where you hand over an open-ended job. Depth and limit bound the shape, and the spend is a ceiling you authorize up front, not a number you guess.

Crawl shape by link depth, the parameter that bounds each row, and the spend ceiling you can authorize for it
Hops from seed	Pages	The cap that stops it	max_credits_allowed	Bill tops out at
0	1	`budget: {"*": 1}` seed page only	100	$0.01
1	8	`depth: 1` seed plus its links	500	$0.05
2	47	`depth: 2` most of a docs site	1,000	$0.10
3	200	`depth: 3` full discovery	2,500	$0.25
any	2,000	`limit: 2000` page cap wins	10,000	$1.00

Billing is two line items per page, bandwidth at $1 per GB of content before transformation and compute at $0.001 per minute of CPU. Most pages land at a fraction of a cent, heavy ones cost more, and failed, blocked, or timed-out pages bill $0. Page counts here are from a typical documentation site.

The last two columns are not a forecast. max_credits_allowed is a ceiling you set in the same credits your balance holds, 10,000 to the dollar, so 1000 means the crawl cannot bill past $0.10 whatever the pages weigh. The costs object on every page response shows the exact charge as it happens.

You cannot accidentally crawl the internet.

Four independent caps bound every run. The crawler stops at whichever it hits first.

limit Page cap

A hard count. limit: 500 ends the crawl at page 500 no matter how many links remain. Set it first on any site you have not measured.

depth Hop ceiling

Distance from the seed URL. It defaults to 25, so a crawl with no caps set at all still has a ceiling.

budget Per-path counts

Page counts by route. {"*": 1} fetches only the root, and {"/docs/": 100} lets docs run to 100 pages while the wildcard holds everything else.

max_credits_allowed Spend ceiling

A credit cap for the whole run, in the same units your balance holds at 10,000 credits to the dollar. max_credits_per_page keeps any single page from eating it.

Everything draws from a prepaid balance, so a crawl can never spend money you have not loaded, and the balance never expires. Top-ups of $500 or more earn a bonus, 5% at $500 and 12% at $2,000.

Running the same large crawls every day? The Unlimited plan swaps per-page credits for flat-rate concurrency from $40 a month.

The machinery you stop owning.

A production crawler is five or six systems pretending to be one. Every one of them ships inside the endpoint.

Frontier and dedup

Every discovered link joins one queue and is deduped before fetch, so the same page never downloads or bills twice.

Politeness per host

robots.txt is honored by default, with delay and concurrency_limit for sites that need gentler pacing.

Browser pool

Smart mode fetches static pages over plain HTTP and sends only JS-rendered ones to a headless Chrome pool you never run.

Proxy rotation

Standard rotation is included in the price. Residential and ISP pools are there for targets that fight back.

Retries and re-runs

Failures come back as error entries billed at $0, and the cache is on by default, so a re-crawl does not pay again for pages it already has.

Content pipeline

HTML becomes markdown with nav, ads, and boilerplate stripped, chunked for embeddings if you ask.

A bounded crawl, end to end.

The request sets three caps. The stream carries each page's exact charge, so the bill is the sum of what you can already see.

from spider import Spider

client = Spider()

# 200 pages, 3 hops, never more than $0.10
pages = client.crawl(
    "https://example.com",
    params={
        "return_format": "markdown",
        "limit": 200,
        "depth": 3,
        "max_credits_allowed": 1000,
    },
)

for page in pages:
    print(page["url"], page["costs"]["total_cost"])

curl -X POST https://api.spider.cloud/crawl \
  -H "Authorization: Bearer $SPIDER_API_KEY" \
  -H "Content-Type: application/jsonl" \
  -d '{
    "url": "https://example.com",
    "return_format": "markdown",
    "limit": 200,
    "depth": 3,
    "max_credits_allowed": 1000
  }'

import Spider from "@spider-cloud/spider-client";

const client = new Spider();

const pages = await client.crawl("https://example.com", {
  return_format: "markdown",
  limit: 200,
  depth: 3,
  max_credits_allowed: 1000,
});

const spent = pages.reduce((t, p) => t + p.costs.total_cost, 0);
console.log(pages.length, "pages for $" + spent.toFixed(4));

Streamed response application/jsonl

[
  {
    "url": "https://example.com/",
    "status": 200,
    "content": "# Example Domain\n\nThis domain is...",
    "error": null,
    "costs": {
      "compute_cost": 0.00001,
      "file_cost": 0.00002,
      "bytes_transferred_cost": 0.00002,
      "total_cost": 0.00004
    }
  },
  // 199 more pages, streamed as each completes
]

total_cost is in dollars. Sum it across the stream and that is the whole bill. Pages with an error contribute $0.

The full parameter reference covers proxies, caching, webhooks, and network filtering.

Output and reach.

Once the crawl is bounded, these decide what comes back and how far the frontier extends.

return_format

markdown, html, text, or bytes. Markdown is the one AI pipelines want.

request

http for speed, browser for JS-heavy sites, or smart to decide per page.

css_extraction_map

CSS and XPath selectors applied to every page, returning only the matched data.

chunking_alg

Split output by words, sentences, or lines to fit embedding context windows.

subdomains

Follow links onto subdomains like docs.example.com. tld does the same for sister domains.

external_domains

Treat listed domains as part of the same crawl. Exact matches and regex.

metadata

Titles, descriptions, and keywords per page. return_headers and return_cookies add the transport layer.

sitemap

Seed the queue from the sitemap, or crawl the sitemap alone with sitemap_only.

What it gets pointed at.

RAG knowledge bases

Docs sites and help centers crawled to markdown on a schedule, chunked and ready to embed.

Training corpora

Whole domains of text collected with one request per site instead of one scraper per site.

Site migrations

Every page with metadata intact, so nothing is lost moving to a new CMS.

Competitive catalogs

A competitor's products, prices, and pages indexed across the whole domain.

More from the API.

Scrape API

One page at a time, same formats

Links API

URLs only, fewer credits per page

Crawling guide

The long-form technical docs

Cap it at a dime and let it run.

Cap a crawl at 1,000 credits and it cannot bill past a dime, however heavy the pages run. Sign up, load a balance, set your caps, and keep everything the stream sends back.

Start crawling Crawl API docs