A web scraper you can run before you sign up.
Paste a URL and press Run. Spider fetches the page and returns it as markdown, with the time and the cost of that request beside it.
A sample, until you press Run: example.com, sent from this page on 2026-11-16. Your run replaces it, and the readings beside it.
- Elapsed
- 1.287 s
- Cost
- $0.0000223
- Status and size
- 200
- 171 B
Keyless access is /scrape only. It is rate limited. On a 429, honor Retry-After. Get a key for anything sustained.
[
{
"content": "Example Domain\nThis domain is for use in documentation examples without needing permission. This is not a service; avoid relying on it for testing and monitoring purposes.",
"costs": {
"ai_cost": 0,
"ai_cost_formatted": "0",
"bytes_transferred_cost": 4.14984375e-7,
"bytes_transferred_cost_formatted": "0.0000004149843749999999903398",
"compute_cost": 0.00000228534375,
"compute_cost_formatted": "0.0000022853437500000001602884",
"file_cost": 8.114062500000002e-7,
"file_cost_formatted": "0.0000008114062500000001156887",
"total_cost": 0.000022261734375000002,
"total_cost_formatted": "0.0000222617343750000013071510",
"transform_cost": 0.000018750000000000002,
"transform_cost_formatted": "0.0000187500000000000010408341"
},
"duration_elasped_ms": 716,
"error": null,
"metadata": {
"automation_data": null,
"crawl_id": null,
"description": "",
"domain": "example.com",
"extracted_data": null,
"file_size": 171,
"final_url": null,
"keywords": null,
"og_image": null,
"original_url": "https://example.com/",
"pathname": "/",
"resource_type": ".md",
"title": "Example Domain",
"url": null,
"user_id": "00000000-0000-0000-0000-000000000000"
},
"status": 200,
"url": "https://example.com/"
}
]The request the tool sent, from your code.
The curl works as written, with no key. The Python and Node clients read SPIDER_API_KEY from the environment. The three send the same url, return_format and metadata.
The response is a JSON array with one object for each page. content holds the page in the format you asked for. status is what the site answered with. costs.total_cost is the price of the request in dollars, to more decimal places than you will want.
The time field is spelled duration_elasped_ms in the API, so spell it that way in your code. Every parameter is in the Scrape API reference.
curl -X POST \
https://api.spider.cloud/scrape \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com",
"return_format": "markdown",
"metadata": true
}'Ten pages came back 15 times smaller as markdown.
On 2026-11-05 we fetched ten public pages as raw HTML and as markdown and counted the tokens in each. Markdown was 15x smaller at the median. The range was wide. The Hacker News front page is already mostly text and shrank 2.9x. A Kubernetes documentation page, a short chapter inside 480 KB of navigation, scripts and inline SVG, shrank 47x. The ten pages, the tokenizer and the curl are in What a web page costs in tokens.
A second run on 2026-11-06 took 60 documentation pages from one site. The HTML was 7,428,798 tokens and the markdown was 202,123, about 37 times smaller. That run is in Build a fine-tuning dataset from the web in two API calls.
The smart default cost the same as plain HTTP on 100 pages.
Some pages send their content in the first HTML response. Others send a shell and build the page with JavaScript, and a plain fetch of those returns almost nothing. With a key, the request field decides how Spider fetches. We counted both kinds on 39 pages in the guide to what each kind of fetch returns.
On 2026-11-06 we sent four sets of 25 pages in http mode and again in smart, the default. The cost per 1,000 pages matched on all four sets, within a cent. That run compared those two modes and no other. The sets and the figures are in Web scraping cost per 1,000 pages, by kind of page.
For the browser itself, on 2026-09-18 we ran a third party's 80-task fidelity bench with its harness unchanged and passed 80 of 80. That is one day's reading, and the sites it tests change on their own schedule. The repository and the command are in Spider Browser passes 80/80 on Browser Fidelity Bench.
A documentation page cost a fifth of a product page.
Pay as you go has two meters: $1 per GB of bandwidth and $0.0001 per minute of compute. Spider bills a page by what it weighs, so the kind of page sets the price. On 2026-11-06 we scraped 25 pages of each kind in smart mode with markdown output. It was the first day of a seven-day run, and the rule that picked each set is in Web scraping cost per 1,000 pages, by kind of page.
An attempt bills only the bandwidth and compute it used. A blocked page usually costs a fraction of a cent. Pay as you go starts at 10,000 requests per minute and rises through four levels to 80,000 as the account uses more.
Unlimited is the flat option. You buy concurrency, from 2 requests in flight for $40 a month, and run unlimited requests through it. Both plans are on the pricing page.
| Kind of page | Pay as you go |
|---|---|
| Documentation | $0.27 |
| Articles | $0.33 |
| Store product pages | $1.38 |
Jeff Mendez ran the measurements on this page between 2026-09-18 and 2026-11-06. Apart from the sample in the tool, every number comes from one of the four posts linked above, and each post holds its method.
Questions before the first request.
Can I use this web scraper without an API key?
Yes. Keyless access is /scrape only. It is rate limited. On a 429, honor Retry-After. Get a key for anything sustained. Without a key, every route other than /scrape returns 402, except /unlimited/*, which returns 401. The tool at the top of this page sends that keyless request.
Does the web scraper return JSON?
Every response is JSON. It is an array with one object per page, and each object carries the content, the status, the final URL and the cost of that request. The JSON view in the tool shows the whole body, indented. With a key, the same endpoint takes css_extraction_map, which returns named fields picked by CSS selector under css_extracted, and return_json_data, which returns the JSON a site embeds in its script tags. Extraction with a schema or a prompt also needs a key.
Why did a page come back empty?
The usual cause is a page that builds its content with JavaScript. A plain HTTP fetch of that page returns the shell and little else. With a key, the request field defaults to smart, which tries HTTP first and moves to a browser when the page needs one. Setting it to browser renders every page. If the site refused the request, the status field carries the site's own answer, such as 403 or 429. A blocked page usually costs a fraction of a cent.
Does it follow robots.txt?
Yes, by default. The respect_robots parameter is true unless you change it. The cost run on this page left it at the default for all of its pages.
What if I do not have the URLs yet?
Two other endpoints find the pages, and both need a key. /search takes a query and can return each result with its page content. On 2026-11-06 one search call returned ten results as markdown in 4.5 seconds for $0.0041. /crawl takes a starting URL and follows its links. On the same day one crawl call returned 60 documentation pages as markdown in 6.5 seconds.
How many tokens does a scraped page take in a model's context?
It depends on the page and the format. In our run of ten public pages on 2026-11-05, the median page was 72,093 tokens as raw HTML and 4,384 as markdown, counted with the o200k_base encoding. Markdown keeps the links and headings, which an agent uses to decide what to fetch next. Plain text was smaller than markdown on 8 of the 10 pages.
How many requests can I send?
Pay as you go starts at 10,000 requests per minute and rises through four levels to 80,000 as the account uses more. Each response carries your current limit in the RateLimit-Limit header. Past the limit the API answers 429. Wait the number of seconds in Retry-After, then send the request again.
A key is for anything sustained.
Pay as you go at $1 per GB of bandwidth plus $0.0001 per CPU minute. No subscription. Top up from $5.