Skip to main content

Web scraping benchmarks, each with its date and its method.

These are the 11 runs we have published, from 2026-02-11 to 2026-11-16. Each one says what we sent, how many pages it covered and how we counted the result.

A run goes on this page after it is published somewhere else first, and each run links to that post or page. A number here that its source does not print is a bug. Every run has its own anchor, so you can link to the one you cite.

Eleven runs, newest first.

The runs.

· Fetching

A plain fetch against a rendered one

29 documentation, news and store product pages, plus 10 pages picked by hand

pages where the plain fetch returned at least 90% of the browser's words
26 of 29
pages where the plain fetch returned at least 90% of the browser's words
median cost per 1,000 requests, plain and rendered
$0.36 and $1.40
median cost per 1,000 requests, plain and rendered
median wall time at the client, plain and rendered
1.5 s and 10.0 s
median wall time at the client, plain and rendered
hand-picked pages where the plain fetch returned under half the words
8 of 10
hand-picked pages where the plain fetch returned under half the words

Run of 2026-11-16.

Each page went to /scrape once as a plain HTTP fetch and once rendered in a browser, with markdown output and four requests in flight. Words are the markdown split on whitespace, and a page counts when both requests answered 200. Cost is the costs.total_cost field of the response. The 10 extra pages were chosen because we expected them to need a browser, so they say nothing about how common such pages are.

Source: How web scraping works: what a plain fetch gets and misses.

· Crawling

One crawl call against a scrape call per URL

The 74 pages of one documentation site, the first of ten sites on a fixed list to qualify

one crawl call, 74 pages, wall time at the client
5.1 s
one crawl call, 74 pages, wall time at the client
one scrape call per URL, six in flight
36.0 s
one scrape call per URL, six in flight
pages with the same markdown from both, 701,514 characters
74 of 74
pages with the same markdown from both, 701,514 characters
median scrape request at the client, 484 ms on the API timer
1.0 s
median scrape request at the client, 484 ms on the API timer

Run of 2026-11-16.

Both arms asked for markdown with a plain HTTP fetch and the cache off. The scrape arm sent one request for every URL in the crawl response, which also held URLs that answered 404, so its 36.0 s covers more than the 74 pages. Each arm ran once.

Source: Web scraping vs web crawling, timed on one documentation site.

· The web

What top sites ask for in robots.txt

The robots.txt of 100 top-ranked domains

domains that served a robots.txt
67 of 100
domains that served a robots.txt
files that hold Disallow: / for crawlers they do not name
13 of 67
files that hold Disallow: / for crawlers they do not name
files that name at least one AI crawler, from a list of 40 fixed before the run
29 of 67
files that name at least one AI crawler, from a list of 40 fixed before the run
files that point to a sitemap
48 of 67
files that point to a sitemap

Run of 2026-11-16.

One request for robots.txt to each domain through /scrape, and one more at www where the bare name served none. The 100 domains come from the public Tranco ranking, and our reading of the full disallows was checked against Python urllib.robotparser. The counts say what the files asked for on one day. They do not say what anyone may do with a page.

Source: Is web scraping legal? What 67 top sites ask for in robots.txt.

· Cost

Cost per 1,000 pages by kind of page

25 pages of each of five kinds, day 1 of a seven-day run

per 1,000 documentation pages
$0.27
per 1,000 documentation pages
per 1,000 articles
$0.33
per 1,000 articles
per 1,000 pages fetched from search results
$0.37
per 1,000 pages fetched from search results
per 1,000 store product pages
$1.38
per 1,000 store product pages
per 1,000 reference articles
$1.48
per 1,000 reference articles

Run of 2026-11-06.

Default settings and markdown output on the pay as you go meter, with the charge read from the costs field of each response. Each set also went out in http mode and in smart mode, and the two matched within a cent. Compute was under 2% of the charge in every set. These are one day's figures.

Source: Web scraping cost per 1,000 pages, by kind of page.

· Output

A fine-tuning dataset from one search and one crawl

One search call and one crawl call of 60 documentation pages

both calls together
11 s
both calls together
the cost fields of both calls added up
$0.052
the cost fields of both calls added up
training records after two short scripts, plus 84 held-out sections
396
training records after two short scripts, plus 84 held-out sections
o200k_base tokens in the 60 pages as raw HTML and as markdown, about 37 times smaller
7,428,798 and 202,123
o200k_base tokens in the 60 pages as raw HTML and as markdown, about 37 times smaller

Run of 2026-11-06.

The search call returned ten results with their content in 4.5 s for $0.0041. The crawl call returned 60 pages as markdown in 6.5 s for $0.048. The same 60 URLs were fetched again as raw HTML to count tokens. Both scripts are in the post.

Source: Build a fine-tuning dataset from the web in two API calls.

· Output

What a web page costs in tokens

Ten public pages of different kinds, four output formats each

fewer tokens as markdown than as raw HTML, at the median
15x
fewer tokens as markdown than as raw HTML, at the median
the spread, from a front page that is mostly text to a documentation page
2.9x to 47x
the spread, from a front page that is mostly text to a documentation page

Run of 2026-11-05.

Each page was fetched as raw HTML, markdown, readability and text. Tokens were counted with gpt-tokenizer 4.0.0 and the o200k_base encoding. Other model families tokenize differently, so the ratios hold better than the absolute counts.

Source: What a web page costs in tokens.

· Browser

Browser Fidelity Bench V1

80 tasks, each one real page on a protected production site

tasks passed, 100%
80 of 80
tasks passed, 100%

Run of 2026-09-18.

An open task suite run on its own harness at revision 46fc6a03. Each task opens a site, does one basic interaction and reads content, with no login. A task passes when the page loads for the browser and fails when the response matches a block pattern. No model judges the result.

Source: The Browser Fidelity Bench post.

Code and data: spider-rs/benchmark.

· Browser

Browser connect and page load latency

44 of 50 rounds completed against google.com, 17 of 20 against example.com

median to open the socket, have the key accepted and get a protocol answer
879 ms
median to open the socket, have the key accepted and get a protocol answer
median connect plus page attach
1184 ms
median connect plus page attach
median example.com load to domcontentloaded, once attached
238 ms
median example.com load to domcontentloaded, once attached

Run of 2026-09-14.

Raw CDP over the public WebSocket with no SDK, from a laptop on the public internet. Ten warm-up rounds were discarded before the measured rounds. Medians are over the rounds that completed.

Source: The cloud browser page.

· Extraction

Structured extraction latency on /ai/scrape

30 sequential requests for one page, after 3 discarded warm-ups

median at the client, 15.2 s at p95
10.2 s
median at the client, 15.2 s at p95
median on the API timer
7.5 s
median on the API timer
requests that succeeded with every schema field filled
30 of 30
requests that succeeded with every schema field filled

Run of 2026-09-14.

One request at a time with a 1.2 s gap, from a laptop on a residential connection. Client time runs from the start of the request to the parsed JSON body. Server time is the duration_elasped_ms field of the response. Success means HTTP 200, page status 200 and no page error.

Source: The AI Studio page.

· Browser

999 URLs through the cloud browser

999 URLs across 254 domains in 18 categories, protected sites included

URLs that returned a title and a content body
999 of 999
URLs that returned a title and a content body
median page time, 39.3 s at p95, 25 sessions in flight
11.5 s
median page time, 39.3 s at p95, 25 sessions in flight
median connect, navigate and close in a separate 50-round speed test
2,509 ms
median connect, navigate and close in a separate 50-round speed test
median session creation in that speed test
705 ms
median session creation in that speed test

Run of 2026-02-17.

For each URL the client connected to a remote browser, navigated, waited for content, extracted it and took a screenshot, with no manual retries. The whole run took about 19 minutes. The speed test made 50 measured rounds after 10 warm-ups against google.com, waiting for domcontentloaded.

Source: The 999 URL browser benchmark post.

Code and data: spider-rs/spider-browser-dataset.

· Fetching

1,000 URLs through the API

400 static pages, 350 JavaScript-heavy apps and 250 protected pages, shuffled once

URLs that returned usable content, 99.6% on the protected 250
99.9%
URLs that returned usable content, 99.6% on the protected 250
throughput over the whole corpus, 182 pages/s on static pages
74 pages/s
throughput over the whole corpus, 182 pages/s on static pages
time to first result on a static page
45 ms
time to first result on a static page

Run of 2026-02-11.

The API was called from one cloud machine in us-east-2, so latency includes the network round trip. Throughput is pages returned divided by wall time, retries included. Usable content means HTTP 200 with a non-empty body. Throughput depends on this URL mix and on concurrency.

Source: The 1,000 URL benchmark post.

Code and data: spider-rs/benchmark.

How we measure, so you can run it again.

Each figure comes from the run on the date beside it. The post for each run prints the rule that picked its pages next to the result. When a run lost a page or a round, the figure says so, as the browser latency run does with 44 of 50 rounds.

Time comes in two kinds, and we keep them apart. Wall time at the client runs from the moment our script sends a request to the moment it has the answer, so it includes the network. The API's own timer is the duration_elasped_ms field of the response, and it stops at our side of the network. Where a run reports both, both are on this page.

Cost is what the API billed for the request, read from the costs field of each response on the pay as you go meter. We do not estimate it from page size. Token counts use the o200k_base encoding, and the run of 2026-11-05 names the tokenizer version.

To repeat a fetch run, send the same request in both modes and compare. This is the plain fetch of one page from the run of 2026-11-16.

curl -s https://api.spider.cloud/scrape \
  -H "Authorization: Bearer $SPIDER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://quotes.toscrape.com/js/",
       "request": "http",
       "return_format": "markdown"}'

Change "http" to "browser" for the rendered fetch.

Every parameter in that request is in the Scrape API reference. The browser runs and the 1,000 URL run have their code and URL lists on GitHub, linked beside each run above.

How to cite a run.

Give the run's date and link to its anchor here or to its post. The date matters more than the link. A run from February measured the product as it was in February. We keep old runs up beside new ones and do not overwrite them.

Spider, "Web scraping benchmarks", run of 2026-11-16, https://spider.cloud/benchmarks/#plain-vs-rendered-2026-11-16

Two things are not measured here. One is sustained throughput at the pay as you go rate limit, which starts at 10,000 requests a minute. The other is the number of requests in flight on an Unlimited plan. Both are limits the docs state, and we quote them as limits, not as results.

Runs that put Spider next to other providers are dated snapshots on the research page.

Run one page yourself and check our numbers.

The web scraping guide walks through what these runs mean for a scraping job. The tool page sends one request for a URL you choose and shows the markdown and JSON that came back.