Skip to main content

Web scraping, measured on pages you can fetch yourself.

Web scraping is a program asking a site for a page, the way a browser does, and keeping the parts you want as text or as named fields. That is the whole idea, and it fits in one request.

We run a scraper, so this guide skips the theory where it can. Each section starts from a run we made, gives its date and how many pages it covered, and links to the post that prints the method.

A plain fetch returned the text on 26 of 29 pages.

On 2026-11-16 we fetched 29 public pages twice through the scrape endpoint. They were 10 documentation pages, 10 news articles and 9 store product pages. The first request was a plain HTTP GET that runs no JavaScript. The second loaded the page in a browser, ran its scripts and read the result. On 26 of the 29 the plain fetch returned at least 90% of the words the browser did, and the lowest page was at 77%.

A scraper starts where a browser starts. It sends a GET for a URL, and the server answers with a status code, headers and HTML. On those 26 pages the server had already written the article or the product description into that HTML, so the only work left was to convert it.

Other pages send a nearly empty document and a script that builds the content after it loads. We picked 10 pages we expected to work that way, and on 8 of them the plain fetch returned under half of the browser's words. One is a practice page whose ten quotes sit inside a script tag. Its plain fetch came back with 8 words and its rendered fetch with 240. We chose those 10 by hand, so they say nothing about how common such pages are.

Rendering has a price. Across the 29 pages the plain fetch had a median cost of $0.36 per 1,000 requests on the pay as you go meter and a median wall time of 1.5 seconds. The rendered fetch came to $1.40 and 10.0 seconds, because a browser also downloads the scripts and data calls a page asks for.

curl -s https://api.spider.cloud/scrape \
  -H "Authorization: Bearer $SPIDER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://quotes.toscrape.com/js/",
       "request": "http",
       "return_format": "markdown"}'

The plain fetch of the practice page. Change "http" to "browser" for the rendered one.

The 39 pages and the rule that picked them are in How web scraping works: what a plain fetch gets and misses.

One page we picked for the browser came back fuller without it.

We put a caniuse.com page on the list of 10, sure that it needed a browser. The plain fetch returned 477 words, and the browser support table is among them. The rendered fetch returned 15 words after 77 seconds, and all 15 are the page title. We were wrong about what the first response holds.

It was not the only page to go that way. On 7 of the 29 documentation, news and product pages the plain fetch returned more words than the rendered one, and the widest gap was a store product page at 1,158 words against 772.

A word count says how much came back. It does not say whether what came back is the page. So send your own URL both ways and read the output before you trust either mode. If you leave request out, it defaults to smart, which tries the plain fetch first and moves to a browser when the page needs one. We sent none of these pages in smart mode, so the run says nothing about it.

Sixty pages were 37 times smaller as markdown than as HTML.

On 2026-11-06 one crawl call returned 60 documentation pages as markdown, and we fetched the same 60 URLs again as raw HTML. Counted in tokens, the HTML was about 37 times the size of the markdown.

  • Raw HTML 7,428,798
  • Markdown 202,123

Tokens in the same 60 pages, o200k_base encoding, 2026-11-06.

The response is JSON with one object per page, and the page is in its content field. With return_format set to markdown, the converter keeps headings, links and text and drops script and style elements. On a documentation page that is most of the bytes. The median page in this run was 486,606 bytes of HTML and 11,673 bytes of markdown. Both requests are in Build a fine-tuning dataset from the web in two API calls.

The ratio depends on the page. A day earlier, on 2026-11-05, ten public pages of different kinds shrank 15x at the median. A front page that is already mostly text shrank 2.9x and a documentation page 47x. If a model reads the result, those are the tokens it bills you for. The ten pages are in What a web page costs in tokens.

A reference article cost five times a documentation page.

On 2026-11-06 we scraped 25 pages of each of four kinds with the default settings and markdown output, and read the charge from the costs field of each response. Per 1,000 pages, documentation came to $0.27, articles to $0.33, store product pages to $1.38 and reference articles to $1.48. It was the first day of a seven-day run, so these are one day's figures. The rule that picked each set of 25 is in Web scraping cost per 1,000 pages, by kind of page.

The spread is page weight. Pay as you go has two meters, $1 per GB of bandwidth and $0.0001 per minute of compute, and compute was under 2% of the charge in that run. A long reference article carries more HTML than a short documentation page, so it costs more to fetch. Within one kind the lightest and heaviest page were far apart too. For documentation they were $0.07 and $0.89 per 1,000.

The default did not cost more than plain HTTP on those pages. Each set went out in http mode and again in smart, and the cost per 1,000 pages matched within a cent. Forcing a browser on every page is a different matter, as the 29 pages above show.

An attempt bills only the bandwidth and compute it used. A blocked page usually costs a fraction of a cent. The other plan, Unlimited, does not bill by the page. It is a flat monthly price for a number of requests in flight, from 2 for $40 a month. Both are on the pricing page.

Thirteen of 67 robots.txt files hold a full disallow for unnamed crawlers.

A robots.txt is the file where a site writes down what it asks of crawlers, and a scraper reads it before the first page. On 2026-11-16 we asked 100 top-ranked domains for theirs, and 67 served one. In 13 of the 67, the group of rules for any crawler the file does not name holds the line Disallow: /. Ten of those have no Allow line beside it. The other three add exceptions, and one of them lists 178 Allow lines ahead of its disallow.

A search for that one line would file all 13 together. A parser that applies the rules would not. AI crawlers get the same mixed treatment. 29 of the 67 files name at least one, and 12 of those 29 hold no full disallow for any of them. Most of the files also say where the pages are, since 48 of the 67 point to a sitemap.

These are counts of what 67 files said on one day. They do not say what you may do with a page after you fetch it, and neither does this guide. RFC 9309, the standard for the file, says its rules "are not a form of access authorization." Whether following a file or ignoring one carries legal weight depends on where you are, on the site's terms and on the data. This is not legal advice.

On Spider the respect_robots option defaults to true. The rule that picked the 100 domains and the check against a second parser are in Is web scraping legal? What 67 top sites ask for in robots.txt.

One crawl call returned 60 pages in 6.5 seconds.

Which request to send depends on what you already have. With a URL in hand, send it to /scrape. With a site and no list of its pages, use /crawl. You give a starting URL and a limit, and the crawler follows the links. With only a question, start at /search and choose your sources from the results.

We ran the last two as one job on 2026-11-06. A search call returned ten results with their page content in 4.5 seconds for $0.0041. Then a crawl call returned 60 documentation pages as markdown in 6.5 seconds for $0.048. Fetching the same 60 pages with one scrape call each, six at a time, took 16.3 seconds. Those scrape calls asked for raw HTML, so the pair differs in format as well as in call count. The 16.3 seconds is no measure of crawl against scrape, which a run on one documentation site timed on 2026-11-16 with both arms asking for markdown.

Keyless access is /scrape only. It is rate limited. On a 429, honor Retry-After. Get a key for anything sustained. Without a key, every route other than /scrape returns 402, except /unlimited/*, which returns 401.

Six posts hold the method behind these numbers.

Jeff Mendez ran the measurements dated 2026-11-05 and 2026-11-06. We made two of the three runs of 2026-11-16 for this guide. Every number above comes from one of these six posts.

What people ask before they scrape a site.

What is web scraping used for?

Any job where a program needs what a web page says. The two we have run and published are giving pages to a language model and building a training file. On 2026-11-06 one search call and one crawl call took 11 seconds and $0.052 together, and two short scripts turned what they returned into 396 training records.

Does web scraping need a headless browser?

Only for pages that build their content with JavaScript. In our run of 2026-11-16 a plain HTTP fetch returned at least 90% of a browser's words on 26 of 29 documentation, news and store product pages. On 10 pages we picked because we expected them to need a browser, it returned under half on 8.

How long does it take to scrape a page?

In the run of 2026-11-16 the plain fetch had a median wall time of 1.5 seconds and the rendered fetch 10.0 seconds, over 29 pages, measured at the client with four requests in flight. The API's own timer put the median plain fetch at 884 milliseconds.

How much does it cost to scrape 1,000 pages?

On Spider's pay as you go meter it depends on the weight of the page. On 2026-11-06, with 25 pages of each kind, 1,000 pages came to $0.27 for documentation, $0.33 for articles, $1.38 for store product pages and $1.48 for reference articles. The Unlimited plan bills by requests in flight and not by the page, from 2 for $40 a month.

Is it legal to scrape a website?

This guide does not answer that. It depends on the jurisdiction, on the site's terms and on the kind of data, and we assess none of them here. What we can count is what sites ask for. On 2026-11-16, 67 of 100 top-ranked domains served a robots.txt, and 13 of the 67 hold Disallow: / for crawlers they do not name.

What is the difference between web scraping and web crawling?

Scraping reads a page whose address you already have. Crawling starts from one address, follows links to find the pages and scrapes each one. On 2026-11-06 one crawl call returned 60 documentation pages as markdown in 6.5 seconds for $0.048.

Paste a URL and read what one scrape returns.

The tool page takes a URL, sends one request and shows you the markdown and the JSON that came back.