Skip to main content

How web scraping works: what a plain fetch gets and misses

How web scraping works, measured on November 16, 2026. A plain HTTP fetch returned 90% or more of a browser's words on 26 of 29 docs, news and product pages.

Jeff Mendez 10 min read

On November 16, 2026, between 02:21 and 02:27 UTC, we fetched 29 public pages twice through the Spider scrape endpoint. One request was a plain HTTP fetch and the other rendered the page in a browser. The pages were 10 documentation pages, 10 news articles and 9 store product pages. On 26 of the 29 the plain fetch returned at least 90% of the words the browser returned, and the lowest page was at 77%. The plain fetch had a median cost of $0.36 per 1,000 requests and a median wall time of 1.5 seconds. The rendered fetch came to $1.40 and 10.0 seconds.

On pages like these a scraper has little to do. You send one request and convert what comes back, the first step in the full scraping guide. A fourth kind of page went the other way. We also fetched 10 pages that build their content in the browser, and on 8 of the 10 the plain fetch returned under half of the browser’s words. On crates.io it returned 4 words against 276.

Three kinds of page had their text in the first response

Kind of pagePagesPlain fetch had at least 90% of the browser’s wordsLowest pageMedian cost per 1,000 requests, plain and renderedMedian wall time, plain and rendered
Documentation108 of 1087%$0.25 and $0.671.3 s and 8.3 s
News articles1010 of 1094%$0.28 and $0.791.5 s and 8.5 s
Store product pages98 of 977%$1.11 and $2.751.8 s and 18.2 s
Pages built in the browser102 of 100.1%$0.05 and $0.661.0 s and 9.0 s

Run of 2026-11-16. Each page was fetched once per mode with markdown output, and words are the markdown split on whitespace. A page counts when both requests answered with status 200. Cost is the costs.total_cost field on the pay as you go meter, and each figure is the median request multiplied by 1,000. Wall time is measured at the client with four requests in flight. The Unlimited plan bills by concurrency and not by request, from $40 a month for 2 requests in flight, so these per-request figures do not apply to it.

The last row is not a claim about the web. We chose those 10 pages as likely to need a browser, so 8 of 10 says the choice was mostly right and no more.

The request is one GET and it took about a second

A scraper starts the way a browser does. It sends an HTTP GET for a URL and the server answers with a status code, headers and a body. For a web page the body is HTML.

The run put a time on it. Over the 29 documentation, news and product pages, the API’s own timer, the duration_elasped_ms field, put the median plain fetch at 884 milliseconds.

On those pages the HTML already held the text. The server, or a build step that ran before you asked, had written the article or the product description into the response. Nothing else has to happen before you can read it. On 13 of the 29 pages the two modes returned the same number of words, to the word.

Rendering runs the page’s scripts, and the bill follows what they download

Some pages send a nearly empty HTML document and a script, and the script builds the content after it loads. A plain fetch does not run scripts, so it gets the empty document.

The serde page on crates.io is a clear case. The run’s plain fetch of it, converted to markdown, came back like this:

{
  "status": 200,
  "url": "https://crates.io/crates/serde",
  "content": "crates.io: Rust Package Registry",
  "costs": { "total_cost": 0.0000212 },
  "duration_elasped_ms": 755
}

Four words. The same page rendered:

{
  "status": 200,
  "url": "https://crates.io/crates/serde",
  "content": "serde - crates.io: Rust Package Registry\n**Serde is a framework for *ser*ializing and *de*serializing Rust data structures efficiently and generically.**\nYou may be looking for:\n* [An overview of Serde](https://serde.rs)\n ...",
  "costs": { "total_cost": 0.000664 },
  "duration_elasped_ms": 10657
}

That one has 276 words. Both are the run’s responses, trimmed to five fields, and the second content is cut after the first link.

A page can carry its data in the first response and still come back empty. After the run, on the same date, we sent three more requests for one page of the run, a practice page that lists ten quotations. Its raw HTML was 5,808 bytes. All ten quotations are in it, as a JavaScript array inside a <script> tag, followed by a loop that calls document.write once per quotation. No quotation appears in the HTML outside that script. The plain fetch as markdown was 8 words and the rendered one 240, the same counts as in the run.

Rendering means a browser loads the page, runs its scripts, waits for the page to settle and then hands over the HTML as it stands. A browser also downloads what the page asks for: scripts, styles, data calls. Those bytes are on the meter. In the rendered request for the practice page, the byte charge was $0.00056 of the $0.00061 total.

Markdown is the same HTML with the scripts and tags taken out

After the fetch, plain or rendered, a converter turns the HTML into the format you asked for. With return_format set to markdown it keeps headings, links and text and drops script and style elements. That is why the practice page’s 5,808 bytes of HTML became 110 characters of markdown. The quotations were in the response the whole time, inside a script the converter threw away.

All the word counts in this guide are counts of that markdown. On ten other pages on November 5, markdown cut the token count 15x at the median against the raw HTML.

Named fields, such as a price or a title, are a second step on the same HTML. You write a CSS selector per field, or you read the JSON-LD block many store pages carry. This run did not test field extraction.

robots.txt is read before the first page

A site’s robots.txt file says which paths it asks crawlers to stay out of, and sometimes how long to wait between requests. A scraper reads it first.

The script for this run fetched each site’s robots.txt through the API before any page, and applied a rule written ahead of time. It dropped a site when the file disallowed the path for all agents, set a crawl delay for all agents, or could not be read. It dropped two candidate sites for a crawl delay and none for a disallowed path.

The API’s respect_robots option defaults to true, and the run’s requests did not set it. The file states what a site asks of crawlers. It does not settle what you may do with a page after you fetch it, and this guide does not either.

On one page the plain fetch returned 477 words and the render returned 15

These are the 10 pages chosen as built in the browser.

PageWords, plain fetchWords, rendered
A practice page that writes ten quotations with a script8240
A practice store page that loads its product list with a background request39157
crates.io, the serde crate4276
bundlephobia.com, react 18.2.08144
hn.algolia.com, a search for “web scraping”61,095
caniuse.com, CSS grid47715
bsky.app, the Bluesky profile33,095
mastodon.social, the about page33
demo.realworld.show1161
todomvc.com, the React example25

Words of markdown per page, one fetch per mode, 2026-11-16.

Seven rows went the way the list assumed. The caniuse.com row went the other way. The plain fetch returned 477 words, which are the page title, the feature description, the support table with a version list per browser, and the resource links. The rendered fetch returned 15 words after 77 seconds, and they are the page title, “CSS Grid Layout (level 1) | Can I use… Support tables for HTML5, CSS3, etc”. Nothing below the title came back from the render.

Both modes returned the same 3 words for the Mastodon page, its title, “About - Mastodon.social”. The wait_for option tells a browser request to wait for a selector or a delay before it reads the page. The run did not set it. The TodoMVC row is under half as well. The plain fetch returned “TodoMVC: React” and the render added “Toggle All Input”.

The store pages had a smaller version of the caniuse result. On 7 of the 29 documentation, news and product pages the plain fetch returned more words than the rendered one. The widest gap was an Allbirds product page at 1,158 words plain and 772 rendered. The plain response has a block under the product heading with the name, the $110 price, the size list, the add to cart and back in stock text, and the alt text of the product photos. The rendered response has an empty heading where that block should be. That block is the whole gap of 386 words. Outside it the two responses have the same word counts.

A word count tells you how much came back. It does not tell you whether it is the page’s content. Read a sample of your own output before you trust either mode.

In practice, leave request at its default. smart tries a plain fetch first and moves to a browser only when the page needs one. This run did not test smart. A separate run on November 6 found it cost the same per 1,000 pages as http on four sets of pages, within a cent. Force browser when you know the page is built in the browser.

The request and the rule that picked the pages

curl -s https://api.spider.cloud/scrape \
  -H "Authorization: Bearer $SPIDER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://crates.io/crates/serde", "request": "http", "return_format": "markdown", "metadata": true}'

Change "request": "http" to "request": "browser" for the rendered fetch. The run itself sent "chrome", which the OpenAPI file lists as an older name for browser. After the run we sent the practice page once more with browser and got the same 8 and 240 words.

The table covers four kinds of page, and we fixed the page lists before the first request. Within a kind, the script took the first URL of each site, then the second of each, until it had 10.

  • Documentation, news and product URLs come from the lists built for the cost post on November 6, which were taken from each site’s sitemap or feed. The documentation sites are FastAPI, Astro, Docker, Pydantic and Vite. The news sites are the GitHub blog, the Stack Overflow blog, Ars Technica and TechCrunch. The stores are Allbirds, Death Wish Coffee, Ugmonk, tentree and Brooklinen. The Verge is on the November 6 list and is left out here because it runs a subscription meter.
  • The 10 pages built in the browser are the ones in the table above. That list was written by hand as 12 candidates, and the script used the first 10 that passed the robots.txt check.

The rule shapes the rows. News has four sites, so the GitHub blog and the Stack Overflow blog supply three pages each. The two documentation pages under 90% are both on vite.dev. Three of the nine product pages sell gift cards.

What this run does not show

  • How common pages built in the browser are. That list is 10 pages chosen by hand.
  • That the rendered markdown is right. The caniuse.com row is the counterexample.
  • Anything about smart mode, which is the default.
  • Much about any one kind. Ten pages is a small set, and one site is 2 or 3 of a kind’s pages.
  • Whether a second try gives the same answer. Each page was fetched once per mode.

Paste one URL into the scrape tool to see a fetch with no key, then send ten of your own URLs both ways and compare the word counts. Create an API key to run it.

Frequently asked questions

How does web scraping work?

A program sends an HTTP request for a URL, reads the HTML that comes back and converts it to markdown, text or named fields. When a page builds its content with JavaScript, a browser has to run the page first. In a run on November 16, 2026 the plain request returned at least 90% of the words a browser returned on 26 of 29 documentation, news and store product pages.

Do I need a headless browser to scrape a website?

For pages that build their content in the browser, yes. On 10 such pages, picked in advance as likely to need a render, the plain fetch returned under half of the browser's words on 8. On the 29 documentation, news and product pages in the same run, no page fell below 77%.

How much slower is a rendered fetch than a plain HTTP fetch?

Over the 29 documentation, news and product pages in the run of November 16, 2026, the plain fetch had a median wall time of 1.5 seconds and the rendered fetch 10.0 seconds, measured at the client with four requests in flight. The median cost on the pay as you go meter was $0.36 per 1,000 requests plain and $1.40 rendered.

Where does robots.txt fit in a scraper?

Before the first page request. The script for this run fetched each site's robots.txt first and dropped a site when the file disallowed the path for all agents, set a crawl delay for all agents or could not be read. It dropped two candidate sites for a crawl delay. The Spider API's respect_robots option defaults to true.

Sources

  1. Measured run, 2026-11-16: public pages by kind, each fetched once with a plain HTTP request and once in a browser, through the Spider scrape endpoint
  2. Spider OpenAPI file, the request field, its values and the respect_robots default
  3. Web scraping cost per 1,000 pages, by kind of page: the smart and http prices, measured 2026-11-06
  4. RFC 9309, the Robots Exclusion Protocol

Keep reading

Is web scraping legal? What 67 top sites ask for in robots.txt

We asked 100 top-ranked domains for robots.txt on 2026-11-16. Of the 67 that served one, 13 hold Disallow: / for unnamed crawlers and 29 name an AI crawler.

Web scraping vs web crawling, timed on one documentation site

Web scraping vs web crawling, measured on November 16, 2026. One crawl call returned 74 pages in 5.1 s. A scrape call per URL returned the same markdown.

Scraping 1 million pages: what actually happens

An engineering log of crawling 1 million pages across 10,000 domains with Spider's API. Throughput curves, failure modes, cost breakdown, and lessons learned.

Run this on a page you care about

The playground sends the request this page describes and shows you the response. Keyless runs work without an account and are rate limited.