Skip to main content

Web scraping vs web crawling, timed on one documentation site

Web scraping vs web crawling, measured on November 16, 2026. One crawl call returned 74 pages in 5.1 s. A scrape call per URL returned the same markdown.

Jeff Mendez 7 min read

On November 16, 2026, between 12:20 and 12:21 UTC, we fetched the Typer documentation site two ways through the Spider API. One crawl call, started at the homepage, returned 74 distinct pages in 5.1 seconds. Then we sent one scrape call for each URL in the crawl’s response, six in flight, and that took 36.0 seconds. On all 74 pages the two calls returned the same markdown, character for character.

The 36.0 seconds needs a note first. The crawl’s response also held URLs that answered 404, and the scrape arm repeated the whole response, so both times cover more than the 74 pages. The scrape requests for the 74 pages alone add up to 90.7 seconds of single request time, which is about 15 seconds across six lanes. We computed that figure and did not time it as an arm of its own.

The distinction is short. Scraping fetches pages you name, which is the request our guide to web scraping measures step by step. Crawling starts at one page, follows its links and fetches what it finds. Most jobs use both, a crawl to learn what a site has and scrapes to refresh the pages you care about. The steps inside one fetch are the same in both.

One call against one call per URL

MeasureOne crawl callOne scrape call per URL
Requests the client sent1One for each URL in the crawl’s response
Distinct pages with status 2007474
Wall time at the client5.1 s36.0 s with six in flight
Cost$0.0210$0.0194
Markdown of the 74 pages701,514 characters701,514 characters

Run of 2026-11-16. Both arms asked for markdown with a plain HTTP fetch and the cache off. Wall time for the crawl is the one call. For the scrape arm it runs from the first request sent to the last answer read. Cost is the sum of the costs.total_cost fields on the pay as you go meter, and both sums cover the crawl’s whole response, the 404 URLs included. The Unlimited plan charges for requests in flight and not per request, from $40 a month for 2, so neither sum applies to it.

The two costs are 0.16 of a cent apart. On this site, on this day, a crawl and a scrape per URL cost about the same, and we would not call either one cheaper. On the median page of the 74 the price was $0.000126 by crawl and $0.000127 by scrape.

About half of a scrape call was the round trip

On the 74 pages the median scrape request took 1.0 second at the client. The API’s own timer for those requests, the duration_elasped_ms field, had a median of 484 milliseconds. The rest of a typical call, about half, passed outside that timer, while the request went out and the answer came back.

The scrape arm paid for that trip once per URL and the crawl paid for it once. Concurrency hides part of it. Six in flight was our setting, and a client with more lanes would finish sooner. We ran one setting, so we have no second point on that curve.

The markdown was identical on all 74 pages

The script compared the two markdown strings for each page and counted a pair as identical only when the strings were equal. It was 74 of 74. Each arm returned 701,514 characters for those pages, which is 84,742 words.

Both calls returned the same content. What differed was how the pages were found and how many times the client went to the API.

One page skews the totals. The median page is 4,433 characters of markdown, and the release notes page is 148,563, a fifth of the site’s markdown. It was not the most expensive page. The reference page for parameters has 71,996 characters, under half as much markdown, and a scrape of it cost $0.00073 against $0.00061 for the release notes. We did not look into why. If you estimate a job from page counts, the markdown length of a page did not predict its price here.

The crawl returned 74 of the sitemap’s 76 pages and none outside it

The site’s sitemap lists 76 URLs, 58 of them under /tutorial/. The crawl started at the homepage with no list and returned 74 of the 76 with status 200. The two it did not return are /environment-variables/ and /virtual-environments/. A crawler reaches a page through a link, and we did not check whether any page on the site links to those two.

In the other direction the crawl returned no page with status 200 that the sitemap does not list. To compare the two lists the script lower-cased the host and dropped fragments, index.html and the trailing slash.

On this site discovery told us nothing the sitemap had not. That is a fair guide to when you need a crawl at all. If a site publishes a sitemap and keeps it current, the sitemap is your list and you can scrape from it. If it has no sitemap, or the sitemap is partial or stale, following links is how you find the pages. The crawl endpoint also has a sitemap option that the API reference describes as “Add pages from the sitemap to the crawl. Default false.” The run left it off, and we have not measured it.

Pick the call by what you already have

You haveCallWhat the run says
A list of URLsOne scrape per URL, several in flightMedian request 1.0 s on 74 pages
One site and no listOne crawl from its start page74 pages in one call of 5.1 s
A question and no siteSearch, then scrape the resultsNot part of this run

The third row comes before either of the others. A crawl needs a start page and a scrape needs a URL, and a search is how you get them when all you have is a question. The API reference says the search route returns ranked results for a query and, with fetch_page_content set to true, fetches each result as well.

The two combine well. Crawl a site once when you first meet it, then scrape the pages that change on a schedule. For the 74 pages here the content was the same from either call.

The two requests and the rule that picked the site

curl -s https://api.spider.cloud/crawl \
  -H "Authorization: Bearer $SPIDER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://typer.tiangolo.com/", "limit": 200, "return_format": "markdown", "request": "http", "metadata": true}'
curl -s https://api.spider.cloud/scrape \
  -H "Authorization: Bearer $SPIDER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://typer.tiangolo.com/tutorial/arguments/", "return_format": "markdown", "request": "http", "metadata": true}'

The crawl returned fewer objects than its limit of 200, so the limit did not cut it short. The run sent one more field in both bodies, cache set to false, so that neither arm could read a stored copy. Add it to repeat the measurement. For other work leave the cache on, which is the default. The scrape arm sent the second request once for each URL in the crawl’s response, six at a time.

We wrote the rule before the first request. Ten documentation sites of open source projects went on a list in a fixed order. A site qualified when its robots.txt, read through the API, allowed the start URL and set no crawl delay over 5 seconds for all agents, and when its sitemap listed 50 to 150 URLs under the start path. The first site to qualify was the one we used, and Typer was first on the list.

The rule has a side effect. Our request for the Typer site’s robots.txt did not come back with a robots file, and the rule reads that as no restrictions. The size limits are there to keep the test small, so they also keep it from saying anything about a site of 10,000 pages.

What this run does not show

  • The two calls on a site that is large, slow or built in the browser. These are 74 static documentation pages fetched without a browser.
  • A second day. Each arm ran once, and the crawl took 5.1 seconds that one time.
  • Another concurrency. The scrape arm ran at six in flight and at no other setting.
  • A scrape arm that starts from the sitemap. That would be 76 calls, and we did not send them.
  • Why two sitemap pages were not in the crawl’s response.
  • A cheaper call. The costs differ by a sixth of a cent.

Pick a site whose sitemap you can read, try one of its pages without a key, then send both requests and count the pages against it. Create an API key to run them.

Frequently asked questions

What is the difference between web scraping and web crawling?

Scraping fetches pages you name. Crawling starts from one page, follows its links and fetches what it finds. In a run on November 16, 2026 the two returned the same markdown on 74 of 74 pages of one documentation site, so the difference was in how the pages were found and how many requests the client sent.

Is one crawl call faster than scraping the same pages one by one?

On one documentation site on November 16, 2026, one crawl call took 5.1 seconds at the client. One scrape call for each URL in the crawl's response, with six requests in flight, took 36.0 seconds. That list held URLs that answered 404 as well as the 74 pages, and the scrape requests for the 74 pages alone add up to 90.7 seconds of single request time.

Does a crawl return different content than a scrape of the same page?

It did not in this run. Both calls asked for markdown with a plain HTTP fetch and the cache off, and the markdown strings were equal on all 74 pages that both returned, 701,514 characters in total.

Do I need a crawler if the site has a sitemap?

On the site in this run the sitemap listed 76 URLs. The crawl returned 74 of them and no page with status 200 that the sitemap lacked. Where a sitemap is that complete it is already your list of URLs. A crawl is for sites that have no sitemap or one you cannot rely on.

Sources

  1. Measured run, 2026-11-16, 12:20 to 12:21 UTC: one crawl call on one documentation site, then one scrape call for each URL the crawl returned, through the Spider API
  2. Spider OpenAPI file, the crawl, scrape and search routes and the limit, cache and sitemap options
  3. The Typer documentation sitemap, the list of 76 URLs the crawl was compared with
  4. Spider pricing page, the Unlimited plan's price by concurrency

Keep reading

How web scraping works: what a plain fetch gets and misses

How web scraping works, measured on November 16, 2026. A plain HTTP fetch returned 90% or more of a browser's words on 26 of 29 docs, news and product pages.

Is web scraping legal? What 67 top sites ask for in robots.txt

We asked 100 top-ranked domains for robots.txt on 2026-11-16. Of the 67 that served one, 13 hold Disallow: / for unnamed crawlers and 29 name an AI crawler.

Scraping 1 million pages: what actually happens

An engineering log of crawling 1 million pages across 10,000 domains with Spider's API. Throughput curves, failure modes, cost breakdown, and lessons learned.

Run this on a page you care about

The playground sends the request this page describes and shows you the response. Keyless runs work without an account and are rate limited.