# Best web scraping APIs for AI and LLM pipelines in 2026

On 60 docs pages, markdown took 3,369 tokens a page and raw HTML 123,813. Seven web scraping APIs for RAG and LLMs, with 100,000-page costs where one is listed.

Canonical: https://spider.cloud/blog/best-web-scraping-apis-for-ai-2026/

An LLM pipeline pays for a web page twice: once to fetch it and again in tokens for as long as the page sits in context. Spider’s [web scraping API](/api/scrape/) costs from $0.16 per 1,000 pages, and cost varies by site. Documentation pages had a median of $0.275 per 1,000 over twelve days of runs in November 2026, and store product pages $1.388. On 60 Kubernetes documentation pages we crawled on November 6, 2026, the markdown came to 202,123 tokens and the raw HTML of the same pages to 7,428,798. Divided by 60, that is 3,369 tokens a page against 123,813.

Scale that crawl to 100,000 pages and the fetch comes to about $80 and the markdown to about 337 million tokens. Kept as raw HTML, the same pages would be about 12.4 billion tokens. All three totals are computed from the 60-page run, which cost $0.048, and none was measured at that volume. Those Kubernetes pages are heavy for documentation. The fixed set of 25 documentation pages we fetch daily costs $0.274 per 1,000 at the low end of its daily range, and at that rate 100,000 pages would come to $27.40. The token bill is the one that grows with every query, so the format an API returns matters as much as its price.

Below are six other APIs used in AI work, with each one’s price for 100,000 pages. Theirs are list prices read from the vendor’s pricing page on the date shown. Ours is computed from measured runs. The rows are different pages on different meters, so they sit side by side and do not divide into a ratio.

## What 100,000 pages cost on each API

| API                       | 100,000 pages in a month                                                                                                                          | As of      | An engine you can run yourself                      |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | --------------------------------------------------- |
| Spider                    | From $16 on light static pages and $27.40 on a fixed set of documentation pages, both computed from measured rates per 1,000. Cost varies by site | 2026-11-17 | Yes, the spider crate, MIT                          |
| Firecrawl                 | $99 list on the Standard plan, or $83 a month billed yearly                                                                                       | 2026-11-17 | Yes, AGPL-3.0                                       |
| ScrapingBee               | $49 list on the Freelance plan at 1 credit a page. A request takes 1 to 75 credits by rendering and proxy tier                                    | 2026-11-17 | No                                                  |
| Bright Data, Web Unlocker | $150, computed at the pay as you go list rate of $1.50 per 1,000 successful requests                                                              | 2026-11-17 | No                                                  |
| Apify                     | No per page price. Compute units cost $0.20 on Starter, and the pages a unit covers depend on the Actor                                           | 2026-09-02 | No                                                  |
| Crawl4AI                  | No license fee. You supply compute, proxies and LLM tokens                                                                                        | 2026-09-02 | Yes, Apache-2.0                                     |
| Jina Reader               | Billed per token, with 10,000,000 free tokens for each new key. No per page price                                                                 | 2026-09-02 | Public repo, Apache-2.0, no self-hosting docs found |

Competitor figures come from each pricing page on the date in the third column. Spider’s light static figure is 25 catalogue pages of a practice site with the cache off, from the [pricing guide](/guides/pricing-and-plans/), and the documentation figure is $0.274 per 1,000, the low end of the daily range over [twelve days of runs](/blog/true-cost-of-web-scraping-at-scale/). Both Spider totals are computed from those rates. Licenses were read from each GitHub repository on November 18, 2026.

## Spider returns a whole site as markdown in one call

A `/crawl` request takes a start URL and a page limit and follows the links itself. Set `return_format` to `markdown` and every page comes back as markdown. The November 6 run above was one such call. It came back in 6.5 seconds with 60 pages, all status 200. You do not write a queue or a link extractor. That run set `request` to `http`, because documentation needs no rendering. Leave the field out and the default `smart` mode decides per page whether to open a browser. At the twelve-day median it cost the same per 1,000 pages as plain HTTP on four fixed page sets, within a third of a cent.

The meter has two terms, $1 per GB and $0.0001 per minute of compute, so a light page costs less than a heavy one. That is why the range is wide, from $0.16 per 1,000 on light static pages to $1.717 on Wikipedia reference articles at the twelve-day median. If you would rather have a fixed bill, [Unlimited](/docs/api/unlimited/) prices by requests at once, from $40 a month for 2 in http mode.

If what you need is one named site with a maintained scraper already written for it, look for it in Apify’s Actor store and in our [scraper catalog](/scrapers/) before you write your own.

## Firecrawl sells a flat monthly plan

Firecrawl is a hosted API built around markdown output, and it sells monthly plans. Standard is $99 a month, or $83 billed yearly, for 100,000 credits. Unused credits do not carry over on Free, Hobby, Standard or Growth. Scale and Enterprise roll them over. That suits a team whose volume is steady and close to a plan’s size. A team whose volume swings will pay for credits it does not use in slow months.

The engine is open source under AGPL-3.0. You can self-host it, and if you change it and offer it as a network service, the license asks you to publish your changes.

## Crawl4AI runs inside your own Python process

Crawl4AI is a Python library, licensed Apache-2.0 with an attribution clause. You import it into the same process as your chunking and embedding code, which is its main appeal. There is no license fee, and that is the whole of what is free. The browsers, the proxies, the retries and any LLM calls for extraction are yours to run and pay for. A hosted Cloud API was in closed beta with no published pricing when we read the docs on September 2, 2026.

## Jina Reader, for a prototype in a notebook

You put `r.jina.ai/` in front of a URL and get the page back as text. It bills per token, and a new key gets 10,000,000 tokens. Each request is one URL, and there is no crawl endpoint, so a whole site means building the list of URLs yourself. For a dozen pages in a notebook, the URL prefix is all the setup it needs. Because it bills by output tokens, long pages cost more, and Jina publishes no per page figure to plan with.

## ScrapingBee prices a request by its options

ScrapingBee sells monthly credit plans, from $19 for 75,000 credits. A plain page is 1 credit, so 100,000 plain pages fit the $49 Freelance plan. Rendering and the higher proxy tiers raise a request to as much as 75 credits, so what 100,000 pages cost depends on which options your sites need.

## Apify, when someone already wrote the scraper

Apify runs scrapers called Actors, and its store holds prebuilt ones for many named sites. Its pricing page sells compute units, $0.20 each on the Starter plan, and does not say how many pages one covers. The memory and run time of the Actor you pick decide that. Use it for the specific site an Actor already covers, and run a small job first to learn what a page costs.

## What Bright Data charges per successful request

Bright Data sells several products, and the one closest to a scrape API is Web Unlocker. Its pay as you go rate is $1.50 per 1,000 successful requests, and a separate free plan covers 5,000 requests a month. Scale plans start at $499 a month for 383,000 requests. Its proxies and its hosted browser bill per GB instead. A per success price is easy to forecast when you know your page count: 100,000 billable requests work out to $150 at the pay as you go rate. Spider’s meter starts from $16 for 100,000 light static pages, and cost varies by site. Store product pages come to $138.80 and Wikipedia reference articles to $171.70 at their twelve-day medians. All three are computed from measured rates per 1,000.

## Crawl a docs site and count its tokens

This is the November 6 request in Python. It needs `requests`, `tiktoken` and a Spider key in `SPIDER_API_KEY`.

```python
import os
import requests
import tiktoken

resp = requests.post(
    "https://api.spider.cloud/crawl",
    headers={"Authorization": f"Bearer {os.environ['SPIDER_API_KEY']}"},
    json={
        "url": "https://kubernetes.io/docs/concepts/",
        "limit": 60,
        "whitelist": ["/docs/concepts/"],
        "return_format": "markdown",
        "metadata": True,
        "request": "http",
    },
    timeout=300,
)
pages = resp.json()

enc = tiktoken.get_encoding("o200k_base")
tokens = sum(len(enc.encode(p.get("content") or "")) for p in pages)
cost = sum((p.get("costs") or {}).get("total_cost", 0) for p in pages)
print(f"{len(pages)} pages, {tokens:,} tokens, ${cost:.4f}")
```

On November 6, 2026 this request returned 60 pages, all status 200, as 860,186 bytes and 202,123 tokens of markdown, and the `costs.total_cost` fields summed to $0.048. That works out to $0.80 per 1,000 pages, above the $0.275 median of our fixed documentation set. The meter follows page weight, and the raw HTML of these 60 pages came to 29,549,940 bytes, about 492,000 a page. The site changes over time, so your counts will drift from ours. Spider reads robots.txt by default, and the [fine-tuning dataset post](/blog/build-fine-tuning-dataset-from-the-web/) has the full run with its license check.

## What these numbers do not cover

Our runs measured documentation, articles, store product pages and reference pages on public sites. They do not measure pages behind a login, and they do not score structured extraction. For either, run your own URLs through each API you are considering and count what comes back.

## Frequently asked questions

### Which web scraping API is best for an LLM or RAG pipeline?

Pick one that returns markdown and bills by what each page weighs. Spider costs from $0.16 per 1,000 pages, and cost varies by site: documentation pages had a median of $0.275 per 1,000 over twelve days of runs in November 2026. On 60 documentation pages fetched on November 6, 2026, the markdown came to 202,123 tokens and the raw HTML to 7,428,798, which works out to 3,369 and 123,813 a page.

### Which web scraping APIs have a free tier for prototyping?

On the dates in the sources: Firecrawl's free plan has 1,000 credits a month, Jina Reader gives 10,000,000 tokens to each new API key, Apify's free plan includes $5 of usage a month and Bright Data's Web Unlocker has a free plan of 5,000 requests a month. Crawl4AI has no license fee and runs on your own machines. Spider's pay as you go has no subscription, and a top up starts at $5.

### What is the best API for crawling a whole site recursively?

One that follows links for you in a single call. A Spider /crawl request takes a start URL and a page limit and, with return_format set to markdown, returns every page it reaches as markdown. On November 6, 2026 one such call returned 60 Kubernetes documentation pages in 6.5 seconds, all with status 200. Jina Reader fetches one URL per request and has no crawl endpoint.

### How many tokens does a scraped page add to a model's context?

It depends on the format you keep. On 60 Kubernetes documentation pages fetched on November 6, 2026 and counted in the o200k_base encoding, raw HTML came to 7,428,798 tokens and markdown to 202,123. On ten mixed public pages fetched on November 5, 2026, the median page was 72,093 tokens raw and 4,384 as markdown.

## Sources

1. [Spider pricing guide, the measured cost table by kind of page, light static pages measured 2026-11-17](https://spider.cloud/guides/pricing-and-plans/)
2. [Web scraping at scale, twelve days of cost per 1,000 pages by kind of page, 2026-11-06 to 2026-11-17](https://spider.cloud/blog/true-cost-of-web-scraping-at-scale/)
3. [Spider Unlimited plan, priced by requests at once, read 2026-11-18](https://spider.cloud/docs/api/unlimited/)
4. [Measured run, 2026-11-06: one crawl call, 60 Kubernetes documentation pages, tokens as raw HTML and as markdown](https://spider.cloud/blog/build-fine-tuning-dataset-from-the-web/)
5. [Measured run, 2026-11-05: tokens in ten public pages as raw HTML and as markdown](https://spider.cloud/blog/what-a-web-page-costs-in-tokens/)
6. [Firecrawl pricing, read 2026-11-17](https://www.firecrawl.dev/pricing)
7. [ScrapingBee pricing, read 2026-11-17](https://www.scrapingbee.com/pricing/)
8. [Bright Data Web Unlocker pricing, pay per successful request, read 2026-11-17](https://brightdata.com/pricing/web-unlocker)
9. [Apify pricing, read 2026-09-02](https://apify.com/pricing)
10. [Crawl4AI documentation, Cloud API notice, read 2026-09-02](https://docs.crawl4ai.com/)
11. [Jina Reader, pricing and rate limits, read 2026-09-02](https://jina.ai/reader/)
12. [GitHub licenses, read 2026-11-18: spider-rs/spider MIT, firecrawl/firecrawl AGPL-3.0, unclecode/crawl4ai Apache-2.0, jina-ai/reader Apache-2.0](https://github.com/spider-rs/spider)
