Crawl4AI vs Firecrawl vs Spider Cloud: 1,000 URLs benchmarked

Note: This benchmark is maintained at Spider Research. Results below are the February 11, 2026 snapshot.

Anyone building a RAG pipeline or an autonomous agent eventually asks which scraping tool to use. Three names keep coming up in Discord threads, on Hacker News, and in “awesome-llm” lists: Firecrawl, Crawl4AI, and Spider Cloud.

All three are open source, all three target LLM workloads, and all three promise clean data from messy pages. The marketing pages do not say how they hold up under a real workload.

If you are weighing Firecrawl vs Crawl4AI specifically, the short version from our runs: Firecrawl produced cleaner markdown and held up better on anti-bot pages, while Crawl4AI cost nothing to run and gave up the most success rate on those same pages. The full numbers are below.

So we ran one from our own infrastructure: same URLs, same hardware, same network, same measurements. Spider Cloud is our product and we designed the test, so read the numbers with that in mind. The method is below so you can rerun it on your own URLs.

Benchmark methodology

Three tools with three different architectures do not produce comparable numbers by accident. We set the test up to remove as many confounding variables as we could.

The URL corpus

We assembled 1,000 URLs split across three categories:

We shuffled the URLs once. Every tool got the same list in the same order.

Hardware and network

All three tools ran on the same machine: an AWS c6i.4xlarge (16 vCPU, 32 GB RAM) in us-east-2, connected to a 10 Gbps network. For Firecrawl and Spider, we called their cloud APIs from that instance so the latency includes the network round trip, the way an application would see it. For Crawl4AI, which has no managed cloud service, we ran the self-hosted Python process on the instance.

Spider and Firecrawl used their own proxy infrastructure for anti-bot pages. Crawl4AI ran without residential proxies (its default), since most users start without a proxy provider, so its anti-bot numbers are the out-of-the-box result, not the best possible configuration.

What we measured

Five metrics, each chosen because it maps to a real engineering concern:

1. Pages/second throughput. Total pages returned divided by wall-clock time, including retries.
2. Success rate. Percentage of URLs that returned usable content (HTTP 200 with a non-empty body).
3. Cost per 1,000 pages. Dollars spent on the cloud API (Firecrawl and Spider) or estimated compute cost (Crawl4AI self-hosted).
4. Markdown quality. Measured by downstream RAG retrieval accuracy (more on this below).
5. Time to first result. How long until the first page of content is available in your application.

For markdown quality, we fed each tool’s output into the same embedding pipeline (OpenAI text-embedding-3-small) and the same vector store (Qdrant). We then ran 200 factual questions against each corpus and measured recall@5, the percentage of questions where the correct answer appeared in the top 5 retrieved chunks. This tells you how much your retrieval quality depends on which scraper produced the markdown.

The tools

A short description of each tool first, because the architecture explains most of the performance gap.

Firecrawl

Language: TypeScript/Node.js License: AGPL-3.0 GitHub: mendableai/firecrawl Cloud: firecrawl.dev

Firecrawl is a TypeScript-based scraping API built by the Mendable team. It focuses on turning web pages into LLM-ready markdown and supports crawling (follow links), scraping (single page), and map (site discovery). The cloud service handles proxy rotation and browser rendering. The open source version is AGPL-3.0 and self-hosted, so you bring your own infrastructure and own the browser and proxy layer yourself.

Cloud pricing starts at $16/month (annual billing) for 3,000 credits (pages), scaling to $599/month for 1,000,000 credits. Credits expire monthly.

Crawl4AI

Language: Python (asyncio) License: Apache 2.0 GitHub: unclecode/crawl4ai Cloud: None (self-hosted only)

Crawl4AI is a Python async crawling framework built for AI data pipelines. It uses Playwright under the hood for browser rendering and provides markdown conversion, chunking strategies, and extraction helpers. It is completely free, with no cloud service and no usage-based pricing.

The trade-off: you run everything yourself. That means provisioning the browser instances, managing proxies, handling retries, and scaling horizontally when throughput matters.

Spider

Language: Rust License: MIT GitHub: spider-rs/spider Cloud: spider.cloud

Spider’s core engine is written in Rust. The cloud API handles proxy rotation, anti-bot bypass, browser rendering, and markdown conversion. The open source crate (MIT licensed) runs standalone if you prefer self-hosting. Cloud pricing is pay-as-you-go with no subscription.

Results

Throughput (pages per second)

This is the headline number. How fast can each tool move through a large URL list?

Spider’s Rust engine processes static pages at nearly 7x the throughput of Firecrawl and 9.5x Crawl4AI. The gap narrows on JS-heavy pages (where the browser is the bottleneck, not the framework), but Spider still leads by 3-4x because its smart mode skips browser rendering entirely for pages that don’t require JavaScript execution.

The 74 pages/s corpus average is specific to this test’s URL mix. 25% of the corpus is heavily anti-bot protected sites (Amazon, Nike, Walmart, Zillow, Bloomberg) that slow all three tools down. Throughput also scales with concurrency. On workloads that are mostly static or JS-rendered content, Spider sustains 182+ pages/s at the concurrency level used in this benchmark. Production workloads with higher concurrency and fewer anti-bot targets will see higher throughput.

On anti-bot protected pages, all three tools slow down. Spider handles sites behind common bot-protection services without extra configuration. Firecrawl requires the cloud tier for reliable bypass. Crawl4AI leaves anti-bot handling to the user.

Success rate

The anti-bot tier is where the gap is widest. Spider’s integrated proxy rotation and fingerprint management kept the failure rate under 1%. Crawl4AI, running without residential proxies (its default configuration), dropped 28% of the anti-bot URLs. Adding a proxy provider would improve Crawl4AI’s numbers here, but that’s additional setup and cost. These failures cascade in production: a missing page means a missing chunk in your vector store.

Cost per 1,000 pages

Cost figures updated as of September 2026; the benchmark results above are unchanged from the February run. Spider bills bandwidth at $1 per GB and compute at $0.001 per CPU minute with no credit multipliers, and failed requests cost nothing. The pricing guide puts a mixed workload at about $8 to $15 per 100,000 pages in smart mode, which is $0.08 to $0.15 per 1,000. Firecrawl’s effective cost depends on which plan tier you’re on. At the Standard tier ($83 per month billed yearly for 100,000 credits), it’s $0.83 per 1K, which is competitive. At the Hobby tier ($16 per month billed yearly for 5,000 credits), it’s $3.20 per 1K (Firecrawl pricing, as of September 2026).

Crawl4AI is free to use. The hidden cost is engineering time: you build and maintain the proxy layer, the retry logic, the scaling infrastructure, and the monitoring. For teams that have that capacity, it is a legitimate option. For teams that don’t, the “free” label is misleading.

Markdown quality (RAG retrieval accuracy)

This metric matters more than most benchmarks acknowledge. If the markdown is noisy (nav bars, cookie banners, footer links, boilerplate), your embeddings carry that noise, and retrieval quality degrades.

All three produce usable markdown. The differences are at the margins, but margins compound. A 7-point gap in recall@5 means your users get wrong or incomplete answers roughly 1 in 14 queries more often with Crawl4AI output than with Spider output. Over thousands of daily queries, that adds up.

Firecrawl’s markdown is good. It handles article content well and strips most boilerplate. Spider edges it out on noisier pages (e-commerce, forums, documentation with heavy sidebars) where the Rust parser’s content extraction heuristics are more aggressive.

Crawl4AI provides knobs to tune extraction (CSS selectors for exclusion, custom chunking), but the defaults are more permissive. Teams willing to spend time configuring per-domain rules can close the gap.

Time to first result

Spider returns the first static page result in under 50 milliseconds. For interactive applications (chatbots that fetch context on demand, agents that browse in real time), this compounds across dozens of sequential requests in a single workflow.

The gap on static pages is almost entirely architectural. Spider’s HTTP client is compiled Rust. Firecrawl’s Node.js runtime and Crawl4AI’s Python asyncio loop both add overhead before the first byte is even processed.

Summary table

Code comparison

The same operation (scrape a URL, get markdown) in all three tools.

Spider (Python SDK)

import requests
import os

response = requests.post(
    "https://api.spider.cloud/crawl",
    headers={
        "Authorization": f"Bearer {os.getenv('SPIDER_API_KEY')}",
        "Content-Type": "application/json",
    },
    json={
        "url": "https://example.com",
        "limit": 10,
        "return_format": "markdown",
        "request": "smart",
    },
)

for page in response.json():
    print(f"{page['url']}: {len(page['content'])} chars")

Spider (Rust crate, self-hosted)

use spider::website::Website;
use spider::configuration::Configuration;

#[tokio::main]
async fn main() {
    let mut config = Configuration::new();
    config.with_limit(10);
    config.with_return_page_links(true);

    let mut website = Website::new("https://example.com")
        .with_configuration(config)
        .build()
        .unwrap();

    website.crawl().await;

    for page in website.get_pages().unwrap().iter() {
        let markdown = page.to_markdown();
        println!("{}: {} chars", page.get_url(), markdown.len());
    }
}

Firecrawl (Python SDK)

from firecrawl import FirecrawlApp

app = FirecrawlApp(api_key="fc-YOUR_API_KEY")

result = app.crawl_url(
    "https://example.com",
    params={
        "limit": 10,
        "scrapeOptions": {
            "formats": ["markdown"],
        },
    },
    poll_interval=2,
)

for page in result.get("data", []):
    print(f"{page['metadata']['url']}: {len(page.get('markdown', ''))} chars")

Crawl4AI (Python, self-hosted)

import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://example.com")
        print(f"{result.url}: {len(result.markdown)} chars")

asyncio.run(main())

Note that Crawl4AI’s arun processes a single URL. Crawling multiple pages with link following requires additional code to manage the URL frontier, deduplication, and concurrency. Spider and Firecrawl handle this with a single limit parameter.

What the benchmarks don’t show

A table misses a few things that matter once you are running in production.

Maintenance burden

Spider is a managed service with a Rust core. You send API requests and get results. The platform handles proxy management, browser pools, anti-bot bypass, and retries. The open source crate is a single binary with no runtime dependencies beyond libc.

Firecrawl has a managed cloud, but self-hosting means running Redis, a Node.js server, Playwright browsers, and optionally a separate worker process for async jobs. That’s multiple moving parts to keep healthy.

Crawl4AI is self-hosted only. You own everything: browser lifecycle management, proxy rotation, error handling, horizontal scaling, and monitoring. For a prototype or a research project, this is fine. For a production service processing millions of pages, it is a significant engineering commitment.

Scaling pain

Scaling a scraping workload from 100 pages to 100,000 pages is where architectural choices become obvious.

Spider’s Rust engine was designed for this from the start. The async runtime (tokio) handles tens of thousands of concurrent connections with predictable memory usage. The cloud API scales horizontally behind the scenes.

Firecrawl’s Node.js runtime handles concurrency well at moderate scale, but memory usage grows with each browser context. The cloud service manages this for you; self-hosted requires careful tuning.

Crawl4AI’s Python/asyncio model works for hundreds of concurrent requests. Beyond that, you hit Python’s GIL limitations and Playwright’s browser memory overhead. Scaling to tens of thousands of concurrent connections means running multiple processes behind a task queue, which you build and maintain yourself.

Community and ecosystem

Spider has a growing community around the Rust crate. SDKs exist for Python, JavaScript, Rust, Go, and a CLI. The MIT license means no restrictions on commercial use or derivative works.

Firecrawl has 175,000+ GitHub stars and an active community. The AGPL-3.0 license matters: if you modify the source and offer it as a service, you must release your changes. For many companies, this means the cloud API is the only practical option.

Crawl4AI has 80,000+ GitHub stars and an active community building extraction strategies and sharing configurations. The Apache 2.0 license is permissive for commercial use. A managed cloud API is in closed beta, which may change the operational picture.

Open source licensing matters

The license on your scraping layer affects your entire stack.

If you’re building a commercial product that incorporates a scraping tool, the AGPL clause on Firecrawl is worth discussing with your legal team. Spider’s MIT and Crawl4AI’s Apache 2.0 carry no such obligation.

Where each tool shines

No tool is the best choice for every scenario.

Firecrawl is a good choice when you want a polished cloud API with solid markdown and don’t need maximum throughput. The documentation is thorough and the crawl, scrape, and map endpoints are well designed. If your workload is under 50,000 pages/month and you value simplicity over speed, it does the job.

Crawl4AI is a good choice when you’re prototyping an AI pipeline on a budget, need full control over the extraction logic, or are doing research where cost must be zero. The Python ecosystem means you can plug it directly into your ML workflow without crossing language boundaries. If you have the engineering capacity to run infrastructure, it holds up.

Spider Cloud is the right choice when throughput, cost, and reliability at scale decide it. Workloads in the hundreds of thousands or millions of pages, latency that users notice, one API call that handles anti-bot, rendering, and markdown, or a need for the MIT license: that is what it was built for.

Conclusion

On our test corpus, Spider Cloud led on throughput, cost, and success rate. Firecrawl produced good markdown and offers a polished developer experience. Crawl4AI is free and flexible for teams with the engineering capacity to run it.

These numbers reflect a specific URL list run on a specific day. Anti-bot configurations change, cloud API performance fluctuates, and your target sites will behave differently from ours. Run the benchmark yourself with your actual production URLs before making a decision.