Measure everything. Publish the method.
Spider Research is the lab layer of Spider Cloud: open benchmarks and field notes on web data quality for AI. Every number links to a method; every method links to a repo.
The record.
Each entry is a dated snapshot of a maintained measurement. Where a repo exists, the numbers reproduce from a clone.
- Spider Browser scores 85% on Browser Use's stealth benchmarkReproduce
Spider Browser scored 85% on Browser Use's open stealth benchmark, beating every other cloud browser tested against 80 anti-bot protected sites.
- Introducing Silk: our custom AI model for web data extraction
Silk is Spider Cloud's own extraction model. It turns raw HTML into structured data and solves captchas on our GPUs, with no external API calls and no per-token billing.
- Spider Browser vs. Kernel vs. Browserbase: 999 URLs benchmarkedReproduce
Kernel benchmarked cold start. We benchmarked reliability: 999 URLs, 254 domains, 18 categories, 100% success and 2.5s median end-to-end latency.
- Crawl4AI vs Firecrawl vs Spider Cloud: 1,000 URLs benchmarkedReproduce
Crawl4AI vs Firecrawl on 1,000 real URLs: 89.7% vs 95.3% success rate, cost per 1K pages, RAG recall@5, and the method so you can rerun it yourself.
- Rust vs. Python for web scraping: why we rewrote everything
Why Spider Cloud abandoned its Python scrapers and rebuilt in Rust: the benchmarks, the architecture decisions, and what the rewrite cost us.
- Scraping 1 million pages: what actually happens
An engineering log of crawling 1 million pages across 10,000 domains with Spider Cloud's API. Throughput curves, failure modes, cost breakdown, and lessons learned.
- The true cost of web scraping at scale
What 10K to 10M pages a month really costs on self-hosted Scrapy, Firecrawl, Apify, Crawl4AI and Spider Cloud, counting proxies, infrastructure and engineer time.
Same tasks, same scoring, every provider.
A leaderboard is only as honest as its instrument. Every Spider benchmark holds three invariants:
Every provider runs the same 71-task list against the same pages, measuring completeness under anti-bot pressure. Each row comes from its own run, not one shared session.
A task passes when the page comes back and fails when the response matches one of the block patterns. Every provider gets the same check, and no model sits in the loop.
We publish the task list and the scoring, and the runs we cite stay on this page. Clone spider-rs/benchmark and rerun the numbers yourself.
Method · spider-rs/benchmark ↗
On the bench.
- Quality Bench v1
Extraction fidelity, HTML against ground-truth markdown.
- Dataset drop
The 1,000-URL evaluation set behind the benchmark.
- Silk structure eval
Structure-conformance scoring for Silk output.
Judge the output yourself.
Post a URL, read what comes back. A failed request bills $0.