Measure everything. Publish the method.
Spider Research is the lab layer of Spider:open benchmarks and field notes on web data quality for AI. Every number links to a method; every method links to a repo.
Spider passed 80/80 tasks on Sep 18, 2026, each a real page on a protected production site. Other providers retain the March 22, 2026 readings published in the blog; they were not re-measured.
The record.
Each entry is a dated snapshot of a maintained measurement. Where a repo exists, the numbers reproduce from a clone.
- Spider Browser passes all 80 tasks on Browser Use's Browser Fidelity BenchReproduce
Spider Browser passed all 80 tasks on Browser Use's open Browser Fidelity Bench V1 in September, up from 68 of 80 in March. Every other cloud browser tested scored lower.
- Introducing Silk: our custom AI model for web data extraction
Silk is Spider's own extraction model. It turns raw HTML into structured data and handles challenge pages on our GPUs, with no external API calls and no per-token billing.
- Spider Browser vs. Kernel vs. Browserbase: 999 URLs benchmarkedReproduce
Kernel benchmarked cold start. We benchmarked reliability: 999 URLs, 254 domains, 18 categories, 100% success and 2.5s median end-to-end latency.
- Crawl4AI vs Firecrawl vs Spider: 1,000 URLs benchmarkedReproduce
Crawl4AI vs Firecrawl on 1,000 real URLs: 89.7% vs 95.3% success rate, Firecrawl pricing per 1K pages, RAG recall@5, and the method to rerun it yourself.
- Rust vs. Python for web scraping: why we rewrote everything
Why Spider abandoned its Python scrapers and rebuilt in Rust: the benchmarks, the architecture decisions, and what the rewrite cost us.
- Scraping 1 million pages: what actually happens
An engineering log of crawling 1 million pages across 10,000 domains with Spider's API. Throughput curves, failure modes, cost breakdown, and lessons learned.
- The true cost of web scraping at scale, 10K to 10M pages
What web scraping costs at 10K to 10M pages a month on self-hosted Scrapy, Firecrawl, Apify, Crawl4AI and Spider, with proxies and engineer time counted.
Same tasks, same scoring, every provider.
A leaderboard is only as honest as its instrument. Every Spider benchmark holds three invariants:
- 01 Same tasks
Every provider runs the same 80-task list against the same pages, real pages on protected production sites, and a task counts only when the page comes back complete. Each row comes from its own run, not one shared session.
- 02 Same scoring
A task passes when the page comes back and fails when the response matches one of the block patterns. Every provider gets the same check, and no model sits in the loop.
- 03 Open repo
We publish the task list and the scoring, and the runs we cite stay on this page. Clone spider-rs/benchmark and rerun the numbers yourself.
Method · spider-rs/benchmark ↗
On the bench.
- Quality Bench v1
Extraction fidelity, HTML against ground-truth markdown.
- Dataset drop
The 1,000-URL evaluation set behind the benchmark.
- Silk structure eval
Structure-conformance scoring for Silk output.
Judge the output yourself.
Post a URL, read what comes back. A failed request bills $0.