Measure everything. Publish the method.
Spider Research is the lab layer of Spider:open benchmarks and field notes on web data quality for AI. Every number links to a method; every method links to a repo.
Spider passed 80/80 tasks on Sep 18, 2026, each a real page on a protected production site. Other providers retain the March 22, 2026 readings published in the blog; they were not re-measured.
The record.
Each entry is a dated snapshot of a maintained measurement. Where a repo exists, the numbers reproduce from a clone.
- What a web page costs in tokens, measured on ten public pages
Raw HTML, markdown, readability and text output for the same ten pages, counted in o200k tokens. Markdown cut the bill 15x at the median, and readability mode left four index pages with under 60 tokens.
- Spider Browser passes 80/80 on Browser Fidelity BenchReproduce
Spider Browser passed all 80 tasks on Browser Use's Browser Fidelity Bench V1 in September, up from 68 in March. Every other cloud browser tested scored lower.
- Introducing Silk: our custom AI model for web data extraction
Silk is Spider's own extraction model. It turns raw HTML into structured data and handles challenge pages on our GPUs, with no external API calls and no per-token billing.
- Spider Browser vs. Kernel vs. Browserbase: 999 URLs benchmarkedReproduce
Kernel benchmarked cold start. We benchmarked reliability: 999 URLs, 254 domains, 18 categories, 100% success and 2.5s median end-to-end latency.
- Crawl4AI vs Firecrawl vs Spider: a 1,000 URL benchmarkReproduce
Crawl4AI vs Firecrawl on 1,000 real URLs. Firecrawl won success rate, 95.3% to 89.7%, and RAG recall@5, 89.0% to 84.5%. Crawl4AI is Apache 2.0 and self-hosted.
- Rust vs. Python for web scraping: why we rewrote everything
Why Spider abandoned its Python scrapers and rebuilt in Rust: the benchmarks, the architecture decisions, and what the rewrite cost us.
- Scraping 1 million pages: what actually happens
An engineering log of crawling 1 million pages across 10,000 domains with Spider's API. Throughput curves, failure modes, cost breakdown, and lessons learned.
- Web scraping cost per 1,000 pages, by kind of page
Day 1 of a seven-day run: web scraping cost $0.27 per 1,000 docs pages and $1.48 for reference, plus the bill at 10 million pages a month.
Same tasks, same scoring, every provider.
A leaderboard is only as honest as its instrument. Every Spider benchmark holds three invariants:
- 01 Same tasks
Every provider runs the same 80-task list against the same pages, real pages on protected production sites, and a task counts only when the page comes back complete. Each row comes from its own run, not one shared session.
- 02 Same scoring
A task passes when the page comes back and fails when the response matches one of the block patterns. Every provider gets the same check, and no model sits in the loop.
- 03 Open repo
We publish the task list and the scoring, and the runs we cite stay on this page. Clone spider-rs/benchmark and rerun the numbers yourself.
Method · spider-rs/benchmark ↗
On the bench.
- Quality Bench v1
Extraction fidelity, HTML against ground-truth markdown.
- Dataset drop
The 1,000-URL evaluation set behind the benchmark.
- Silk structure eval
Structure-conformance scoring for Silk output.
Judge the output yourself.
Post a URL, read what comes back. A blocked page usually costs a fraction of a cent.