Measure everything. Publish the method.
Spider Research is the lab layer of Spider: open benchmarks and field notes on web data quality for AI. Every number links to a method; every method links to a repo.
The record.
Each entry is a dated snapshot of a maintained measurement. Where a repo exists, the numbers reproduce from a clone.
- Spider Browser Scores 85% on Browser Use's Stealth BenchmarkReproduce
Browser Use open-sourced a stealth benchmark testing cloud browsers against 80 anti-bot protected sites. We ran it with Spider Browser and scored 85%.
- Introducing Silk: Our Custom AI Model for Web Data Extraction
Spider runs Silk, a purpose-built extraction model that converts raw HTML into structured data and solves captchas on dedicated GPU infrastructure. No external API calls, no per-token billing, no data leaving our network.
- Spider Browser vs. Kernel vs. Browserbase: 999 URLs, 100% Pass RateReproduce
Kernel benchmarked cold start speed. We benchmarked what matters: reliability across 999 URLs, 254 domains, and 18 categories, with a 100% success rate and 2.5s median end-to-end latency.
- Crawl4AI vs. Firecrawl vs. Spider: Honest BenchmarkReproduce
Crawl4AI vs. Firecrawl vs. Spider, benchmarked on 1,000 real URLs: see who wins on throughput, success rate, cost, and markdown quality. Reproduce it yourself.
- Rust vs. Python for Web Scraping: Why We Rewrote Everything
The engineering story behind Spider's decision to abandon Python scrapers and rebuild from scratch in Rust. Concrete benchmarks, architecture decisions, and lessons learned.
- Scraping 1 Million Pages: What Actually Happens
An engineering log of crawling 1 million pages across 10,000 domains with Spider's cloud API. Throughput curves, failure modes, cost breakdown, and lessons learned.
- The True Cost of Web Scraping at Scale
A detailed cost breakdown of web scraping at 10K to 10M pages per month, comparing self-hosted Scrapy, Firecrawl, Apify, Crawl4AI, and Spider across infrastructure, proxies, engineering time, and total cost of ownership.
Stealth Bench V1.
Completeness under anti-bot pressure — the same tasks, the same judge, every provider. Clone the repo and rerun it.
Method · spider-rs/benchmark ↗
Next · Quality Bench v1 — extraction fidelity, HTML → ground-truth markdown · 1,000-URL dataset drop · Silk structure-conformance eval
Judge the output yourself.
Post a URL, read what comes back. A failed request bills $0.