Notes from the engineering team.
Writeups, benchmarks, and field notes on web data collection and AI infrastructure.
Latest 1 month ago
Flat-rate web scraping vs. pay-as-you-go: which to put where
How the flat-rate Unlimited plan puts steady high-volume crawling on a fixed bill, where pay-as-you-go still wins, and how both run on one API key.
Archive
How to scrape the web at scale from your terminal
Every crawl mode the Spider CLI supports, with real examples, and how to go from one page to millions without leaving the terminal.
Spider Browser scores 85% on Browser Use's stealth benchmark
Spider Browser scored 85% on Browser Use's open stealth benchmark, beating every other cloud browser tested against 80 anti-bot protected sites.
Real-time web search for RAG: stop feeding your LLM stale data
Static document stores go stale in days. Add live web search to a RAG pipeline so the model answers with current data, in LangChain and in plain Python.
Web search API for AI agents: search and extract in one call
Stitching a SERP API, a scraper and a parser together is fragile and slow. Spider's Search API does all three in one request, which is what agents need.
Introducing Silk: our custom AI model for web data extraction
Silk is Spider's own extraction model. It turns raw HTML into structured data and solves captchas on our GPUs, with no external API calls and no per-token billing.
Case study: how a RAG pipeline went from 6 hours to 15 minutes
A Series A AI company replaced three Python microservices, a proxy provider, and half an engineer's time with a single Spider API call. Here's exactly what changed.
The 7 best web scraping APIs for AI in 2026
Spider, Firecrawl, Crawl4AI, ScrapingBee, Apify, Bright Data and Jina Reader compared on real pricing and benchmarks, with the trade-off each one asks you to accept.
Spider vs. Oxylabs: JavaScript rendering & structured data
Does the Oxylabs Web Scraper API handle JavaScript-heavy sites and return structured data without your own browsers? Rendering, parsers and pricing compared.
Spider vs. Apify: compute units, expired credits, real cost
Apify's compute unit folds memory, time and proxy bandwidth into one formula most teams cannot predict. Spider bills bandwidth plus compute, no expiring credits.
Spider vs. Bright Data: enterprise stack vs. a single API
Bright Data operates the largest proxy network in the world and sells six separate scraping products. Spider does the same job through one API with no minimum spend.
Spider vs. ZenRows: credit multipliers and real cost per page
A 25x multiplier for JS rendering plus premium proxies turns 250,000 ZenRows credits into 10,000 requests. Spider has no multipliers and no expiring credits.
Spider vs. Crawl4AI: managed API vs. self-hosted Python
Spider's managed Rust API against Crawl4AI's free Python framework: benchmarks, total cost of ownership, and when each one is the right call.
Firecrawl pricing and MCP crawl speed vs Spider, benchmarked
Firecrawl's plans priced against Spider's usage billing, plus the crawl speeds and success rates both MCP servers hit on the same 1,000 URLs.
Spider vs. Jina Reader: full crawling vs. URL-to-markdown
Jina Reader turns one URL into markdown from a prefix. Spider crawls whole sites with proxy rotation and anti-bot bypass. Where each fits, and what it costs.
Spider vs. ScrapFly: credit multipliers vs. transparent pricing
ScrapFly's credit multipliers make cost hard to predict. Spider charges flat bandwidth plus compute. A look at the pricing math behind credit-based APIs.
Spider vs. Crawlera (Zyte): pricing and full browser control
Migrating from Crawlera? Zyte's complexity tiers make per-request costs unpredictable. Spider bills flat bandwidth plus compute, with full browser control.
NetNut alternatives: why a proxy network alone isn't enough
NetNut sells proxy bandwidth, not scraping. Most NetNut alternatives stop there too. Spider handles TLS, rendering, challenges and extraction in one API.
ScraperAPI free plan and pricing: credits, limits, costs
ScraperAPI's free plan is 1,000 credits a month, and one protected page can cost 75 of them. See what the free tier and each paid plan really cost.
Spider vs. ScrapingBee: pricing without credit multipliers
ScrapingBee charges up to 75 credits per request with its stealth multiplier. Spider bills bandwidth plus compute, with browser automation and AI extraction.
Spider Browser vs. Kernel vs. Browserbase: 999 URLs benchmarked
Kernel benchmarked cold start. We benchmarked reliability: 999 URLs, 254 domains, 18 categories, 100% success and 2.5s median end-to-end latency.
Spider MCP v2: browser automation for AI agents
Spider's MCP server now ships 22 tools, 9 of them browser automation, giving agents direct control of cloud browsers with anti-bot bypass and proxy rotation.
Build a production RAG pipeline with web data in under 30 minutes
Crawl a site with Spider, chunk the markdown, embed it, store it in a vector database and query it. Working code for LangChain, LlamaIndex, CrewAI and AutoGen.
Building AI agents that browse the web
Architecture patterns and working code for web-browsing agents: research, monitoring and extraction, built on CrewAI and AutoGen with Spider as the backend.
Building an MCP server for web scraping
Build an MCP server in TypeScript that wraps Spider's API, giving any AI model the ability to crawl, scrape, search and extract structured data.
How to bypass Cloudflare, DataDome, and PerimeterX in 2026
A technical breakdown of how modern anti-bot systems detect scrapers, why manual bypass is unsustainable, and how Spider handles it automatically.
The developer's guide to choosing a scraping stack in 2026
DIY libraries, open source frameworks, managed APIs, AI-native extractors and browser automation, with a decision matrix and a hidden-cost audit for each.
Crawl4AI vs Firecrawl vs Spider: 1,000 URLs benchmarked
Crawl4AI vs Firecrawl on 1,000 real URLs: 89.7% vs 95.3% success rate, cost per 1K pages, RAG recall@5, and the method so you can rerun it yourself.
Open source web scraping: why MIT license matters
How MIT, Apache 2.0, AGPL and BSL affect your right to build commercial products on scraping tools, and why Spider picked MIT.
Rust vs. Python for web scraping: why we rewrote everything
Why Spider abandoned its Python scrapers and rebuilt in Rust: the benchmarks, the architecture decisions, and what the rewrite cost us.
Scraping 1 million pages: what actually happens
An engineering log of crawling 1 million pages across 10,000 domains with Spider's cloud API. Throughput curves, failure modes, cost breakdown, and lessons learned.
How Spider went to market: what worked and what we'd redo
How we built Spider's go-to-market from zero: the channels that worked, the pricing mistakes, and the content that actually converted.
Top 5 data collection platforms for AI and web scraping in 2026
The leading data collection platforms compared on cost, speed, reliability and AI readiness, for teams building RAG pipelines and agents.
The true cost of web scraping at scale
What 10K to 10M pages a month really costs on self-hosted Scrapy, Firecrawl, Apify, Crawl4AI and Spider, counting proxies, infrastructure and engineer time.
From web page to vector database: the complete pipeline
The full pipeline from raw URL to queryable vector store: crawling, extraction, chunking, embedding and indexing, with working code and a cost breakdown.
Web scraping for AI training data: legal and technical guide 2026
What the EU AI Act, US fair use, and GDPR actually require of AI training data, and how to build a scraping pipeline that holds up in 2026.
Start crawling in 30 seconds.
One API key. No servers to manage.
Free balance on signup · No card required