# Spider Blog - Web Data, Crawling and AI Engineering

Engineering writeups, benchmarks, and field notes on web data collection and AI infrastructure, from the team that builds Spider.

Canonical: https://spider.cloud/blog/

## Notes from the engineering team.

Writeups, benchmarks, and field notes on web data collection and AI infrastructure.

## Latest post

[Fetching the web is the cheap part of an agent's research bill](https://spider.cloud/blog/ai-agent-research-bill/)

[We priced the fetch and the page input for five agent research tasks on 2026-11-18. Fetching ten pages was the small line. Reading them as raw HTML took 6.5x to 22.5x the tokens of markdown on four tasks and 1.8x on the fifth.](https://spider.cloud/blog/ai-agent-research-bill/)

[Latest Nov 18, 2026 3 days ago](https://spider.cloud/blog/ai-agent-research-bill/)

[AI · cost-analysis · benchmarks Read](https://spider.cloud/blog/ai-agent-research-bill/)

## Archive

[Nov 17, 2026](https://spider.cloud/blog/how-to-scrape-a-website/)

### [How to scrape a website, step by step](https://spider.cloud/blog/how-to-scrape-a-website/)

[How to scrape a website, from reading robots.txt to saving a crawl one page at a time, with the curl command for each step and what it returned.](https://spider.cloud/blog/how-to-scrape-a-website/)

[web-scraping · guides ](https://spider.cloud/blog/how-to-scrape-a-website/)[Nov 17, 2026](https://spider.cloud/blog/proxies-for-web-scraping/)

### [Proxies for web scraping, built into every request](https://spider.cloud/blog/proxies-for-web-scraping/)

[Proxies for web scraping come built into every Spider request and rotate on their own. What they cost, and how to pick the exit country or use your own.](https://spider.cloud/blog/proxies-for-web-scraping/)

[web-scraping · engineering · benchmarks ](https://spider.cloud/blog/proxies-for-web-scraping/)[Nov 17, 2026](https://spider.cloud/blog/web-scraping-for-ai-agents/)

### [Web scraping for AI agents: search, scrape or crawl](https://spider.cloud/blog/web-scraping-for-ai-agents/)

[Web scraping for AI agents: when an agent should search, scrape or crawl, and how many tokens of its context window each call fills, counted on real pages.](https://spider.cloud/blog/web-scraping-for-ai-agents/)

[web-scraping · ai-agents · engineering ](https://spider.cloud/blog/web-scraping-for-ai-agents/)[Nov 17, 2026](https://spider.cloud/blog/web-scraping-to-markdown/)

### [Web scraping to markdown: a whole docs site into files](https://spider.cloud/blog/web-scraping-to-markdown/)

[Web scraping to markdown with one crawl call and a short Python script that writes a file per page, then a check of the code blocks and tables it kept.](https://spider.cloud/blog/web-scraping-to-markdown/)

[web-scraping · engineering · benchmarks ](https://spider.cloud/blog/web-scraping-to-markdown/)[Nov 17, 2026](https://spider.cloud/blog/web-scraping-with-a-headless-browser/)

### [Web scraping with a headless browser, without running one](https://spider.cloud/blog/web-scraping-with-a-headless-browser/)

[How to scrape pages that build their text with JavaScript, using a browser on Spider's side, and how to tell which pages need one and what a render costs.](https://spider.cloud/blog/web-scraping-with-a-headless-browser/)

[web-scraping · guides ](https://spider.cloud/blog/web-scraping-with-a-headless-browser/)[Nov 17, 2026](https://spider.cloud/blog/web-scraping-with-javascript/)

### [Web scraping with JavaScript and Node.js](https://spider.cloud/blog/web-scraping-with-javascript/)

[Web scraping with JavaScript: a Node.js script with one package that scrapes a page as markdown, pulls a field by CSS selector and streams a crawl to a file.](https://spider.cloud/blog/web-scraping-with-javascript/)

[web-scraping · engineering · javascript ](https://spider.cloud/blog/web-scraping-with-javascript/)[Nov 16, 2026](https://spider.cloud/blog/free-web-scraper/)

### [Free web scraper: three ways to scrape a page without paying](https://spider.cloud/blog/free-web-scraper/)

[Scrape a page for free three ways: in the browser with no key, from curl with no key, or with the MIT-licensed spider CLI on your own machine.](https://spider.cloud/blog/free-web-scraper/)

[web-scraping · tutorial · open-source ](https://spider.cloud/blog/free-web-scraper/)[Nov 16, 2026](https://spider.cloud/blog/how-web-scraping-works/)

### [How web scraping works: what a plain fetch gets and misses](https://spider.cloud/blog/how-web-scraping-works/)

[How web scraping works, from the HTTP request to the markdown you keep, and how to tell when a page needs a browser to run its scripts first.](https://spider.cloud/blog/how-web-scraping-works/)

[web-scraping · engineering · benchmarks ](https://spider.cloud/blog/how-web-scraping-works/)[Nov 16, 2026](https://spider.cloud/blog/is-web-scraping-legal-robots-txt-survey/)

### [Is web scraping legal? What 67 top sites ask for in robots.txt](https://spider.cloud/blog/is-web-scraping-legal-robots-txt-survey/)

[Is web scraping legal? US and EU rulings and statutes by name and year, beside the robots.txt of 67 top sites. 13 hold Disallow: / for unnamed crawlers.](https://spider.cloud/blog/is-web-scraping-legal-robots-txt-survey/)

[web-scraping · legal · benchmarks ](https://spider.cloud/blog/is-web-scraping-legal-robots-txt-survey/)[Nov 16, 2026](https://spider.cloud/blog/web-scraping-vs-web-crawling/)

### [Web scraping vs web crawling: when to crawl and when to scrape](https://spider.cloud/blog/web-scraping-vs-web-crawling/)

[The difference between web scraping and web crawling, timed on one docs site, and which call to use for a URL list, a whole site or only a question.](https://spider.cloud/blog/web-scraping-vs-web-crawling/)

[web-scraping · engineering · benchmarks ](https://spider.cloud/blog/web-scraping-vs-web-crawling/)[Nov 16, 2026](https://spider.cloud/blog/web-scraping-with-python/)

### [Web scraping with Python using only the standard library](https://spider.cloud/blog/web-scraping-with-python/)

[A Python web scraper that uses only the standard library and gets each page back as markdown. The whole script, the pages we ran it on and what it returned.](https://spider.cloud/blog/web-scraping-with-python/)

[web-scraping · engineering · python ](https://spider.cloud/blog/web-scraping-with-python/)[Nov 16, 2026](https://spider.cloud/blog/what-is-a-web-scraping-api/)

### [What is a web scraping API? One request, read field by field](https://spider.cloud/blog/what-is-a-web-scraping-api/)

[What a web scraping API does between your request and its answer, why a 200 call can carry a 404 page, and five things to check in any scraping API.](https://spider.cloud/blog/what-is-a-web-scraping-api/)

[web-scraping · engineering · API ](https://spider.cloud/blog/what-is-a-web-scraping-api/)[Nov 6, 2026](https://spider.cloud/blog/build-fine-tuning-dataset-from-the-web/)

### [Build a fine-tuning dataset from the web in two API calls](https://spider.cloud/blog/build-fine-tuning-dataset-from-the-web/)

[One search call and one crawl call turned a topic into 396 training records for $0.05. Every number comes from a run on 60 public documentation pages.](https://spider.cloud/blog/build-fine-tuning-dataset-from-the-web/)

[engineering · AI · tutorials ](https://spider.cloud/blog/build-fine-tuning-dataset-from-the-web/)[Nov 5, 2026](https://spider.cloud/blog/what-a-web-page-costs-in-tokens/)

### [What a web page costs in tokens, measured on ten public pages](https://spider.cloud/blog/what-a-web-page-costs-in-tokens/)

[Raw HTML, markdown, readability and text output for the same ten pages, counted in o200k tokens. Markdown cut the bill 15x at the median, and readability mode left four index pages with under 60 tokens.](https://spider.cloud/blog/what-a-web-page-costs-in-tokens/)

[engineering · AI · benchmarks ](https://spider.cloud/blog/what-a-web-page-costs-in-tokens/)[Oct 29, 2026](https://spider.cloud/blog/spider-peers-share-bandwidth/)

### [Spider Peers: earn from the internet you already pay for](https://spider.cloud/blog/spider-peers-share-bandwidth/)

[Install Spider Peers on a computer, phone or TV and earn from your spare home bandwidth. Set it up once and it runs in the background.](https://spider.cloud/blog/spider-peers-share-bandwidth/)

[peers · bandwidth-sharing ](https://spider.cloud/blog/spider-peers-share-bandwidth/)[Oct 29, 2026](https://spider.cloud/blog/spider-shield-free-blocker/)

### [Spider Shield: a free blocker for ads, trackers, phishing and malware](https://spider.cloud/blog/spider-shield-free-blocker/)

[Spider Shield blocks ads, trackers, phishing and malware in every app on Mac, Windows, Linux and Android. It's free and needs no account.](https://spider.cloud/blog/spider-shield-free-blocker/)

[shield · privacy · security ](https://spider.cloud/blog/spider-shield-free-blocker/)[Sep 2, 2026](https://spider.cloud/blog/best-firecrawl-alternatives/)

### [Firecrawl alternatives: Spider at 99.9% success, 4.6x throughput](https://spider.cloud/blog/best-firecrawl-alternatives/)

[Spider finished 1,000 URLs at 99.9% success to Firecrawl's 95.7%, with 4.6x the throughput. It costs from $0.16 per 1,000 pages. Cost varies by site.](https://spider.cloud/blog/best-firecrawl-alternatives/)

[comparisons · alternatives · web-scraping ](https://spider.cloud/blog/best-firecrawl-alternatives/)[Jul 20, 2026](https://spider.cloud/blog/flat-rate-web-scraping-unlimited-vs-pay-as-you-go/)

### [Flat-rate web scraping vs. pay-as-you-go: which to put where](https://spider.cloud/blog/flat-rate-web-scraping-unlimited-vs-pay-as-you-go/)

[How the flat-rate Unlimited plan puts steady high-volume crawling on a fixed bill, where pay-as-you-go still wins, and how both run on one API key.](https://spider.cloud/blog/flat-rate-web-scraping-unlimited-vs-pay-as-you-go/)

[web-scraping · cost-analysis · pricing ](https://spider.cloud/blog/flat-rate-web-scraping-unlimited-vs-pay-as-you-go/)[Apr 1, 2026](https://spider.cloud/blog/spider-cli-web-scraping-at-scale/)

### [Command line web scraping: 100 pages in about 3 seconds](https://spider.cloud/blog/spider-cli-web-scraping-at-scale/)

[Three Spider CLI runs each crawled 100 FastAPI docs pages to markdown in about 3 seconds. Install it, add a key, then pipe the JSON to jq or write a WARC file.](https://spider.cloud/blog/spider-cli-web-scraping-at-scale/)

[tutorial · cli · web-scraping ](https://spider.cloud/blog/spider-cli-web-scraping-at-scale/)[Mar 22, 2026](https://spider.cloud/blog/spider-browser-stealth-benchmark/)

### [Spider Browser passes 80/80 on Browser Fidelity Bench](https://spider.cloud/blog/spider-browser-stealth-benchmark/)

[Spider Browser passed all 80 tasks on Browser Use's Browser Fidelity Bench V1 in September, up from 68 in March. Every other cloud browser tested scored lower.](https://spider.cloud/blog/spider-browser-stealth-benchmark/)

[benchmarks · browser-fidelity · browser-automation ](https://spider.cloud/blog/spider-browser-stealth-benchmark/)[Mar 16, 2026](https://spider.cloud/blog/real-time-web-search-for-rag-pipelines/)

### [Keep RAG context fresh with real-time web search](https://spider.cloud/blog/real-time-web-search-for-rag-pipelines/)

[Static knowledge bases go stale in days. Build a RAG pipeline on live web data by adding web search at query time, with LangChain and plain Python examples.](https://spider.cloud/blog/real-time-web-search-for-rag-pipelines/)

[AI · RAG · search ](https://spider.cloud/blog/real-time-web-search-for-rag-pipelines/)[Mar 16, 2026](https://spider.cloud/blog/web-search-api-for-ai-agents/)

### [Web search API for AI agents: search and extract in one call](https://spider.cloud/blog/web-search-api-for-ai-agents/)

[Spider's Search API searches, crawls and cleans in one request, so one agent covers live web search, deep research and page monitoring without glue code.](https://spider.cloud/blog/web-search-api-for-ai-agents/)

[AI · search · agents ](https://spider.cloud/blog/web-search-api-for-ai-agents/)[Mar 11, 2026](https://spider.cloud/blog/spider-custom-ai-extraction-model/)

### [Introducing Silk: our custom AI model for web data extraction](https://spider.cloud/blog/spider-custom-ai-extraction-model/)

[Silk, Spider's own extraction model, turns raw HTML into structured data and handles challenge pages on our GPUs, with no outside API calls or per-token billing.](https://spider.cloud/blog/spider-custom-ai-extraction-model/)

[engineering · AI · extraction ](https://spider.cloud/blog/spider-custom-ai-extraction-model/)[Mar 8, 2026](https://spider.cloud/blog/case-study-rag-pipeline-spider-replaced-three-microservices/)

### [Case study: how a RAG pipeline went from 6 hours to 15 minutes](https://spider.cloud/blog/case-study-rag-pipeline-spider-replaced-three-microservices/)

[A Series A AI company replaced three Python microservices, a proxy provider, and half an engineer's time with a single Spider API call. This is what changed.](https://spider.cloud/blog/case-study-rag-pipeline-spider-replaced-three-microservices/)

[case-study · RAG · AI ](https://spider.cloud/blog/case-study-rag-pipeline-spider-replaced-three-microservices/)[Feb 25, 2026](https://spider.cloud/blog/best-web-scraping-apis-for-ai-2026/)

### [Best web scraping APIs for AI and LLM pipelines in 2026](https://spider.cloud/blog/best-web-scraping-apis-for-ai-2026/)

[On 60 docs pages, markdown took 3,369 tokens a page and raw HTML 123,813. Seven web scraping APIs for RAG and LLMs, with 100,000-page costs where one is listed.](https://spider.cloud/blog/best-web-scraping-apis-for-ai-2026/)

[comparisons · web-scraping · AI ](https://spider.cloud/blog/best-web-scraping-apis-for-ai-2026/)[Feb 25, 2026](https://spider.cloud/blog/spider-vs-oxylabs/)

### [Oxylabs Web Scraper API pricing by plan, compared to Spider](https://spider.cloud/blog/spider-vs-oxylabs/)

[Oxylabs lists unrendered general sites at $1.00 per 1,000 results on Starter, $0.75 on Enterprise. Spider: from $0.16 per 1,000 static pages on Nov 17, varies by site.](https://spider.cloud/blog/spider-vs-oxylabs/)

[comparisons · web-scraping · alternatives ](https://spider.cloud/blog/spider-vs-oxylabs/)[Feb 20, 2026](https://spider.cloud/blog/spider-alternative-to-apify/)

### [Apify alternative 2026: Spider vs Apify pricing and CUs](https://spider.cloud/blog/spider-alternative-to-apify/)

[What each Apify plan buys in compute units, why proxy bandwidth and Actor memory push the bill higher, what Spider costs per kind of page, and a migration example.](https://spider.cloud/blog/spider-alternative-to-apify/)

[comparisons · web-scraping · alternatives ](https://spider.cloud/blog/spider-alternative-to-apify/)[Feb 20, 2026](https://spider.cloud/blog/spider-alternative-to-brightdata/)

### [From $0.16 per 1,000 pages: a Bright Data alternative](https://spider.cloud/blog/spider-alternative-to-brightdata/)

[Spider: from $0.16 per 1,000 pages, Nov 17 static pages. Cost varies by site. Bright Data Web Unlocker list price, Nov 17, 2026: $1.50 pay as you go, $1.00 on Scale.](https://spider.cloud/blog/spider-alternative-to-brightdata/)

[comparisons · web-scraping · alternatives ](https://spider.cloud/blog/spider-alternative-to-brightdata/)[Feb 20, 2026](https://spider.cloud/blog/spider-alternative-to-zenrows/)

### [ZenRows alternative? From $0.16 per 1,000 pages, no plan](https://spider.cloud/blog/spider-alternative-to-zenrows/)

[Spider costs from $0.16 per 1,000 pages on Nov 17 static pages. Cost varies by site. ZenRows' pricing page that day lists Build at $19 a month for 45,000 credits.](https://spider.cloud/blog/spider-alternative-to-zenrows/)

[comparisons · web-scraping · alternatives ](https://spider.cloud/blog/spider-alternative-to-zenrows/)[Feb 20, 2026](https://spider.cloud/blog/spider-vs-crawl4ai/)

### [Crawl4AI vs Spider in 2026: self-hosted vs managed](https://spider.cloud/blog/spider-vs-crawl4ai/)

[Crawl4AI is free Python you run yourself. Spider is a managed Rust crawler. The same 1,000 URLs on both, what self-hosting really costs, and code for each.](https://spider.cloud/blog/spider-vs-crawl4ai/)

[comparisons · web-scraping · alternatives ](https://spider.cloud/blog/spider-vs-crawl4ai/)[Feb 20, 2026](https://spider.cloud/blog/spider-vs-firecrawl/)

### [Firecrawl vs Spider 2026: MCP crawl speed and pricing](https://spider.cloud/blog/spider-vs-firecrawl/)

[Which web crawl MCP has the better crawl speed? On our 1,000-URL run Spider averaged 74 pages a second to Firecrawl's 16. Pricing and licenses compared.](https://spider.cloud/blog/spider-vs-firecrawl/)

[comparisons · web-scraping · alternatives ](https://spider.cloud/blog/spider-vs-firecrawl/)[Feb 20, 2026](https://spider.cloud/blog/spider-vs-jina-reader/)

### [Jina Reader alternative that crawls a whole site in one call](https://spider.cloud/blog/spider-vs-jina-reader/)

[How r.jina.ai works, its X-Return-Format header and its 500 requests a minute on a paid key. Spider crawls a site in one call and starts at 10,000 a minute.](https://spider.cloud/blog/spider-vs-jina-reader/)

[comparisons · web-scraping · alternatives ](https://spider.cloud/blog/spider-vs-jina-reader/)[Feb 20, 2026](https://spider.cloud/blog/spider-vs-scrapfly/)

### [Scrapfly vs Spider in 2026: credit multipliers and cost](https://spider.cloud/blog/spider-vs-scrapfly/)

[Scrapfly's 2026 plans and credit multipliers worked through at 10K, 100K and 1M pages against Spider usage billing, plus the independent success-rate numbers.](https://spider.cloud/blog/spider-vs-scrapfly/)

[comparisons · web-scraping · alternatives ](https://spider.cloud/blog/spider-vs-scrapfly/)[Feb 17, 2026](https://spider.cloud/blog/spider-alternative-to-crawlera-zyte/)

### [Crawlera alternative 2026: Spider vs Zyte API pricing](https://spider.cloud/blog/spider-alternative-to-crawlera-zyte/)

[Crawlera is now Zyte API, billed per request by a site tier Zyte assigns. What those bands cost, how Spider prices the same crawl, and a migration example.](https://spider.cloud/blog/spider-alternative-to-crawlera-zyte/)

[comparisons · web-scraping · alternatives ](https://spider.cloud/blog/spider-alternative-to-crawlera-zyte/)[Feb 17, 2026](https://spider.cloud/blog/spider-alternative-to-netnut/)

### [NetNut alternatives in 2026: what to use after the seizure](https://spider.cloud/blog/spider-alternative-to-netnut/)

[NetNut alternatives after the July 2, 2026 FBI seizure: what to swap in this week, a two-line proxy migration, and when to replace the whole stack.](https://spider.cloud/blog/spider-alternative-to-netnut/)

[comparisons · web-scraping · alternatives ](https://spider.cloud/blog/spider-alternative-to-netnut/)[Feb 17, 2026](https://spider.cloud/blog/spider-alternative-to-scraperapi/)

### [ScraperAPI pricing, free plan and an alternative with no plan](https://spider.cloud/blog/spider-alternative-to-scraperapi/)

[ScraperAPI plans list at $49 to $1,975 a month, rendering at 10 credits a page. Spider with no plan: from $0.16 per 1,000 static pages on Nov 17, varying by site.](https://spider.cloud/blog/spider-alternative-to-scraperapi/)

[comparisons · web-scraping · alternatives ](https://spider.cloud/blog/spider-alternative-to-scraperapi/)[Feb 17, 2026](https://spider.cloud/blog/spider-alternative-to-scrapingbee/)

### [ScrapingBee alternative billed from $0.16 per 1,000 pages](https://spider.cloud/blog/spider-alternative-to-scrapingbee/)

[ScrapingBee's rendered default is $0.37 to $1.27 per 1,000 on a fully used standard plan. Spider ran from $0.16 per 1,000 static pages Nov 17. Cost varies by site.](https://spider.cloud/blog/spider-alternative-to-scrapingbee/)

[comparisons · web-scraping · alternatives ](https://spider.cloud/blog/spider-alternative-to-scrapingbee/)[Feb 17, 2026](https://spider.cloud/blog/spider-browser-vs-kernel-browserbase-benchmark/)

### [Spider Browser vs. Kernel vs. Browserbase: 999 URLs benchmarked](https://spider.cloud/blog/spider-browser-vs-kernel-browserbase-benchmark/)

[Kernel benchmarked cold start. We benchmarked reliability: 999 URLs, 254 domains, 18 categories, 100% success and 2.5s median end-to-end latency.](https://spider.cloud/blog/spider-browser-vs-kernel-browserbase-benchmark/)

[benchmarks · browser-automation · comparisons ](https://spider.cloud/blog/spider-browser-vs-kernel-browserbase-benchmark/)[Feb 16, 2026](https://spider.cloud/blog/spider-mcp-v2-browser-automation-for-ai-agents/)

### [Spider MCP v2: browser automation for AI agents](https://spider.cloud/blog/spider-mcp-v2-browser-automation-for-ai-agents/)

[Spider's MCP server now ships 21 tools, 9 of them browser automation, giving agents direct control of native cloud browsers that load protected pages completely, with proxy rotation built in.](https://spider.cloud/blog/spider-mcp-v2-browser-automation-for-ai-agents/)

[MCP · AI · web-scraping ](https://spider.cloud/blog/spider-mcp-v2-browser-automation-for-ai-agents/)[Feb 11, 2026](https://spider.cloud/blog/build-production-rag-pipeline-web-data-30-minutes/)

### [Build a production RAG pipeline with web data in under 30 minutes](https://spider.cloud/blog/build-production-rag-pipeline-web-data-30-minutes/)

[Crawl a site with Spider, chunk the markdown, embed it, store it in a vector database and query it. Working code for LangChain, LlamaIndex, CrewAI and AutoGen.](https://spider.cloud/blog/build-production-rag-pipeline-web-data-30-minutes/)

[AI · RAG · tutorial ](https://spider.cloud/blog/build-production-rag-pipeline-web-data-30-minutes/)[Feb 11, 2026](https://spider.cloud/blog/building-ai-agents-that-browse-the-web/)

### [Building AI agents that browse the web](https://spider.cloud/blog/building-ai-agents-that-browse-the-web/)

[Architecture patterns and working code for web-browsing agents: research, monitoring and extraction, built on CrewAI and AutoGen with Spider as the backend.](https://spider.cloud/blog/building-ai-agents-that-browse-the-web/)

[AI · agents · architecture ](https://spider.cloud/blog/building-ai-agents-that-browse-the-web/)[Feb 11, 2026](https://spider.cloud/blog/building-mcp-server-for-web-scraping/)

### [Building an MCP server for web scraping](https://spider.cloud/blog/building-mcp-server-for-web-scraping/)

[Build an MCP server in TypeScript that wraps Spider's API, giving any AI model the ability to crawl, scrape, search and extract structured data.](https://spider.cloud/blog/building-mcp-server-for-web-scraping/)

[AI · MCP · tutorial ](https://spider.cloud/blog/building-mcp-server-for-web-scraping/)[Feb 11, 2026](https://spider.cloud/blog/bypass-cloudflare-datadome-perimeterx-2026/)

### [Why patched headless browsers keep breaking on modern sites](https://spider.cloud/blog/bypass-cloudflare-datadome-perimeterx-2026/)

[What a demanding site checks before it serves a page, why a patched headless browser fails those checks within weeks, and why a native engine does not.](https://spider.cloud/blog/bypass-cloudflare-datadome-perimeterx-2026/)

[web-scraping · browser-automation · engineering ](https://spider.cloud/blog/bypass-cloudflare-datadome-perimeterx-2026/)[Feb 11, 2026](https://spider.cloud/blog/developers-guide-choosing-scraping-stack-2026/)

### [Choosing a web scraping stack in 2026](https://spider.cloud/blog/developers-guide-choosing-scraping-stack-2026/)

[Does your scraping stack need a browser? In a November 2026 run a plain fetch returned at least 90% of a browser's words on 26 of 29 pages.](https://spider.cloud/blog/developers-guide-choosing-scraping-stack-2026/)

[web-scraping · developers · comparisons ](https://spider.cloud/blog/developers-guide-choosing-scraping-stack-2026/)[Feb 11, 2026](https://spider.cloud/blog/firecrawl-vs-crawl4ai-vs-spider-honest-benchmark/)

### [Crawl4AI vs Firecrawl vs Spider: 99.9% success on 1,000 URLs](https://spider.cloud/blog/firecrawl-vs-crawl4ai-vs-spider-honest-benchmark/)

[In our February 2026 run on 1,000 URLs, Firecrawl beat Crawl4AI on every metric, 95.7% success to 90.4%. Spider returned 99.9% at 74 pages a second.](https://spider.cloud/blog/firecrawl-vs-crawl4ai-vs-spider-honest-benchmark/)

[benchmarks · web-scraping · comparisons ](https://spider.cloud/blog/firecrawl-vs-crawl4ai-vs-spider-honest-benchmark/)[Feb 11, 2026](https://spider.cloud/blog/open-source-web-scraping-why-mit-license-matters/)

### [Open source web scraping: why MIT license matters](https://spider.cloud/blog/open-source-web-scraping-why-mit-license-matters/)

[How MIT, Apache 2.0, AGPL and BSL affect your right to build commercial products on scraping tools, and why Spider picked MIT.](https://spider.cloud/blog/open-source-web-scraping-why-mit-license-matters/)

[open-source · web-scraping · engineering ](https://spider.cloud/blog/open-source-web-scraping-why-mit-license-matters/)[Feb 11, 2026](https://spider.cloud/blog/rust-vs-python-web-scraping-why-we-rewrote-everything/)

### [Rust vs. Python for web scraping: why we rewrote everything](https://spider.cloud/blog/rust-vs-python-web-scraping-why-we-rewrote-everything/)

[Why Spider rebuilt its scrapers in Rust, and a dated crawl of five documentation sites with the open source spider crate 2.53.9 that you can run again.](https://spider.cloud/blog/rust-vs-python-web-scraping-why-we-rewrote-everything/)

[engineering · rust · performance ](https://spider.cloud/blog/rust-vs-python-web-scraping-why-we-rewrote-everything/)[Feb 11, 2026](https://spider.cloud/blog/scraping-one-million-pages-what-actually-happens/)

### [Scraping 1 million pages: what actually happens](https://spider.cloud/blog/scraping-one-million-pages-what-actually-happens/)

[An engineering log of crawling 1 million pages across 10,000 domains with Spider's API. Throughput curves, failure modes, cost breakdown, and lessons learned.](https://spider.cloud/blog/scraping-one-million-pages-what-actually-happens/)

[benchmarks · engineering · web-scraping ](https://spider.cloud/blog/scraping-one-million-pages-what-actually-happens/)[Feb 11, 2026](https://spider.cloud/blog/spider-go-to-market-how-we-grew/)

### [How Spider went to market: what worked and what we'd redo](https://spider.cloud/blog/spider-go-to-market-how-we-grew/)

[How we built Spider's go-to-market from zero: the channels that worked, the pricing mistakes, and the content that actually converted.](https://spider.cloud/blog/spider-go-to-market-how-we-grew/)

[engineering · startup · go-to-market ](https://spider.cloud/blog/spider-go-to-market-how-we-grew/)[Feb 11, 2026](https://spider.cloud/blog/top-5-data-collection-platforms/)

### [Data collection platforms for AI: 250 JSON rows from one call without a model](https://spider.cloud/blog/top-5-data-collection-platforms/)

[Data collection platforms for AI, compared by what they return. One Spider call returned 250 rows of four named fields as JSON in 1.4 seconds with no model.](https://spider.cloud/blog/top-5-data-collection-platforms/)

[AI · web-scraping · developers ](https://spider.cloud/blog/top-5-data-collection-platforms/)[Feb 11, 2026](https://spider.cloud/blog/true-cost-of-web-scraping-at-scale/)

### [Web scraping at scale: cost per 1,000 pages by kind of site](https://spider.cloud/blog/true-cost-of-web-scraping-at-scale/)

[Web scraping cost per 1,000 pages, measured daily from Nov 6 to 17, 2026 on five kinds of page. Documentation, 25 fixed pages, ran from $0.274. Cost varies by site.](https://spider.cloud/blog/true-cost-of-web-scraping-at-scale/)

[web-scraping · cost-analysis · developers ](https://spider.cloud/blog/true-cost-of-web-scraping-at-scale/)[Feb 11, 2026](https://spider.cloud/blog/web-page-to-vector-database-complete-pipeline/)

### [Web data into a vector database: the complete pipeline](https://spider.cloud/blog/web-page-to-vector-database-complete-pipeline/)

[Turn web pages into vectors in Pinecone, Weaviate, Chroma or pgvector: crawl, extract, chunk, embed and index, with working Python and a 10K-page cost breakdown.](https://spider.cloud/blog/web-page-to-vector-database-complete-pipeline/)

[AI · vector-databases · pipeline ](https://spider.cloud/blog/web-page-to-vector-database-complete-pipeline/)[Feb 11, 2026](https://spider.cloud/blog/web-scraping-ai-training-data-legal-technical-guide-2026/)

### [Web scraping for AI training data: legal and technical guide 2026](https://spider.cloud/blog/web-scraping-ai-training-data-legal-technical-guide-2026/)

[What the EU AI Act, US fair use, and GDPR require of AI training data, and how to build a scraping pipeline that holds up in 2026.](https://spider.cloud/blog/web-scraping-ai-training-data-legal-technical-guide-2026/)

[AI · legal · web-scraping](https://spider.cloud/blog/web-scraping-ai-training-data-legal-technical-guide-2026/)

## Start crawling in 30 seconds.

One API key. No servers to manage.

2,500 free credits, worth $0.25, if your signup passes our checks. No card required.

[Get an API key](https://spider.cloud/dashboard/) [Read the docs](https://spider.cloud/docs/overview/)
