Notes from the engineering team.
Writeups, benchmarks, and field notes on web data collection and AI infrastructure.
Latest 18 hours ago
How to scrape a website: six steps, each one a request we sent
How to scrape a website, step by step for beginners. We sent every step to a public practice site on November 17, 2026 and print what came back, with the time and cost of each request sent with a key.
Archive
Built-in proxies for web scraping: median $0.0025 a complete page
Proxies for web scraping come built into every Spider request. On November 17, 2026, a complete category or listing page cost a median of $0.0025, proxy included.
Web scraping for AI agents: what each call adds to the context
Web scraping for AI agents, measured November 17, 2026. Tokens, seconds and cost for a search with page content at 3, 5 and 10 results, one scrape and one crawl.
Web scraping to markdown: a 63-page docs site in one call
Web scraping to markdown, measured on November 17, 2026. One crawl call returned a 63-page docs site as markdown in 3.5 seconds for $0.0138.
Web scraping with JavaScript: a Node.js script that streams each page to disk
Web scraping with JavaScript in Node.js and the Spider client. One script scrapes a page, reads a field by CSS selector and streams a 25 page crawl to a file. Run against the API on November 17, 2026.
How web scraping works: what a plain fetch gets and misses
How web scraping works, measured on November 16, 2026. A plain HTTP fetch returned 90% or more of a browser's words on 26 of 29 docs, news and product pages.
Is web scraping legal? What 67 top sites ask for in robots.txt
We asked 100 top-ranked domains for robots.txt on 2026-11-16. Of the 67 that served one, 13 hold Disallow: / for unnamed crawlers and 29 name an AI crawler.
Web scraping vs web crawling, timed on one documentation site
Web scraping vs web crawling, measured on November 16, 2026. One crawl call returned 74 pages in 5.1 s. A scrape call per URL returned the same markdown.
Web scraping with Python: one script, 25 pages, three runs
A web scraping script in Python's standard library, printed whole. On November 16, 2026 it returned 25 of 25 pages as markdown in each of three runs.
Build a fine-tuning dataset from the web in two API calls
One search call and one crawl call turned a topic into 396 training records for $0.05. Every number comes from a run on 60 public documentation pages.
What a web page costs in tokens, measured on ten public pages
Raw HTML, markdown, readability and text output for the same ten pages, counted in o200k tokens. Markdown cut the bill 15x at the median, and readability mode left four index pages with under 60 tokens.
Spider Peers: earn from the internet you already pay for
Install Spider Peers on a computer, phone or TV and earn from your spare home bandwidth. Set it up once and it runs in the background.
Spider Shield: a free blocker for ads, trackers, phishing and malware
Spider Shield blocks ads, trackers, phishing and malware in every app on Mac, Windows, Linux and Android. It's free and needs no account.
8 best Apify alternatives and competitors in 2026
Eight Apify alternatives, including Spider, Firecrawl, Crawl4AI and Bright Data, compared on September 2026 pricing, cost per 100,000 pages, license and MCP.
Firecrawl alternatives: 8 competitors priced at 100K pages
Firecrawl pricing against eight competitors, with September 2026 list prices, cost at 100,000 pages, license and MCP support, and where each falls short.
ScrapingBee competitors: 8 APIs priced at 100,000 pages
Eight ScrapingBee competitors on price, credit multipliers, cost at 100,000 pages and MCP support. September 2026 prices, part rechecked in November.
8 best ZenRows alternatives in 2026 (pricing, benchmarks)
Eight ZenRows alternatives compared on September 2026 pricing, credit multipliers, cost at 100,000 pages, license and MCP support, with where each falls short.
Flat-rate web scraping vs. pay-as-you-go: which to put where
How the flat-rate Unlimited plan puts steady high-volume crawling on a fixed bill, where pay-as-you-go still wins, and how both run on one API key.
How to scrape the web at scale from your terminal
Every crawl mode the Spider CLI supports, with real examples, and how to go from one page to millions without leaving the terminal.
Spider Browser passes 80/80 on Browser Fidelity Bench
Spider Browser passed all 80 tasks on Browser Use's Browser Fidelity Bench V1 in September, up from 68 in March. Every other cloud browser tested scored lower.
Keep RAG context fresh with real-time web search
Static knowledge bases go stale in days. Build a RAG pipeline on live web data by adding web search at query time, with LangChain and plain Python examples.
Web search API for AI agents: search and extract in one call
Spider's Search API searches, crawls and cleans in one request, so one agent covers live web search, deep research and page monitoring without glue code.
Introducing Silk: our custom AI model for web data extraction
Silk, Spider's own extraction model, turns raw HTML into structured data and handles challenge pages on our GPUs, with no outside API calls or per-token billing.
Case study: how a RAG pipeline went from 6 hours to 15 minutes
A Series A AI company replaced three Python microservices, a proxy provider, and half an engineer's time with a single Spider API call. This is what changed.
The 7 best web scraping APIs for AI in 2026
The best web scraping APIs in 2026 for AI: Spider, Firecrawl, Crawl4AI, ScrapingBee, Apify, Bright Data and Jina Reader compared on real pricing and benchmarks.
Oxylabs Web Scraper API pricing (Nov 2026) and alternatives
Oxylabs plans start at $15 a month, and a general site costs $1.00 per 1,000 results, $1.50 rendered. Compared with Spider, Apify and ScrapingBee.
Apify alternative 2026: Spider vs Apify pricing and CUs
How Apify compute units, proxy bandwidth and plan floors add up at 10K, 100K and 1M pages, priced against Spider's usage billing, with a migration example.
Bright Data alternative and pricing: $1.50 per 1K vs Spider
Bright Data's Web Unlocker is $1.50 per 1,000 requests pay as you go. 1M pages is about $1,301 on the $499 Scale plan. Priced against Spider.
ZenRows alternative: plans from $19, and the 25 credit page
ZenRows plans run $19 to $2,999 a month in November 2026, and a rendered page on premium proxies costs 25 credits. How Spider's usage bill compares.
Crawl4AI vs Spider in 2026: self-hosted vs managed
Crawl4AI is free Python you run yourself. Spider is a managed Rust crawler. The same 1,000 URLs on both, what self-hosting really costs, and code for each.
Firecrawl vs Spider 2026: MCP crawl speed and pricing
Which web crawl MCP has the better crawl speed? On our 1,000-URL run Spider averaged 74 pages a second to Firecrawl's 16. Pricing and licenses compared.
Jina Reader alternative: r.jina.ai headers, limits and cost
How r.jina.ai works, with X-Return-Format, X-Respond-With and 500 requests a minute on a free or paid key, and when Spider's whole-site crawl fits an agent.
Scrapfly vs Spider in 2026: credit multipliers and cost
Scrapfly's 2026 plans and credit multipliers worked through at 10K, 100K and 1M pages against Spider usage billing, plus the independent success-rate numbers.
Crawlera alternative 2026: Spider vs Zyte API pricing
Crawlera is now Zyte API, billed per request by a site tier Zyte assigns. What those bands cost, how Spider prices the same crawl, and a migration example.
NetNut alternatives in 2026: what to use after the seizure
NetNut alternatives after the July 2, 2026 FBI seizure: what to swap in this week, a two-line proxy migration, and when to replace the whole stack.
ScraperAPI pricing 2026: $49 to $1,975 a month, by plan
ScraperAPI plans run $49 to $1,975 a month as of November 2026, and a rendered protected page costs 75 credits. Spider's flat rate starts at $40 a month.
ScrapingBee alternative: 13,333 stealth requests for $99
ScrapingBee's stealth proxy costs 75 credits a request, so its $99 plan buys 13,333 of them. Pricing as of November 2026, and a usage-billed option.
Spider Browser vs. Kernel vs. Browserbase: 999 URLs benchmarked
Kernel benchmarked cold start. We benchmarked reliability: 999 URLs, 254 domains, 18 categories, 100% success and 2.5s median end-to-end latency.
Spider MCP v2: browser automation for AI agents
Spider's MCP server now ships 21 tools, 9 of them browser automation, giving agents direct control of native cloud browsers that load protected pages completely, with proxy rotation built in.
Build a production RAG pipeline with web data in under 30 minutes
Crawl a site with Spider, chunk the markdown, embed it, store it in a vector database and query it. Working code for LangChain, LlamaIndex, CrewAI and AutoGen.
Building AI agents that browse the web
Architecture patterns and working code for web-browsing agents: research, monitoring and extraction, built on CrewAI and AutoGen with Spider as the backend.
Building an MCP server for web scraping
Build an MCP server in TypeScript that wraps Spider's API, giving any AI model the ability to crawl, scrape, search and extract structured data.
Why patched headless browsers keep breaking on modern sites
What a demanding site checks before it serves a page, why a patched headless browser fails those checks within weeks, and why a native engine does not.
The developer's guide to choosing a scraping stack in 2026
DIY libraries, open source frameworks, managed APIs, AI-native extractors and browser automation, with a decision matrix and a hidden-cost audit for each.
Crawl4AI vs Firecrawl vs Spider: a 1,000 URL benchmark
Crawl4AI vs Firecrawl on 1,000 real URLs. Firecrawl won success rate, 95.3% to 89.7%, and RAG recall@5, 89.0% to 84.5%. Crawl4AI is Apache 2.0 and self-hosted.
Open source web scraping: why MIT license matters
How MIT, Apache 2.0, AGPL and BSL affect your right to build commercial products on scraping tools, and why Spider picked MIT.
Rust vs. Python for web scraping: why we rewrote everything
Why Spider rebuilt its scrapers in Rust, and a dated crawl of five documentation sites with the open source spider crate 2.53.9 that you can run again.
Scraping 1 million pages: what actually happens
An engineering log of crawling 1 million pages across 10,000 domains with Spider's API. Throughput curves, failure modes, cost breakdown, and lessons learned.
How Spider went to market: what worked and what we'd redo
How we built Spider's go-to-market from zero: the channels that worked, the pricing mistakes, and the content that actually converted.
Top 5 data collection platforms for AI and web scraping in 2026
The leading web scraping platforms for AI agents and RAG, compared on cost per 1,000 pages, speed, reliability and the trade-offs of each.
Web scraping cost per 1,000 pages, by kind of page
Day 1 of a seven-day run: web scraping cost $0.27 per 1,000 docs pages and $1.48 for reference, plus the bill at 10 million pages a month.
Web data into a vector database: the complete pipeline
Turn web pages into vectors in Pinecone, Weaviate, Chroma or pgvector: crawl, extract, chunk, embed and index, with working Python and a 10K-page cost breakdown.
Web scraping for AI training data: legal and technical guide 2026
What the EU AI Act, US fair use, and GDPR require of AI training data, and how to build a scraping pipeline that holds up in 2026.
Start crawling in 30 seconds.
One API key. No servers to manage.
2,500 free credits, worth $0.25, if your signup passes our checks. No card required.