Skip to main content

Notes from the engineering team.

Writeups, benchmarks, and field notes on web data collection and AI infrastructure.

Latest 18 hours ago

web-scraping guides

How to scrape a website: six steps, each one a request we sent

How to scrape a website, step by step for beginners. We sent every step to a public practice site on November 17, 2026 and print what came back, with the time and cost of each request sent with a key.

Jeff Mendez
Read

Archive

Built-in proxies for web scraping: median $0.0025 a complete page

Proxies for web scraping come built into every Spider request. On November 17, 2026, a complete category or listing page cost a median of $0.0025, proxy included.

Jeff Mendez web-scraping · engineering · benchmarks

Web scraping for AI agents: what each call adds to the context

Web scraping for AI agents, measured November 17, 2026. Tokens, seconds and cost for a search with page content at 3, 5 and 10 results, one scrape and one crawl.

Jeff Mendez web-scraping · ai-agents · engineering

Web scraping to markdown: a 63-page docs site in one call

Web scraping to markdown, measured on November 17, 2026. One crawl call returned a 63-page docs site as markdown in 3.5 seconds for $0.0138.

Jeff Mendez web-scraping · engineering · benchmarks

Web scraping with JavaScript: a Node.js script that streams each page to disk

Web scraping with JavaScript in Node.js and the Spider client. One script scrapes a page, reads a field by CSS selector and streams a 25 page crawl to a file. Run against the API on November 17, 2026.

Jeff Mendez web-scraping · engineering · javascript

How web scraping works: what a plain fetch gets and misses

How web scraping works, measured on November 16, 2026. A plain HTTP fetch returned 90% or more of a browser's words on 26 of 29 docs, news and product pages.

Jeff Mendez web-scraping · engineering · benchmarks

Is web scraping legal? What 67 top sites ask for in robots.txt

We asked 100 top-ranked domains for robots.txt on 2026-11-16. Of the 67 that served one, 13 hold Disallow: / for unnamed crawlers and 29 name an AI crawler.

Jeff Mendez web-scraping · engineering · benchmarks

Web scraping vs web crawling, timed on one documentation site

Web scraping vs web crawling, measured on November 16, 2026. One crawl call returned 74 pages in 5.1 s. A scrape call per URL returned the same markdown.

Jeff Mendez web-scraping · engineering · benchmarks

Web scraping with Python: one script, 25 pages, three runs

A web scraping script in Python's standard library, printed whole. On November 16, 2026 it returned 25 of 25 pages as markdown in each of three runs.

Jeff Mendez web-scraping · engineering · python

Build a fine-tuning dataset from the web in two API calls

One search call and one crawl call turned a topic into 396 training records for $0.05. Every number comes from a run on 60 public documentation pages.

Jeff Mendez engineering · AI · tutorials

What a web page costs in tokens, measured on ten public pages

Raw HTML, markdown, readability and text output for the same ten pages, counted in o200k tokens. Markdown cut the bill 15x at the median, and readability mode left four index pages with under 60 tokens.

Jeff Mendez engineering · AI · benchmarks

Spider Peers: earn from the internet you already pay for

Install Spider Peers on a computer, phone or TV and earn from your spare home bandwidth. Set it up once and it runs in the background.

Jeff Mendez peers · bandwidth-sharing

Spider Shield: a free blocker for ads, trackers, phishing and malware

Spider Shield blocks ads, trackers, phishing and malware in every app on Mac, Windows, Linux and Android. It's free and needs no account.

Jeff Mendez shield · privacy · security

8 best Apify alternatives and competitors in 2026

Eight Apify alternatives, including Spider, Firecrawl, Crawl4AI and Bright Data, compared on September 2026 pricing, cost per 100,000 pages, license and MCP.

Jeff Mendez comparisons · alternatives · web-scraping

Firecrawl alternatives: 8 competitors priced at 100K pages

Firecrawl pricing against eight competitors, with September 2026 list prices, cost at 100,000 pages, license and MCP support, and where each falls short.

Jeff Mendez comparisons · alternatives · web-scraping

ScrapingBee competitors: 8 APIs priced at 100,000 pages

Eight ScrapingBee competitors on price, credit multipliers, cost at 100,000 pages and MCP support. September 2026 prices, part rechecked in November.

Jeff Mendez comparisons · alternatives · web-scraping

8 best ZenRows alternatives in 2026 (pricing, benchmarks)

Eight ZenRows alternatives compared on September 2026 pricing, credit multipliers, cost at 100,000 pages, license and MCP support, with where each falls short.

Jeff Mendez comparisons · alternatives · web-scraping

Flat-rate web scraping vs. pay-as-you-go: which to put where

How the flat-rate Unlimited plan puts steady high-volume crawling on a fixed bill, where pay-as-you-go still wins, and how both run on one API key.

Jeff Mendez web-scraping · cost-analysis · pricing

How to scrape the web at scale from your terminal

Every crawl mode the Spider CLI supports, with real examples, and how to go from one page to millions without leaving the terminal.

Jeff Mendez tutorial · cli · web-scraping

Spider Browser passes 80/80 on Browser Fidelity Bench

Spider Browser passed all 80 tasks on Browser Use's Browser Fidelity Bench V1 in September, up from 68 in March. Every other cloud browser tested scored lower.

Jeff Mendez benchmarks · browser-fidelity · browser-automation

Keep RAG context fresh with real-time web search

Static knowledge bases go stale in days. Build a RAG pipeline on live web data by adding web search at query time, with LangChain and plain Python examples.

Jeff Mendez AI · RAG · search

Web search API for AI agents: search and extract in one call

Spider's Search API searches, crawls and cleans in one request, so one agent covers live web search, deep research and page monitoring without glue code.

Jeff Mendez AI · search · agents

Introducing Silk: our custom AI model for web data extraction

Silk, Spider's own extraction model, turns raw HTML into structured data and handles challenge pages on our GPUs, with no outside API calls or per-token billing.

Jeff Mendez engineering · AI · extraction

Case study: how a RAG pipeline went from 6 hours to 15 minutes

A Series A AI company replaced three Python microservices, a proxy provider, and half an engineer's time with a single Spider API call. This is what changed.

Jeff Mendez case-study · RAG · AI

The 7 best web scraping APIs for AI in 2026

The best web scraping APIs in 2026 for AI: Spider, Firecrawl, Crawl4AI, ScrapingBee, Apify, Bright Data and Jina Reader compared on real pricing and benchmarks.

Jeff Mendez comparisons · web-scraping · AI

Oxylabs Web Scraper API pricing (Nov 2026) and alternatives

Oxylabs plans start at $15 a month, and a general site costs $1.00 per 1,000 results, $1.50 rendered. Compared with Spider, Apify and ScrapingBee.

Jeff Mendez comparisons · web-scraping · alternatives

Apify alternative 2026: Spider vs Apify pricing and CUs

How Apify compute units, proxy bandwidth and plan floors add up at 10K, 100K and 1M pages, priced against Spider's usage billing, with a migration example.

Jeff Mendez comparisons · web-scraping · alternatives

Bright Data alternative and pricing: $1.50 per 1K vs Spider

Bright Data's Web Unlocker is $1.50 per 1,000 requests pay as you go. 1M pages is about $1,301 on the $499 Scale plan. Priced against Spider.

Jeff Mendez comparisons · web-scraping · alternatives

ZenRows alternative: plans from $19, and the 25 credit page

ZenRows plans run $19 to $2,999 a month in November 2026, and a rendered page on premium proxies costs 25 credits. How Spider's usage bill compares.

Jeff Mendez comparisons · web-scraping · alternatives

Crawl4AI vs Spider in 2026: self-hosted vs managed

Crawl4AI is free Python you run yourself. Spider is a managed Rust crawler. The same 1,000 URLs on both, what self-hosting really costs, and code for each.

Jeff Mendez comparisons · web-scraping · alternatives

Firecrawl vs Spider 2026: MCP crawl speed and pricing

Which web crawl MCP has the better crawl speed? On our 1,000-URL run Spider averaged 74 pages a second to Firecrawl's 16. Pricing and licenses compared.

Jeff Mendez comparisons · web-scraping · alternatives

Jina Reader alternative: r.jina.ai headers, limits and cost

How r.jina.ai works, with X-Return-Format, X-Respond-With and 500 requests a minute on a free or paid key, and when Spider's whole-site crawl fits an agent.

Jeff Mendez comparisons · web-scraping · alternatives

Scrapfly vs Spider in 2026: credit multipliers and cost

Scrapfly's 2026 plans and credit multipliers worked through at 10K, 100K and 1M pages against Spider usage billing, plus the independent success-rate numbers.

Jeff Mendez comparisons · web-scraping · alternatives

Crawlera alternative 2026: Spider vs Zyte API pricing

Crawlera is now Zyte API, billed per request by a site tier Zyte assigns. What those bands cost, how Spider prices the same crawl, and a migration example.

Jeff Mendez comparisons · web-scraping · alternatives

NetNut alternatives in 2026: what to use after the seizure

NetNut alternatives after the July 2, 2026 FBI seizure: what to swap in this week, a two-line proxy migration, and when to replace the whole stack.

Jeff Mendez comparisons · web-scraping · alternatives

ScraperAPI pricing 2026: $49 to $1,975 a month, by plan

ScraperAPI plans run $49 to $1,975 a month as of November 2026, and a rendered protected page costs 75 credits. Spider's flat rate starts at $40 a month.

Jeff Mendez comparisons · web-scraping · alternatives

ScrapingBee alternative: 13,333 stealth requests for $99

ScrapingBee's stealth proxy costs 75 credits a request, so its $99 plan buys 13,333 of them. Pricing as of November 2026, and a usage-billed option.

Jeff Mendez comparisons · web-scraping · alternatives

Spider Browser vs. Kernel vs. Browserbase: 999 URLs benchmarked

Kernel benchmarked cold start. We benchmarked reliability: 999 URLs, 254 domains, 18 categories, 100% success and 2.5s median end-to-end latency.

Jeff Mendez benchmarks · browser-automation · comparisons

Spider MCP v2: browser automation for AI agents

Spider's MCP server now ships 21 tools, 9 of them browser automation, giving agents direct control of native cloud browsers that load protected pages completely, with proxy rotation built in.

Jeff Mendez MCP · AI · web-scraping

Build a production RAG pipeline with web data in under 30 minutes

Crawl a site with Spider, chunk the markdown, embed it, store it in a vector database and query it. Working code for LangChain, LlamaIndex, CrewAI and AutoGen.

Jeff Mendez AI · RAG · tutorial

Building AI agents that browse the web

Architecture patterns and working code for web-browsing agents: research, monitoring and extraction, built on CrewAI and AutoGen with Spider as the backend.

Jeff Mendez AI · agents · architecture

Building an MCP server for web scraping

Build an MCP server in TypeScript that wraps Spider's API, giving any AI model the ability to crawl, scrape, search and extract structured data.

Jeff Mendez AI · MCP · tutorial

Why patched headless browsers keep breaking on modern sites

What a demanding site checks before it serves a page, why a patched headless browser fails those checks within weeks, and why a native engine does not.

Jeff Mendez web-scraping · browser-automation · engineering

The developer's guide to choosing a scraping stack in 2026

DIY libraries, open source frameworks, managed APIs, AI-native extractors and browser automation, with a decision matrix and a hidden-cost audit for each.

Jeff Mendez web-scraping · developers · comparisons

Crawl4AI vs Firecrawl vs Spider: a 1,000 URL benchmark

Crawl4AI vs Firecrawl on 1,000 real URLs. Firecrawl won success rate, 95.3% to 89.7%, and RAG recall@5, 89.0% to 84.5%. Crawl4AI is Apache 2.0 and self-hosted.

Jeff Mendez benchmarks · web-scraping · comparisons

Open source web scraping: why MIT license matters

How MIT, Apache 2.0, AGPL and BSL affect your right to build commercial products on scraping tools, and why Spider picked MIT.

Jeff Mendez open-source · web-scraping · engineering

Rust vs. Python for web scraping: why we rewrote everything

Why Spider rebuilt its scrapers in Rust, and a dated crawl of five documentation sites with the open source spider crate 2.53.9 that you can run again.

Jeff Mendez engineering · rust · performance

Scraping 1 million pages: what actually happens

An engineering log of crawling 1 million pages across 10,000 domains with Spider's API. Throughput curves, failure modes, cost breakdown, and lessons learned.

Jeff Mendez benchmarks · engineering · web-scraping

How Spider went to market: what worked and what we'd redo

How we built Spider's go-to-market from zero: the channels that worked, the pricing mistakes, and the content that actually converted.

Jeff Mendez engineering · startup · go-to-market

Top 5 data collection platforms for AI and web scraping in 2026

The leading web scraping platforms for AI agents and RAG, compared on cost per 1,000 pages, speed, reliability and the trade-offs of each.

Jeff Mendez AI · web-scraping · developers

Web scraping cost per 1,000 pages, by kind of page

Day 1 of a seven-day run: web scraping cost $0.27 per 1,000 docs pages and $1.48 for reference, plus the bill at 10 million pages a month.

Jeff Mendez web-scraping · cost-analysis · developers

Web data into a vector database: the complete pipeline

Turn web pages into vectors in Pinecone, Weaviate, Chroma or pgvector: crawl, extract, chunk, embed and index, with working Python and a 10K-page cost breakdown.

Jeff Mendez AI · vector-databases · pipeline

Web scraping for AI training data: legal and technical guide 2026

What the EU AI Act, US fair use, and GDPR require of AI training data, and how to build a scraping pipeline that holds up in 2026.

Jeff Mendez AI · legal · web-scraping

Start crawling in 30 seconds.

One API key. No servers to manage.

2,500 free credits, worth $0.25, if your signup passes our checks. No card required.

Get started freeRead the docs