Skip to main content

Notes from the engineering team.

Writeups, benchmarks, and field notes on web data collection and AI infrastructure.

Latest 1 day ago

AI cost-analysis benchmarks

Fetching the web is the cheap part of an agent's research bill

We priced the fetch and the page input for five agent research tasks on 2026-11-18. Fetching ten pages was the small line. Reading them as raw HTML took 6.5x to 22.5x the tokens of markdown on four tasks and 1.8x on the fifth.

Read

Archive

How to scrape a website, step by step

How to scrape a website, from reading robots.txt to saving a crawl one page at a time, with the curl command for each step and what it returned.

web-scraping · guides

Proxies for web scraping, built into every request

Proxies for web scraping come built into every Spider request and rotate on their own. What they cost, and how to pick the exit country or use your own.

web-scraping · engineering · benchmarks

Web scraping for AI agents: search, scrape or crawl

Web scraping for AI agents: when an agent should search, scrape or crawl, and how many tokens of its context window each call fills, counted on real pages.

web-scraping · ai-agents · engineering

Web scraping to markdown: a whole docs site into files

Web scraping to markdown with one crawl call and a short Python script that writes a file per page, then a check of the code blocks and tables it kept.

web-scraping · engineering · benchmarks

Web scraping with a headless browser, without running one

How to scrape pages that build their text with JavaScript, using a browser on Spider's side, and how to tell which pages need one and what a render costs.

web-scraping · guides

Web scraping with JavaScript and Node.js

Web scraping with JavaScript: a Node.js script with one package that scrapes a page as markdown, pulls a field by CSS selector and streams a crawl to a file.

web-scraping · engineering · javascript

Free web scraper: three ways to scrape a page without paying

Scrape a page for free three ways: in the browser with no key, from curl with no key, or with the MIT-licensed spider CLI on your own machine.

web-scraping · tutorial · open-source

How web scraping works: what a plain fetch gets and misses

How web scraping works, from the HTTP request to the markdown you keep, and how to tell when a page needs a browser to run its scripts first.

web-scraping · engineering · benchmarks

Is web scraping legal? What 67 top sites ask for in robots.txt

Is web scraping legal? US and EU rulings and statutes by name and year, beside the robots.txt of 67 top sites. 13 hold Disallow: / for unnamed crawlers.

web-scraping · legal · benchmarks

Web scraping vs web crawling: when to crawl and when to scrape

The difference between web scraping and web crawling, timed on one docs site, and which call to use for a URL list, a whole site or only a question.

web-scraping · engineering · benchmarks

Web scraping with Python using only the standard library

A Python web scraper that uses only the standard library and gets each page back as markdown. The whole script, the pages we ran it on and what it returned.

web-scraping · engineering · python

What is a web scraping API? One request, read field by field

What a web scraping API does between your request and its answer, why a 200 call can carry a 404 page, and five things to check in any scraping API.

web-scraping · engineering · API

Build a fine-tuning dataset from the web in two API calls

One search call and one crawl call turned a topic into 396 training records for $0.05. Every number comes from a run on 60 public documentation pages.

engineering · AI · tutorials

What a web page costs in tokens, measured on ten public pages

Raw HTML, markdown, readability and text output for the same ten pages, counted in o200k tokens. Markdown cut the bill 15x at the median, and readability mode left four index pages with under 60 tokens.

engineering · AI · benchmarks

Spider Peers: earn from the internet you already pay for

Install Spider Peers on a computer, phone or TV and earn from your spare home bandwidth. Set it up once and it runs in the background.

peers · bandwidth-sharing

Spider Shield: a free blocker for ads, trackers, phishing and malware

Spider Shield blocks ads, trackers, phishing and malware in every app on Mac, Windows, Linux and Android. It's free and needs no account.

shield · privacy · security

Firecrawl alternatives: Spider at 99.9% success, 4.6x throughput

Spider finished 1,000 URLs at 99.9% success to Firecrawl's 95.7%, with 4.6x the throughput. It costs from $0.16 per 1,000 pages. Cost varies by site.

comparisons · alternatives · web-scraping

Flat-rate web scraping vs. pay-as-you-go: which to put where

How the flat-rate Unlimited plan puts steady high-volume crawling on a fixed bill, where pay-as-you-go still wins, and how both run on one API key.

web-scraping · cost-analysis · pricing

Command line web scraping: 100 pages in about 3 seconds

Three Spider CLI runs each crawled 100 FastAPI docs pages to markdown in about 3 seconds. Install it, add a key, then pipe the JSON to jq or write a WARC file.

tutorial · cli · web-scraping

Spider Browser passes 80/80 on Browser Fidelity Bench

Spider Browser passed all 80 tasks on Browser Use's Browser Fidelity Bench V1 in September, up from 68 in March. Every other cloud browser tested scored lower.

benchmarks · browser-fidelity · browser-automation

Keep RAG context fresh with real-time web search

Static knowledge bases go stale in days. Build a RAG pipeline on live web data by adding web search at query time, with LangChain and plain Python examples.

AI · RAG · search

Web search API for AI agents: search and extract in one call

Spider's Search API searches, crawls and cleans in one request, so one agent covers live web search, deep research and page monitoring without glue code.

AI · search · agents

Introducing Silk: our custom AI model for web data extraction

Silk, Spider's own extraction model, turns raw HTML into structured data and handles challenge pages on our GPUs, with no outside API calls or per-token billing.

engineering · AI · extraction

Case study: how a RAG pipeline went from 6 hours to 15 minutes

A Series A AI company replaced three Python microservices, a proxy provider, and half an engineer's time with a single Spider API call. This is what changed.

case-study · RAG · AI

Best web scraping APIs for AI and LLM pipelines in 2026

On 60 docs pages, markdown took 3,369 tokens a page and raw HTML 123,813. Seven web scraping APIs for RAG and LLMs, with 100,000-page costs where one is listed.

comparisons · web-scraping · AI

Oxylabs Web Scraper API pricing by plan, compared to Spider

Oxylabs lists unrendered general sites at $1.00 per 1,000 results on Starter, $0.75 on Enterprise. Spider: from $0.16 per 1,000 static pages on Nov 17, varies by site.

comparisons · web-scraping · alternatives

Apify alternative 2026: Spider vs Apify pricing and CUs

What each Apify plan buys in compute units, why proxy bandwidth and Actor memory push the bill higher, what Spider costs per kind of page, and a migration example.

comparisons · web-scraping · alternatives

From $0.16 per 1,000 pages: a Bright Data alternative

Spider: from $0.16 per 1,000 pages, Nov 17 static pages. Cost varies by site. Bright Data Web Unlocker list price, Nov 17, 2026: $1.50 pay as you go, $1.00 on Scale.

comparisons · web-scraping · alternatives

ZenRows alternative? From $0.16 per 1,000 pages, no plan

Spider costs from $0.16 per 1,000 pages on Nov 17 static pages. Cost varies by site. ZenRows' pricing page that day lists Build at $19 a month for 45,000 credits.

comparisons · web-scraping · alternatives

Crawl4AI vs Spider in 2026: self-hosted vs managed

Crawl4AI is free Python you run yourself. Spider is a managed Rust crawler. The same 1,000 URLs on both, what self-hosting really costs, and code for each.

comparisons · web-scraping · alternatives

Firecrawl vs Spider 2026: MCP crawl speed and pricing

Which web crawl MCP has the better crawl speed? On our 1,000-URL run Spider averaged 74 pages a second to Firecrawl's 16. Pricing and licenses compared.

comparisons · web-scraping · alternatives

Jina Reader alternative that crawls a whole site in one call

How r.jina.ai works, its X-Return-Format header and its 500 requests a minute on a paid key. Spider crawls a site in one call and starts at 10,000 a minute.

comparisons · web-scraping · alternatives

Scrapfly vs Spider in 2026: credit multipliers and cost

Scrapfly's 2026 plans and credit multipliers worked through at 10K, 100K and 1M pages against Spider usage billing, plus the independent success-rate numbers.

comparisons · web-scraping · alternatives

Crawlera alternative 2026: Spider vs Zyte API pricing

Crawlera is now Zyte API, billed per request by a site tier Zyte assigns. What those bands cost, how Spider prices the same crawl, and a migration example.

comparisons · web-scraping · alternatives

NetNut alternatives in 2026: what to use after the seizure

NetNut alternatives after the July 2, 2026 FBI seizure: what to swap in this week, a two-line proxy migration, and when to replace the whole stack.

comparisons · web-scraping · alternatives

ScraperAPI pricing, free plan and an alternative with no plan

ScraperAPI plans list at $49 to $1,975 a month, rendering at 10 credits a page. Spider with no plan: from $0.16 per 1,000 static pages on Nov 17, varying by site.

comparisons · web-scraping · alternatives

ScrapingBee alternative billed from $0.16 per 1,000 pages

ScrapingBee's rendered default costs $0.37 to $1.27 per 1,000 on a fully used standard plan. Spider ran from $0.16 per 1,000 static pages on Nov 17. Cost varies by site.

comparisons · web-scraping · alternatives

Spider Browser vs. Kernel vs. Browserbase: 999 URLs benchmarked

Kernel benchmarked cold start. We benchmarked reliability: 999 URLs, 254 domains, 18 categories, 100% success and 2.5s median end-to-end latency.

benchmarks · browser-automation · comparisons

Spider MCP v2: browser automation for AI agents

Spider's MCP server now ships 21 tools, 9 of them browser automation, giving agents direct control of native cloud browsers that load protected pages completely, with proxy rotation built in.

MCP · AI · web-scraping

Build a production RAG pipeline with web data in under 30 minutes

Crawl a site with Spider, chunk the markdown, embed it, store it in a vector database and query it. Working code for LangChain, LlamaIndex, CrewAI and AutoGen.

AI · RAG · tutorial

Building AI agents that browse the web

Architecture patterns and working code for web-browsing agents: research, monitoring and extraction, built on CrewAI and AutoGen with Spider as the backend.

AI · agents · architecture

Building an MCP server for web scraping

Build an MCP server in TypeScript that wraps Spider's API, giving any AI model the ability to crawl, scrape, search and extract structured data.

AI · MCP · tutorial

Why patched headless browsers keep breaking on modern sites

What a demanding site checks before it serves a page, why a patched headless browser fails those checks within weeks, and why a native engine does not.

web-scraping · browser-automation · engineering

Choosing a web scraping stack in 2026

Does your scraping stack need a browser? In a November 2026 run a plain fetch returned at least 90% of a browser's words on 26 of 29 pages.

web-scraping · developers · comparisons

Crawl4AI vs Firecrawl vs Spider: 99.9% success on 1,000 URLs

In our February 2026 run on 1,000 URLs, Firecrawl beat Crawl4AI on every metric, 95.7% success to 90.4%. Spider returned 99.9% at 74 pages a second.

benchmarks · web-scraping · comparisons

Open source web scraping: why MIT license matters

How MIT, Apache 2.0, AGPL and BSL affect your right to build commercial products on scraping tools, and why Spider picked MIT.

open-source · web-scraping · engineering

Rust vs. Python for web scraping: why we rewrote everything

Why Spider rebuilt its scrapers in Rust, and a dated crawl of five documentation sites with the open source spider crate 2.53.9 that you can run again.

engineering · rust · performance

Scraping 1 million pages: what actually happens

An engineering log of crawling 1 million pages across 10,000 domains with Spider's API. Throughput curves, failure modes, cost breakdown, and lessons learned.

benchmarks · engineering · web-scraping

How Spider went to market: what worked and what we'd redo

How we built Spider's go-to-market from zero: the channels that worked, the pricing mistakes, and the content that actually converted.

engineering · startup · go-to-market

Data collection platforms for AI: 250 JSON rows from one call without a model

Data collection platforms for AI, compared by what they return. One Spider call returned 250 rows of four named fields as JSON in 1.4 seconds with no model.

AI · web-scraping · developers

Web scraping at scale: cost per 1,000 pages by kind of site

Web scraping cost per 1,000 pages, measured daily from Nov 6 to 17, 2026 on five kinds of page. Documentation, 25 fixed pages, ran from $0.274. Cost varies by site.

web-scraping · cost-analysis · developers

Web data into a vector database: the complete pipeline

Turn web pages into vectors in Pinecone, Weaviate, Chroma or pgvector: crawl, extract, chunk, embed and index, with working Python and a 10K-page cost breakdown.

AI · vector-databases · pipeline

Web scraping for AI training data: legal and technical guide 2026

What the EU AI Act, US fair use, and GDPR require of AI training data, and how to build a scraping pipeline that holds up in 2026.

AI · legal · web-scraping

Start crawling in 30 seconds.

One API key. No servers to manage.

2,500 free credits, worth $0.25, if your signup passes our checks. No card required.

Get an API keyRead the docs