Notes from the engineering team.
Writeups, benchmarks, and field notes on web data collection and AI infrastructure.
Latest 1 day ago
Fetching the web is the cheap part of an agent's research bill
We priced the fetch and the page input for five agent research tasks on 2026-11-18. Fetching ten pages was the small line. Reading them as raw HTML took 6.5x to 22.5x the tokens of markdown on four tasks and 1.8x on the fifth.
Archive
How to scrape a website, step by step
How to scrape a website, from reading robots.txt to saving a crawl one page at a time, with the curl command for each step and what it returned.
Proxies for web scraping, built into every request
Proxies for web scraping come built into every Spider request and rotate on their own. What they cost, and how to pick the exit country or use your own.
Web scraping for AI agents: search, scrape or crawl
Web scraping for AI agents: when an agent should search, scrape or crawl, and how many tokens of its context window each call fills, counted on real pages.
Web scraping to markdown: a whole docs site into files
Web scraping to markdown with one crawl call and a short Python script that writes a file per page, then a check of the code blocks and tables it kept.
Web scraping with a headless browser, without running one
How to scrape pages that build their text with JavaScript, using a browser on Spider's side, and how to tell which pages need one and what a render costs.
Web scraping with JavaScript and Node.js
Web scraping with JavaScript: a Node.js script with one package that scrapes a page as markdown, pulls a field by CSS selector and streams a crawl to a file.
Free web scraper: three ways to scrape a page without paying
Scrape a page for free three ways: in the browser with no key, from curl with no key, or with the MIT-licensed spider CLI on your own machine.
How web scraping works: what a plain fetch gets and misses
How web scraping works, from the HTTP request to the markdown you keep, and how to tell when a page needs a browser to run its scripts first.
Is web scraping legal? What 67 top sites ask for in robots.txt
Is web scraping legal? US and EU rulings and statutes by name and year, beside the robots.txt of 67 top sites. 13 hold Disallow: / for unnamed crawlers.
Web scraping vs web crawling: when to crawl and when to scrape
The difference between web scraping and web crawling, timed on one docs site, and which call to use for a URL list, a whole site or only a question.
Web scraping with Python using only the standard library
A Python web scraper that uses only the standard library and gets each page back as markdown. The whole script, the pages we ran it on and what it returned.
What is a web scraping API? One request, read field by field
What a web scraping API does between your request and its answer, why a 200 call can carry a 404 page, and five things to check in any scraping API.
Build a fine-tuning dataset from the web in two API calls
One search call and one crawl call turned a topic into 396 training records for $0.05. Every number comes from a run on 60 public documentation pages.
What a web page costs in tokens, measured on ten public pages
Raw HTML, markdown, readability and text output for the same ten pages, counted in o200k tokens. Markdown cut the bill 15x at the median, and readability mode left four index pages with under 60 tokens.
Spider Peers: earn from the internet you already pay for
Install Spider Peers on a computer, phone or TV and earn from your spare home bandwidth. Set it up once and it runs in the background.
Spider Shield: a free blocker for ads, trackers, phishing and malware
Spider Shield blocks ads, trackers, phishing and malware in every app on Mac, Windows, Linux and Android. It's free and needs no account.
Firecrawl alternatives: Spider at 99.9% success, 4.6x throughput
Spider finished 1,000 URLs at 99.9% success to Firecrawl's 95.7%, with 4.6x the throughput. It costs from $0.16 per 1,000 pages. Cost varies by site.
Flat-rate web scraping vs. pay-as-you-go: which to put where
How the flat-rate Unlimited plan puts steady high-volume crawling on a fixed bill, where pay-as-you-go still wins, and how both run on one API key.
Command line web scraping: 100 pages in about 3 seconds
Three Spider CLI runs each crawled 100 FastAPI docs pages to markdown in about 3 seconds. Install it, add a key, then pipe the JSON to jq or write a WARC file.
Spider Browser passes 80/80 on Browser Fidelity Bench
Spider Browser passed all 80 tasks on Browser Use's Browser Fidelity Bench V1 in September, up from 68 in March. Every other cloud browser tested scored lower.
Keep RAG context fresh with real-time web search
Static knowledge bases go stale in days. Build a RAG pipeline on live web data by adding web search at query time, with LangChain and plain Python examples.
Web search API for AI agents: search and extract in one call
Spider's Search API searches, crawls and cleans in one request, so one agent covers live web search, deep research and page monitoring without glue code.
Introducing Silk: our custom AI model for web data extraction
Silk, Spider's own extraction model, turns raw HTML into structured data and handles challenge pages on our GPUs, with no outside API calls or per-token billing.
Case study: how a RAG pipeline went from 6 hours to 15 minutes
A Series A AI company replaced three Python microservices, a proxy provider, and half an engineer's time with a single Spider API call. This is what changed.
Best web scraping APIs for AI and LLM pipelines in 2026
On 60 docs pages, markdown took 3,369 tokens a page and raw HTML 123,813. Seven web scraping APIs for RAG and LLMs, with 100,000-page costs where one is listed.
Oxylabs Web Scraper API pricing by plan, compared to Spider
Oxylabs lists unrendered general sites at $1.00 per 1,000 results on Starter, $0.75 on Enterprise. Spider: from $0.16 per 1,000 static pages on Nov 17, varies by site.
Apify alternative 2026: Spider vs Apify pricing and CUs
What each Apify plan buys in compute units, why proxy bandwidth and Actor memory push the bill higher, what Spider costs per kind of page, and a migration example.
From $0.16 per 1,000 pages: a Bright Data alternative
Spider: from $0.16 per 1,000 pages, Nov 17 static pages. Cost varies by site. Bright Data Web Unlocker list price, Nov 17, 2026: $1.50 pay as you go, $1.00 on Scale.
ZenRows alternative? From $0.16 per 1,000 pages, no plan
Spider costs from $0.16 per 1,000 pages on Nov 17 static pages. Cost varies by site. ZenRows' pricing page that day lists Build at $19 a month for 45,000 credits.
Crawl4AI vs Spider in 2026: self-hosted vs managed
Crawl4AI is free Python you run yourself. Spider is a managed Rust crawler. The same 1,000 URLs on both, what self-hosting really costs, and code for each.
Firecrawl vs Spider 2026: MCP crawl speed and pricing
Which web crawl MCP has the better crawl speed? On our 1,000-URL run Spider averaged 74 pages a second to Firecrawl's 16. Pricing and licenses compared.
Jina Reader alternative that crawls a whole site in one call
How r.jina.ai works, its X-Return-Format header and its 500 requests a minute on a paid key. Spider crawls a site in one call and starts at 10,000 a minute.
Scrapfly vs Spider in 2026: credit multipliers and cost
Scrapfly's 2026 plans and credit multipliers worked through at 10K, 100K and 1M pages against Spider usage billing, plus the independent success-rate numbers.
Crawlera alternative 2026: Spider vs Zyte API pricing
Crawlera is now Zyte API, billed per request by a site tier Zyte assigns. What those bands cost, how Spider prices the same crawl, and a migration example.
NetNut alternatives in 2026: what to use after the seizure
NetNut alternatives after the July 2, 2026 FBI seizure: what to swap in this week, a two-line proxy migration, and when to replace the whole stack.
ScraperAPI pricing, free plan and an alternative with no plan
ScraperAPI plans list at $49 to $1,975 a month, rendering at 10 credits a page. Spider with no plan: from $0.16 per 1,000 static pages on Nov 17, varying by site.
ScrapingBee alternative billed from $0.16 per 1,000 pages
ScrapingBee's rendered default costs $0.37 to $1.27 per 1,000 on a fully used standard plan. Spider ran from $0.16 per 1,000 static pages on Nov 17. Cost varies by site.
Spider Browser vs. Kernel vs. Browserbase: 999 URLs benchmarked
Kernel benchmarked cold start. We benchmarked reliability: 999 URLs, 254 domains, 18 categories, 100% success and 2.5s median end-to-end latency.
Spider MCP v2: browser automation for AI agents
Spider's MCP server now ships 21 tools, 9 of them browser automation, giving agents direct control of native cloud browsers that load protected pages completely, with proxy rotation built in.
Build a production RAG pipeline with web data in under 30 minutes
Crawl a site with Spider, chunk the markdown, embed it, store it in a vector database and query it. Working code for LangChain, LlamaIndex, CrewAI and AutoGen.
Building AI agents that browse the web
Architecture patterns and working code for web-browsing agents: research, monitoring and extraction, built on CrewAI and AutoGen with Spider as the backend.
Building an MCP server for web scraping
Build an MCP server in TypeScript that wraps Spider's API, giving any AI model the ability to crawl, scrape, search and extract structured data.
Why patched headless browsers keep breaking on modern sites
What a demanding site checks before it serves a page, why a patched headless browser fails those checks within weeks, and why a native engine does not.
Choosing a web scraping stack in 2026
Does your scraping stack need a browser? In a November 2026 run a plain fetch returned at least 90% of a browser's words on 26 of 29 pages.
Crawl4AI vs Firecrawl vs Spider: 99.9% success on 1,000 URLs
In our February 2026 run on 1,000 URLs, Firecrawl beat Crawl4AI on every metric, 95.7% success to 90.4%. Spider returned 99.9% at 74 pages a second.
Open source web scraping: why MIT license matters
How MIT, Apache 2.0, AGPL and BSL affect your right to build commercial products on scraping tools, and why Spider picked MIT.
Rust vs. Python for web scraping: why we rewrote everything
Why Spider rebuilt its scrapers in Rust, and a dated crawl of five documentation sites with the open source spider crate 2.53.9 that you can run again.
Scraping 1 million pages: what actually happens
An engineering log of crawling 1 million pages across 10,000 domains with Spider's API. Throughput curves, failure modes, cost breakdown, and lessons learned.
How Spider went to market: what worked and what we'd redo
How we built Spider's go-to-market from zero: the channels that worked, the pricing mistakes, and the content that actually converted.
Data collection platforms for AI: 250 JSON rows from one call without a model
Data collection platforms for AI, compared by what they return. One Spider call returned 250 rows of four named fields as JSON in 1.4 seconds with no model.
Web scraping at scale: cost per 1,000 pages by kind of site
Web scraping cost per 1,000 pages, measured daily from Nov 6 to 17, 2026 on five kinds of page. Documentation, 25 fixed pages, ran from $0.274. Cost varies by site.
Web data into a vector database: the complete pipeline
Turn web pages into vectors in Pinecone, Weaviate, Chroma or pgvector: crawl, extract, chunk, embed and index, with working Python and a 10K-page cost breakdown.
Web scraping for AI training data: legal and technical guide 2026
What the EU AI Act, US fair use, and GDPR require of AI training data, and how to build a scraping pipeline that holds up in 2026.
Start crawling in 30 seconds.
One API key. No servers to manage.
2,500 free credits, worth $0.25, if your signup passes our checks. No card required.