Guides · Task manual
From first crawl to production.
Each guide walks one job end to end, in code that runs as-is. Follow a track in order, or jump straight to the task in front of you.
20 guides 5 tracks Updated Mar 2026
Track 01
Fundamentals
The API surface end to end — crawl, scrape, search, and what it costs.
3 guides · ~12 min
- 01Spider API: Crawl, Scrape & Search the WebSpider API turns any URL into markdown, HTML, or JSON in one request. See endpoints, code examples, and how to start crawling or scraping the web today.
- 02Web Search API (SERP) - Search and ScrapeQuery the web and scrape results in one call with Spider's web search API. Covers parameters, geo-targeting, time filters, pagination, and code examples.
- 03Spider Pricing and Plans: What You Pay and WhyA clear breakdown of Spider's pay-as-you-go pricing, per-endpoint costs, volume discounts, and how billing actually works for web scraping and browser automation.
Track 02
AI & agents
Feed crawls into LLMs, agent frameworks, and RAG pipelines.
6 guides · ~49 min
- 01Build an AI Agent from ScratchBuild a research agent that searches the web with Spider, evaluates results, and forms answers with OpenAI.
- 02LangChain + Groq + Spider Integration GuideCrawl multiple URLs with Spider's LangChain loader, then summarize the results with Groq and Llama 3.
- 03Scrape & Crawl Agent with Microsoft's AutogenSet up an Autogen agent that scrapes and crawls websites using the Spider API.
- 04Stock Research Assistant Using crewAI and SpiderBuild a crewAI research pipeline that uses Spider to scrape financial data and write stock analysis reports.
- 05Speedy Resilient Web Scraper for RAG AI: Part 1Choosing your scraper, cleaning HTML for RAG, deduplicating content, and testing on a single site before scaling up.
- 06Speedy Resilient Web Scraper for RAG AI: Part 2Scaling web scraping for RAG pipelines. Error-first design, retry strategies, and handling failures at volume.
Track 03
Harder targets
Login walls, proxy routing, and headless Chrome at scale.
3 guides · ~7 min
- 01Crawling Authenticated PagesTwo methods for crawling pages behind login walls: cookies and execution scripts.
- 02Proxy Mode - HTTP, HTTPS & SOCKS5 EndpointsRoute scraping traffic through the Spider proxy over HTTP, HTTPS, or SOCKS5. Rotating datacenter and residential proxies, same credits as the standard API.
- 03Scaling Headless Chrome for High-Performance ApplicationsPractical strategies for scaling headless Chrome, from container orchestration to Rust-based CDP handlers and ALB configuration.
Track 04
Applied workflows
Complete tasks built on the API — leads, outreach, search, archives.
5 guides · ~16 min
- 01Extract LeadsExtract contact information from any website using Spider's AI-powered pipeline. Emails, phone numbers, and more.
- 02Automated Cold Email Outreach Using SpiderExtract company info from inbound emails, scrape their website with Spider, and generate personalized replies with RAG.
- 03Discord Real-Time Data RetrievalSet up Spider Bot on your Discord server to fetch and analyze web data using slash commands.
- 04Set Up Automated Free Website Static SearchAdd full-text static search to any website using Spider and Pagefind.
- 05Website ArchivingArchive web pages with Spider. Capture full page resources, automate regular crawls, and store content for long-term access.
Track 05
Open source & self-hosting
Run the Rust crawler yourself, and the OSS projects behind Spider.
2 guides · ~10 min
- 01Self-Hosting Spider: Using the Open Source Rust CrateBuild your own web crawler with the open source spider Rust crate. Quick start, Docker setup, configuration, and when to upgrade to the cloud API.
- 02The Spider Open Source EcosystemA guide to all open source Spider projects: the core crawler, browser client, HTML transformer, TLS fingerprinting, and more. Quick-start examples for each.
Get started
Start crawling in 30 seconds.
One API key. No servers to manage.
Free balance on signup · No card required