Skip to main content
AI Studio  add-on for Spider.

From first crawl to production.

Each guide walks one job end to end, in code that runs as-is. Follow a track in order, or jump straight to the task in front of you.

20 guides 5 tracks Updated Mar 2026
Start here How to crawl and scrape a website with an API Crawl and scrape any website with the Spider API. Create a key, run your first crawl, then tune request modes, proxies, streaming, and markdown for LLMs. 5 min read

Fundamentals

Crawl, scrape, search, and what each one costs.

3 guides · ~13 min
  1. 01 Spider API: crawl, scrape & search the web Spider API turns any URL into markdown, HTML, or JSON in one request. See endpoints, code examples, and how to start crawling or scraping the web today.
  2. 02 Web search API (SERP) to search and scrape Query the web and scrape every result in one call with Spider's SERP API. Covers parameters, geo-targeting, time filters, pagination, and code examples.
  3. 03 Spider pricing and plans: what you pay and why A clear breakdown of Spider's pay-as-you-go pricing, per-endpoint costs, volume discounts, and how billing actually works for web scraping and browser automation.

AI & agents

Feed crawls into LLMs, agent frameworks, and RAG pipelines.

6 guides · ~49 min
  1. 01 Build an AI agent from scratch in Python Build an AI agent from scratch in Python. It writes its own search queries, searches the web with Spider, judges the results, and refines until it has an answer.
  2. 02 LangChain + Groq + Spider integration guide Crawl multiple URLs with Spider's LangChain loader, then summarize the results with Groq and Llama 3.
  3. 03 Scrape & crawl agent with Microsoft's Autogen Set up an Autogen agent that scrapes and crawls websites using the Spider API.
  4. 04 Stock research assistant using crewAI and Spider Build a crewAI research pipeline that uses Spider to scrape financial data and write stock analysis reports.
  5. 05 Speedy resilient web scraper for RAG AI: part 1 Choosing your scraper, cleaning HTML for RAG, deduplicating content, and testing on a single site before scaling up.
  6. 06 Speedy resilient web scraper for RAG AI: part 2 Scaling web scraping for RAG pipelines. Error-first design, retry strategies, and handling failures at volume.

Harder targets

Login walls, proxy routing, and headless Browser at scale.

3 guides · ~7 min
  1. 01 Crawling authenticated pages behind a login Crawl authenticated pages behind a login wall with Spider. Pass a session cookie, or run an execution script that submits the login form, then scrape the result.
  2. 02 Spider Proxy - rotating HTTP, HTTPS & SOCKS5 IPs Point any HTTP, HTTPS, or SOCKS5 client at the Spider proxy: API key as the username, API parameters as the password. Rotating datacenter and residential IPs.
  3. 03 Scaling headless Chrome for high-performance applications Practical strategies for scaling headless Chrome, from container orchestration to Rust-based CDP handlers and ALB configuration.

Applied workflows

Build lead lists, outreach, search, and archives on top of the API.

5 guides · ~16 min
  1. 01 Extract leads Extract contact information from any website using Spider's AI-powered pipeline. Emails, phone numbers, and more.
  2. 02 Automated cold email outreach using Spider Extract company info from inbound emails, scrape their website with Spider, and generate personalized replies with RAG.
  3. 03 Discord real-time data retrieval Set up Spider Bot on your Discord server to fetch and analyze web data using slash commands.
  4. 04 Set up automated free website static search Add full-text static search to any website using Spider and Pagefind.
  5. 05 Website archiving Archive web pages with Spider. Capture full page resources, automate regular crawls, and store content for long-term access.

Open source & self-hosting

Run the Rust crawler yourself, and the OSS projects behind Spider.

2 guides · ~10 min
  1. 01 Self-hosting Spider: using the open source Rust crate Build your own web crawler with the open source spider Rust crate. Quick start, Docker setup, configuration, and when to upgrade to the cloud API.
  2. 02 The Spider open source ecosystem A guide to all open source Spider projects: the core crawler, browser client, HTML transformer, TLS fingerprinting, and more. Quick-start examples for each.

Start crawling in 30 seconds.

One API key. No servers to manage.

Free balance on signup · No card required

Get started freeRead the docs