Built with Spider.
Real integrations shipping in production, using Spider for web crawling and scraping in AI tooling and accessibility.
Used in production.
LangChain SpiderLoader
First-class document loader in langchain_community. Feed it any URL, get back LLM-ready markdown. Supports scrape and crawl modes with full Spider API parameters. Used by teams building RAG pipelines who need clean, structured web content without managing scrapers.
A11yWatch
Automated WCAG accessibility auditing platform. Spider replaced an entire crawling infrastructure of dedicated servers, headless browsers, and proxy rotation. Monthly cost went from $2,000 to $13 for the same workload. The engineering time freed up went back into building better audits.
Wired into the AI tools you already use.
LlamaIndex SpiderWebReader
Web reader integration for LlamaIndex. Crawls and scrapes URLs into LlamaIndex Document objects with configurable return formats, readability mode, and metadata extraction. Ships as part of llama-index-readers-web.
CrewAI SpiderTool
Official tool in the CrewAI multi-agent framework. Gives CrewAI agents the ability to scrape and crawl websites as part of their task execution. Agents can pull live web data mid-workflow without custom code.
Dify Spider Tool
Built-in scraper and crawler tool inside Dify, the open-source LLM app platform. Available in both Workflow and Agent modes. Configure your Spider API key in the Dify UI and any workflow can pull live web content.
Flowise Spider Node
Visual document loader node in Flowise, the low-code LLM flow builder. Drag and drop Spider into any workflow to scrape or crawl URLs. No code required. Connects directly to vector stores, text splitters, and LLM chains.
AutoGen Spider Notebook
Official AutoGen example showing how to wire Spider into multi-agent conversations. A scraper agent and crawler agent work together to extract and summarize web content. Part of the AutoGen 0.2 documentation.
Spider MCP Server
Model Context Protocol server that gives any MCP-compatible AI (Claude, etc.) the ability to crawl and scrape the web through Spider. Install it once, and your AI assistant can pull live web data during conversations.
Free account, single API call.
Every project above started the same way.
Built something with Spider?
Open source or commercial, big or small. Send us the link.