Give Your AI Agents
Real-Time Web Access
AI agents need to interact with the real world. Spider gives your agents the ability to crawl, scrape, search, and extract web data at 100K+ pages per second — through a simple API or MCP server.
Without Spider
- Agents can't access real-time web data
- Building browser automation is complex and fragile
- Anti-bot protections block naive HTTP requests
- Raw HTML output is noisy and wastes context tokens
With Spider
- Agents get clean, token-efficient markdown from any URL
- MCP server for native integration with Claude, Cursor, Windsurf
- Anti-bot bypass with 99.5% success rate
- Native SDKs for LangChain, CrewAI, Agno, and AutoGen
How Agents Use Spider
MCP Server
Install spider-cloud-mcp for native web access in Claude, Cursor, and Windsurf. One command setup.
REST API
Simple JSON API with SDKs for Python, Node.js, Rust, and Go. Your agent calls one endpoint, gets clean data back.
Framework Integrations
Drop-in tools for LangChain, CrewAI, Agno, AutoGen, and FlowiseAI. Use Spider as a document loader or agent tool.
Web Search
Real-time search API lets agents find information across the web and optionally crawl the results.
AI Extraction
Natural language prompts to extract structured data. Agents describe what they need, Spider returns clean JSON.
Browser Automation
Agents can interact with web pages — click, fill forms, scroll — using natural language instructions.
Quick Start with MCP
# Claude Code
claude mcp add spider -- npx -y spider-cloud-mcp
# Or add to Claude Desktop / Cursor config:
{
"mcpServers": {
"spider": {
"command": "npx",
"args": ["-y", "spider-cloud-mcp"],
"env": {
"SPIDER_API_KEY": "your-api-key"
}
}
}
}Framework Examples
from crewai_tools import SpiderTool
spider = SpiderTool()
agent = Agent(
role="Research Analyst",
tools=[spider],
goal="Find and summarize data",
)from langchain_community.document_loaders \
import SpiderLoader
loader = SpiderLoader(
url="https://docs.example.com",
mode="crawl",
)
docs = loader.load()Related Resources
Ready to give your agents web access?
Start crawling the web from your AI agent in minutes.