AI agents

Crawl, search, and extract as agent tools.

Spider lets your agents crawl, search, and extract through a single API or MCP server. Returns clean markdown that fits in context windows without burning tokens on HTML.

Get started free MCP server setup

MCP tools your agent gets Live

spider_crawl Follow links across a site, return content.
spider_scrape Grab one page, no link following.
spider_search Search the web, optionally crawl results.
spider_links List every link on a page.
spider_screenshot Full-page or viewport screenshot.
spider_transform Convert HTML to markdown locally.

+ AI tools · spider_ai_crawl / scrape / search / browser

01 · MCP quick start

One command to web-enable your AI.

Add Spider as an MCP server and your AI tool gets crawling, search, and extraction out of the box. No wrapper code needed.

Claude Cursor Windsurf Claude Code

Install npx

# Claude Code
claude mcp add spider -- npx -y spider-cloud-mcp

Or add to Claude Desktop / Cursor config mcp.json

{
  "mcpServers": {
    "spider": {
      "command": "npx",
      "args": ["-y", "spider-cloud-mcp"],
      "env": {
        "SPIDER_API_KEY": "your-api-key"
      }
    }
  }
}

02 · Capabilities

What your agent gets out of the box.

Crawl Core

Read entire websites

Submit a URL, get clean markdown for every page. No HTML parsing. Output is token-efficient and fits directly in context.

Search Real-time

Query the live web

Search the web and optionally crawl results in the same request. Ground agent responses in live data instead of training-cutoff knowledge.

Extract AI

Pull structured data

Set extra_ai_data: true with a custom_prompt and Spider extracts data matching your description. Or use css_extraction_map for CSS/XPath selectors.

Actions Browser

Interact with pages

The Browser API provides a CDP WebSocket connection: navigate, click, type, scroll, capture screenshots. Use the spider_ai_browser MCP tool for natural-language automation.

MCP Protocol

Native tool for AI IDEs

Install the MCP server and Claude, Cursor, or Windsurf can call Spider directly. One command setup, zero config files to maintain.

Frameworks SDK

Drop-in agent tools

Published integrations for LangChain (document loader) and CrewAI (SpiderTool). The REST API and Python/Node SDKs work with any framework.

03 · Framework integrations

Drop into the framework you already use.

CrewAI Python

from crewai_tools import SpiderTool

spider = SpiderTool()

agent = Agent(
    role="Research Analyst",
    tools=[spider],
    goal="Find and summarize data",
)

LangChain Python

from langchain_community.document_loaders \
    import SpiderLoader

loader = SpiderLoader(
    url="https://docs.example.com",
    mode="crawl",
)
docs = loader.load()

04 · Resources

Keep reading.

Guide

Your agent is one command from the live web.

Add Spider as an MCP tool and start crawling. No credit card required.

claude mcp add spider -- npx -y spider-cloud-mcp

Get started free MCP setup guide