Spider MCP v2: browser automation for AI agents

An agent can reason its way through a task and then stall at a login form, because reading a page and operating one are different jobs. Clicking, typing, and waiting for a redirect all need a browser. Spider MCP v2 gives any MCP client, Claude, Cursor, or Windsurf, 22 tools for the web, 9 of them new browser automation tools wired to Spider’s cloud browsers.

There is no local Chrome install, no Selenium setup, and no CAPTCHA wall to get through. Run npx spider-cloud-mcp and your agent can browse the web.

What changed from v1 to v2

v1 shipped 13 tools, 8 core REST API tools and 5 AI tools. Every one was a one-shot HTTP call. Send a URL, get content back. That covers most scraping and crawling, and it falls apart the moment a task takes more than one step.

Pulling data out of a dashboard behind a login takes seven:

1. Navigate to the login page
2. Fill in the email field
3. Fill in the password field
4. Click the submit button
5. Wait for the redirect
6. Navigate to the reports page
7. Extract the data

v1 could not do any of that. The 9 new browser tools hold a remote session open across calls, so the agent can work through steps like these one at a time.

The tools

The core and AI tools are unchanged from v1. Every existing workflow keeps working.

The browser tools

Each browser tool operates on a session. You open a session, perform actions, then close it. Sessions run in Spider Cloud, so the agent never needs a local browser.

Opening a session

spider_browser_open: {
  browser: "chrome",
  stealth: 2,
  country: "us",
  mode: "scraping"
}

This returns a session_id that every later browser call needs. stealth sets proxy quality from 0 to 3, higher for sites that fight back harder. mode picks between scraping, which is headless, fast, and cheap, and cua, which renders fully and supports screenshots and video.

Sessions auto-close after 5 minutes of inactivity. You can have up to 5 concurrent sessions.

Navigation and content

spider_browser_navigate: {
  session_id: "abc-123",
  url: "https://example.com",
  wait_until: "networkidle0"
}

The wait_until parameter supports four modes: load (default), domcontentloaded (faster), networkidle0 (waits for all requests to finish), and networkidle2 (allows up to 2 in-flight requests). Pick networkidle0 for SPAs that load data asynchronously.

To read the page after navigating:

spider_browser_content: {
  session_id: "abc-123",
  format: "text"
}

Returns the visible text content. Use format: "html" for the full DOM.

Interacting with elements

Click a button:

spider_browser_click: {
  session_id: "abc-123",
  selector: "button[type='submit']"
}

Fill a form field (clears existing text first):

spider_browser_fill: {
  session_id: "abc-123",
  selector: "input[name='email']",
  value: "user@example.com"
}

Both tools wait for the element to appear before acting, with a configurable timeout. If the selector doesn’t match anything within 10 seconds (default), the tool returns an error instead of hanging.

Screenshots

spider_browser_screenshot: {
  session_id: "abc-123",
  full_page: true
}

Returns a base64 PNG as an MCP image content block, so the agent sees the page rather than a description of it. That is what makes “did the form actually submit?” answerable without a human looking.

You can also screenshot a specific element:

spider_browser_screenshot: {
  session_id: "abc-123",
  selector: "#chart-container"
}

Running JavaScript

spider_browser_evaluate covers whatever the other tools do not:

spider_browser_evaluate: {
  session_id: "abc-123",
  expression: "document.querySelectorAll('.product-card').length"
}

The expression runs in the page context with full DOM access. Use it for scrolling (window.scrollBy(0, 1000)), complex data extraction, or triggering custom events.

Waiting for dynamic content

SPAs and dynamic pages need explicit waits after interactions:

spider_browser_wait_for: {
  session_id: "abc-123",
  selector: ".results-loaded"
}

Three wait modes: selector (element appears in DOM), navigation (page navigates), or neither (defaults to network idle, meaning no requests for 500ms).

Closing sessions

spider_browser_close: {
  session_id: "abc-123"
}

Always close sessions when done. Open sessions consume resources and credits. The MCP server also closes all sessions on shutdown and cleans up idle sessions automatically, but explicit cleanup is the right pattern.

How it works

The browser tools use spider-browser, our TypeScript CDP client, to reach browser.spider.cloud over a WebSocket speaking the Chrome DevTools Protocol. Nothing installs a Chrome binary locally. spider-browser is a protocol client and nothing else.

AI Agent ─── MCP ──→ Spider MCP Server ─── CDP/WebSocket ──→ browser.spider.cloud

Puppeteer and Playwright connect to the same endpoint. The MCP server splits it into discrete tool calls so an agent never touches a WebSocket or a raw CDP command.

Each session gets its own isolated browser context. Cookies, storage, and state don’t leak between sessions.

Session lifecycle

1. spider_browser_open connects over the CDP WebSocket and stores the session.
2. Each later tool call looks up the session by ID, refreshes the idle timer, runs the action, and returns the result.
3. spider_browser_close disconnects and removes the session.
4. The server closes any session idle for 5+ minutes.
5. On SIGINT/SIGTERM the server closes every session.

Speed, cost, reliability

Every one of the 22 tools gets judged on these three.

Speed

The REST API tools inherit Spider’s crawling speed: 100K+ pages per second for spider_crawl, sub-second responses for spider_scrape. The request: "smart" default auto-detects whether a page needs JavaScript rendering and picks HTTP or Chrome accordingly. Most pages don’t need Chrome, so most requests complete in the fast path.

For browser sessions, spider_browser_open connects to a cloud browser without a cold-start penalty. Navigation speed depends on the target site, not on Spider.

Cost

Core tools are pay-per-use credits. No subscription, no monthly minimum. Check your balance anytime with spider_get_credits. Credit costs scale with page complexity and whether JavaScript rendering is needed. HTTP-only requests are cheapest, Chrome rendering costs more, and premium proxies multiply the base cost. See spider.cloud/credits/new for exact pricing.

Browser sessions are metered per-second based on bandwidth. The stealth parameter controls proxy quality and cost. Level 1 is cheapest, and level 3 uses premium mobile proxies for the hardest-to-access sites.

AI tools need a subscription and remove the extraction logic, CSS selectors, and automation scripts you would otherwise write. On a one-off job that trade is usually worth it.

Reliability

Every browser session carries anti-bot protection, tuned by the same stealth tier.

For a single protected page with no follow-up steps, spider_unblocker is the better call. It does fingerprinting, proxy rotation, and retries in one request.

Pick the engine in spider_browser_open. Underneath, every session runs on the Spider Browser runtime, which drives Chromium, Gecko, and WebKit in one process and can present itself as Chrome, Firefox, Safari, Brave, or Edge. Each profile is built from the engine that renders the page, so nothing about it contradicts anything else, which is the failure mode that gives away a patched headless Chrome. That runtime scored 85% on Browser Use’s stealth benchmark, the highest of any cloud browser tested.

Getting started

Install

claude mcp add spider -- npx -y spider-cloud-mcp

Or for Claude Desktop, add to your config:

{
  "mcpServers": {
    "spider": {
      "command": "npx",
      "args": ["-y", "spider-cloud-mcp"],
      "env": {
        "SPIDER_API_KEY": "your-api-key"
      }
    }
  }
}

Get your API key at spider.cloud/api-keys.

Try it

Once connected, ask your AI agent:

• “Scrape spider.cloud and give me the pricing details” runs spider_scrape
• “Search for recent papers on retrieval-augmented generation” runs spider_search
• “Open a browser, go to Hacker News, and get the top 10 story titles” runs spider_browser_*

The agent picks the right tool based on the task. One-shot content retrieval uses the REST tools. Multi-step interaction uses the browser tools. You don’t have to specify which.

Workflows

Monitoring a competitor’s pricing page

1. spider_scrape: {
     url: "https://competitor.com/pricing",
     return_format: "markdown",
     cache: { maxAge: 3600 }
   }

One call. Returns clean markdown with all plan names, prices, and feature lists. The cache parameter avoids redundant requests if you check multiple times per hour.

Filling out a web form

1. spider_browser_open: { mode: "cua" }
2. spider_browser_navigate: { url: "https://forms.example.com/apply" }
3. spider_browser_fill: { selector: "#name", value: "Acme Corp" }
4. spider_browser_fill: { selector: "#email", value: "contact@acme.com" }
5. spider_browser_click: { selector: "select#industry" }
6. spider_browser_evaluate: {
     expression: "document.querySelector('select#industry').value = 'technology'"
   }
7. spider_browser_click: { selector: "button[type='submit']" }
8. spider_browser_wait_for: { selector: ".confirmation-message" }
9. spider_browser_content: { format: "text" }
10. spider_browser_close: {}

The agent works the form one step at a time. A missing element or a navigation timeout comes back as a described error rather than a hang, which is what lets the agent recover on its own, usually by retrying a different selector or screenshotting the page to see what it is looking at.

Building a RAG pipeline

1. spider_crawl: {
     url: "https://docs.example.com",
     limit: 200,
     return_format: "markdown",
     filter_output_main_only: true,
     readability: true
   }

Returns up to 200 pages of clean markdown with navigation, ads, and boilerplate stripped. Feed the output directly into your vector database or RAG pipeline.

Extracting structured data without selectors

1. spider_ai_scrape: {
     url: "https://news.ycombinator.com",
     prompt: "Extract the top 30 stories as JSON with fields: rank, title, url, points, author, comment_count"
   }

No CSS selectors, no DOM inspection. The model reads the page structure and returns JSON. This is the one to reach for when the markup is messy, inconsistent, or deliberately obfuscated.

Architecture

The MCP server is 4 TypeScript files totaling ~870 lines:

Three runtime dependencies: @modelcontextprotocol/sdk, spider-browser, and zod. No bundled browser binary. The package is 21KB compressed.

Every tool handles errors the same way. Try the operation, catch anything thrown, return it as a structured MCP error with isError: true. A failed tool call never takes the server down. The server truncates JSONL responses at 200K characters so a large crawl cannot blow out the context window; if you hit that, narrow the result set with limit.

When to use which tool

Default to the REST tools. They are faster and cheaper because nothing holds a browser open. Reach for the browser tools when the task needs state carried between calls, which means login flows, multi-page forms, and anything where step 4 depends on what step 3 returned.

The AI tools sit in between. They take natural language but still run as one-shot HTTP calls. spider_ai_scrape gets you structured data without selectors. spider_ai_browser gets you automation without spelling out each click and fill.

Source code

The full source is on GitHub: spider-rs/spider-cloud-mcp-v2

Install with:

npx -y spider-cloud-mcp

Or add as an MCP server to Claude Code:

claude mcp add spider -- npx -y spider-cloud-mcp