MCP Server for
Web Data
Your AI can crawl sites, scrape pages, search the web, and control a real browser. Point
your client at mcp.spider.cloud and it works in seconds.
Connect your tool
Two ways to set up. Pick one, paste the config, and you're done.
Claude Code
One commandClaude Desktop / Cursor / Windsurf
Add to your MCP config file
No dependencies. Spider handles the crawling, proxies, and browsers on our end.
22 tools, one connection
Your AI picks the right one for the job. Here's what it has to work with.
Core Tools
8spider_crawl Follow links and pull content from an entire site or section.
spider_scrape Grab one page. No link following, just the content you asked for.
spider_search Search the web, then optionally crawl whatever comes back.
spider_links List every link on a page without downloading anything else.
spider_screenshot Full-page or viewport screenshot, returned as base64.
spider_unblocker Get past bot detection walls that would block a normal request.
spider_transform Turn raw HTML into clean markdown locally. No network call.
spider_get_credits Check how many credits you have left.
AI Tools
5 SUBSCRIPTIONspider_ai_crawl Tell it what to crawl in plain English, like "only product pages".
spider_ai_scrape Describe the data you want and get structured JSON back.
spider_ai_search Web search that ranks results by what actually matters to your query.
spider_ai_browser Describe what to do in the browser and it figures out the clicks.
spider_ai_links Extract links with AI deciding which ones are relevant.
Browser Tools
9spider_browser_open Spin up a cloud browser with stealth and anti-bot built in.
spider_browser_navigate Go to a URL and wait for the page to finish loading.
spider_browser_click Click something on the page using a CSS selector.
spider_browser_fill Type into a form field.
spider_browser_screenshot Take a screenshot of what the browser sees right now.
spider_browser_content Pull the page HTML or just the visible text.
spider_browser_evaluate Run arbitrary JavaScript on the page.
spider_browser_wait_for Pause until a selector appears, a nav finishes, or the network settles.
spider_browser_close End the session and free up the browser.
Under the hood
Markdown, not HTML soup
Nav bars, ads, cookie banners, footers. All stripped. You get clean markdown that uses 60-80% fewer tokens than raw HTML, so the context window goes toward your actual task instead of boilerplate.
Sub-second responses
A single page comes back in under a second. Bulk crawls hit 100K+ pages per second. That matters because every slow tool call is dead time where your agent is waiting instead of thinking.
Anti-bot walls handled
Cloudflare, Akamai, PerimeterX, CAPTCHAs. Spider deals with all of it so your agent doesn't burn tool calls retrying blocked requests. In production we see a 99.9% success rate across protected sites.
Real browser, not curl
SPAs, lazy-loaded images, infinite scroll, client-rendered content. Spider runs a real browser so you get the page people actually see, not the empty shell a plain HTTP request gives you.
Browser you can drive
Open, navigate, click, type, screenshot, run JS, close. Your agent can log into dashboards, fill out forms, or walk through multi-step flows the way a person would. 9 tools, one browser session.
199+ country proxies
Residential, ISP, mobile, datacenter. Proxies rotate automatically and geo-restricted content just works. Pass a country code if you need a specific location, otherwise Spider picks the best route.
What it looks like
Two real examples. You ask a question, the AI figures out which tools to call.
Research how Stripe handles idempotency keys and summarize the approach.
I'll search for Stripe's idempotency documentation and crawl the relevant pages.
Stripe uses a header-based idempotency system. Here's how it works:
1. Client sends
Idempotency-Key
header with a unique UUID...
2. Server caches the response for 24 hours, keyed by that UUID...
3. Retries with the same key return the cached response, not a new charge...
Go to news.ycombinator.com, find the top post, and screenshot it for me.
I'll open a browser, navigate to Hacker News, grab the top story, and take a screenshot.
The top post on Hacker News right now is "Show HN: ...". Here's the screenshot of the linked article:
[screenshot image displayed]