Spider Browser

A remote browser you control through code. Spider Browser handles stealth, anti-bot bypasses, CAPTCHA solving, and content extraction so you can focus on what to do with the data. Connect over WebSocket, navigate to any URL, and get back clean content, screenshots, or structured fields.

Quick Start

Install the SDK and connect to a remote browser in under a minute. Works with TypeScript, Python, and Rust.

Connect and Scrape in TypeScript

import { SpiderBrowser } from "spider-browser";

const spider = new SpiderBrowser({
  apiKey: process.env.SPIDER_API_KEY,
});

await spider.init();

// Navigate to any page, stealth is handled automatically
await spider.page.goto("https://www.example.com");

// Get the page content as clean text
const content = await spider.page.content();
console.log(content);

await spider.close();

Pro Tip:

Spider Browser connects to pre-warmed browser instances. There is no cold start penalty. Your first navigation is as fast as your hundredth.

Installation

Pick your language. Each SDK connects to the same remote browser infrastructure.

Install the SDK

# TypeScript / JavaScript
npm install spider-browser

# Python
pip install spider-browser

# Rust
cargo add spider-browser

Navigating Pages

Use page.goto() to navigate. Spider Browser automatically detects anti-bot challenges, rotates browser profiles, and solves CAPTCHAs. You get the loaded page back without writing any retry logic.

Navigation Methods

// Standard navigation, waits for DOM content
await spider.page.goto("https://www.amazon.com/dp/B0DGJHM7QN");

// Fast navigation, returns as soon as possible
await spider.page.goto_fast("https://news.ycombinator.com");

// Wait for full DOM readiness with a minimum content length
const html = await spider.page.content({
  wait_ms: 5000,
  min_length: 500,
});

Extracting Structured Data

Pull specific fields from any page with extractFields(). Pass CSS selectors and get back a clean JSON object. No parsing, no regex, no DOM traversal on your end.

Field Extraction

const product = await spider.page.extractFields({
  title: "#productTitle",
  price: ".a-price .a-offscreen",
  rating: "#acrPopover .a-icon-alt",
  reviews: "#acrCustomerReviewText",
  image: { selector: "#landingImage", attribute: "src" },
});

console.log(product);
// {
//   title: "Apple AirPods Pro 2",
//   price: "$189.99",
//   rating: "4.7 out of 5 stars",
//   reviews: "48,239 ratings",
//   image: "https://m.media-amazon.com/images/..."
// }

Natural Language Extraction

When you don't know the page structure ahead of time, use extract() with a plain English prompt. Spider Browser reads the page and returns what you asked for.

AI-Powered Extraction

const result = await spider.page.extract(
  "What is the product name, price, and return policy?"
);

console.log(result);
// "The product is the Apple AirPods Pro 2, priced at $189.99.
//  The return policy allows returns within 30 days of delivery."

Screenshots

Capture full-page or viewport screenshots. Returns a base64-encoded PNG.

Screenshots

// Full page screenshot
const screenshot = await spider.page.screenshot();

// Save to file
import fs from "fs";
fs.writeFileSync("page.png", Buffer.from(screenshot, "base64"));

Page Interactions

Click buttons, fill forms, and interact with page elements using act(). Describe what you want in plain English and Spider Browser figures out the DOM interactions.

Page Interactions

// Click, type, and interact with natural language
await spider.page.act("Click the 'Sign In' button");
await spider.page.act("Type 'hello world' into the search box and press Enter");

// Or use direct DOM methods
await spider.page.click("#submit-btn");
await spider.page.type("#email", "user@example.com");

Autonomous Agent Mode

For complex, multi-step workflows, use agent(). Give it a goal and Spider Browser autonomously navigates, clicks, fills forms, and extracts data across multiple pages. Works with any LLM backend.

Autonomous Agent

const result = await spider.page.agent(
  "Go to Hacker News, find the top 3 posts from today, " +
  "and return their titles, URLs, and point counts as JSON."
);

console.log(result);

Stealth and Anti-Bot

Spider Browser handles stealth automatically. Every session gets a unique browser fingerprint. The system rotates between Chromium, Firefox, and Safari engines to match what real users look like to anti-bot systems. You configure stealth levels on the connection, not per request.

Stealth Configuration

const spider = new SpiderBrowser({
  apiKey: process.env.SPIDER_API_KEY,
  stealth: 0,           // Start with maximum stealth
  maxStealthLevels: 3,  // Allow up to 3 escalation steps
  captcha: "solve",     // Automatically solve CAPTCHAs
  smartRetry: true,     // Retry with different strategies on failure
  hedge: true,          // Race multiple approaches for speed
});

Pro Tip:

Setting stealth: 0 starts with the strongest stealth profile. The system escalates only if needed, keeping requests fast for sites that don't require heavy protection.

Connection Options

Full reference for SpiderBrowserOptions.

SpiderBrowserOptions

interface SpiderBrowserOptions {
  apiKey: string;              // Your Spider API key
  serverUrl?: string;          // WebSocket server (default: wss://browser.spider.cloud)
  browser?: string;            // Browser type: "auto", "chrome", "firefox" (default: "auto")
  stealth?: number;            // Stealth level 0-3, lower = more stealth (default: 0)
  maxStealthLevels?: number;   // Max escalation steps (default: 3)
  captcha?: "solve" | "skip";  // CAPTCHA handling (default: "solve")
  smartRetry?: boolean;        // Auto-retry with different strategies (default: true)
  maxRetries?: number;         // Maximum retry attempts (default: 12)
  hedge?: boolean;             // Race multiple approaches (default: false)
  country?: string;            // Proxy country code, e.g. "US" (default: none)
  mode?: string;               // "scraping" or "automation" (default: "scraping")
  connectTimeoutMs?: number;   // Connection timeout in ms (default: 30000)
  commandTimeoutMs?: number;   // Command timeout in ms (default: 30000)
}

Python Quick Start

The Python SDK has the same API surface. Install with pip install spider-browser and use async/await.

Python Example

import asyncio
from spider_browser import SpiderBrowser, SpiderBrowserOptions

async def main():
    opts = SpiderBrowserOptions(
        api_key="your-api-key",
        stealth=0,
        captcha="solve",
    )

    async with SpiderBrowser(opts) as browser:
        page = browser.page

        await page.goto("https://www.example.com")
        content = await page.content()
        print(content)

        screenshot = await page.screenshot()
        print(f"Screenshot: {len(screenshot)} bytes")

asyncio.run(main())

Ready to try it? Get your API key and start scraping in minutes. Full SDK source and examples are on GitHub.