Skip to main content New gottem — one API for every web scraping vendor.
Open source

Built in public.

Spider runs on open-source repositories. The crawler, the browser engine, and the content pipeline are all on GitHub under the MIT license.

github.com/spider-rs ↗ Open source vs cloud
Repositories
18
GitHub stars
3.0k
License
MIT

Core

6
spider Rust

Low-latency web data collector. Concurrent crawling in Rust.

$ cargo add spider
2.5k
gottem Rust

One API for every web-scraping vendor. Tiered routing across HTTP, headless, and vendors.

$ cargo install gottem-cli
3
spider-browser TypeScript

Browser client for AI agents with stealth and smart retries.

$ npm install spider-browser
1
spider-py Python

Spider ported to Python. Rust crawler speed from a Python API.

$ pip install spider_rs
107
spider-nodejs JavaScript

Spider ported to Node.js. Native bindings for JavaScript.

$ npm install @spider-rs/spider-rs
59
spider-clients Multi

Python, JavaScript, and Rust client libraries for the Spider API.

26

Browser

3
headless-browser Rust

Headless browser setup for Fargate, Cloud Run, and Kubernetes.

84
chromey Rust

Chrome DevTools Protocol implementation for Rust.

45
spider-mcp JavaScript

Model Context Protocol server for Claude, Cursor, and Windsurf.

$ npx spider-cloud-mcp
2

Content

3
html2md Rust

HTML to Markdown for Rust. Built for LLM content preparation.

71
readability Rust

Readability extraction for Rust. Pulls the main content out of a page.

18
spider_transformations Rust

HTML transformation library for Rust.

2

Utilities

6
ua_generator Rust

Pre-compiled real User-Agents for Rust. Updated weekly.

39
spider_fingerprint Rust

Browser fingerprinting library for Rust.

15
auto-encoder Rust

Bytes-to-string encoding detection for Rust.

1
async_job Rust

Simple async cron jobs for Rust.

2
firewall Rust

Firewall utilities for Rust services.

1
proxier Rust

Auto proxy whitelisting for third-party services.

3
Skip the ops

Don't want to run it yourself?

Spider Cloud runs the same crawler with proxies, stealth, and auto-scaling handled. Free balance on signup. Top up later.

Get started free See what changes
See also OSS vs Cloud · Integrations · Docs · Why Spider