Open source crawlers, built in public.

Spider runs on open source code. The Rust web crawler, the browser engine, and the scraping and markdown libraries are all on GitHub under the MIT license.

github.com/spider-rs ↗ Open source vs cloud

at a glance

Repositories: 18
GitHub stars: 3.2k
License: MIT

repositories

Crawlers and clients

spider Rust

Low-latency web data collector. Concurrent crawling in Rust.

$ cargo add spider

2.7k

Rust

One API for every web-scraping vendor. Tiered routing across HTTP, headless, and vendors.

$ cargo install gottem-cli

spider-browser TypeScript

Browser client for AI agents with stealth and smart retries.

$ npm install spider-browser

spider-py Python

Spider ported to Python. Rust crawler speed from a Python API.

$ pip install spider_rs

109

spider-nodejs JavaScript

Spider ported to Node.js. Native bindings for JavaScript.

$ npm install @spider-rs/spider-rs

spider-clients Multi

Python, JavaScript, and Rust client libraries for the Spider API.

repositories

Browser automation

headless-browser Rust

Headless browser setup for Fargate, Cloud Run, and Kubernetes.

chromey Rust

Chrome DevTools Protocol implementation for Rust.

spider-mcp JavaScript

Model Context Protocol server for Claude, Cursor, and Windsurf.

$ npx spider-cloud-mcp

repositories

Content extraction

html2md Rust

HTML to Markdown for Rust. Built for LLM content preparation.

readability Rust

Readability extraction for Rust. Pulls the main content out of a page.

spider_transformations Rust

HTML transformation library for Rust.

repositories

Utilities

ua_generator Rust

Pre-compiled real User-Agents for Rust. Updated weekly.

spider_fingerprint Rust

Browser fingerprinting library for Rust.

auto-encoder Rust

Bytes-to-string encoding detection for Rust.

async_job Rust

Simple async cron jobs for Rust.

firewall Rust

Firewall utilities for Rust services.

proxier Rust

Auto proxy whitelisting for third-party services.

managed

Don't want to run it yourself?

Spider runs the same crawler with proxies, stealth, and auto-scaling handled. Free balance on signup. Top up later.

Get started free See what changes