Open Source
Spider is built on open source. The repositories that power our crawler, browser, and content pipeline.
Core
- 2.5kspider Rust
Low-latency web data collector. Concurrent crawling in Rust.
$ cargo add spider - 3Rust
One API for every web scraping vendor. Tiered routing — HTTP, headless, vendors.
$ cargo install gottem-cli - 1spider-browser TypeScript
Browser client for AI agents with stealth and smart retries.
$ npm install spider-browser - 107spider-py Python
Spider ported to Python. Rust crawler speed from a Python API.
$ pip install spider_rs - 59spider-nodejs JavaScript
Spider ported to Node.js. Native bindings for JavaScript.
$ npm install @spider-rs/spider-rs - 26spider-clients Multi
Python, JavaScript, and Rust client libraries for the Spider API.
Browser
Content
Utilities
- 39ua_generator Rust
Pre-compiled real User-Agents for Rust. Updated weekly.
- 15spider_fingerprint Rust
Browser fingerprinting library for Rust.
- 1auto-encoder Rust
Bytes-to-string encoding detection for Rust.
- 2async_job Rust
Simple async cron jobs for Rust.
- 1firewall Rust
Firewall utilities for Rust services.
- 3proxier Rust
Auto proxy whitelisting for third-party services.