Open Source
Built in the Open
Spider is built on open source. Transparency, collaboration, and giving back to the community that makes our work possible.
Core
Core Projects
The foundation of Spider — high-performance web crawling in multiple languages.
spider
The fastest web crawler written in Rust. High performance, concurrent crawling with smart rate limiting and robots.txt compliance.
spider-browser
Full browser automation client with AI agents, stealth engine, and smart retries. CDP & BiDi support across 6 browser backends.
npm install spider-browser spider-py
Spider ported to Python. Get the power of Rust crawling with Python simplicity.
pip install spider_rs spider-nodejs
Spider ported to Node.js. Native bindings for lightning-fast crawling in JavaScript.
npm install @aspect-build/spider Browser
Browser & Automation
Tools for headless browser control, CDP, and AI-powered automation.
headless-browser
Reliable high performance headless browser setup for cloud providers like Fargate, CloudRun, Kubernetes, and more.
chromey
Chrome DevTools Protocol implementation for Rust. Control Chrome/Chromium programmatically.
spider-mcp
Model Context Protocol server for Claude, Cursor, Windsurf, and other AI tools.
npx spider-cloud-mcp Content
Content Processing
Libraries for extracting, transforming, and processing web content.
html2md
Fast HTML to Markdown conversion for Rust. Perfect for LLM content preparation.
readability
The readability extraction library for Rust. Extract main content from web pages.
Utilities
Utilities & Libraries
Supporting libraries that power the Spider ecosystem.
ua_generator
Pre-compiled random real User-Agents. Updated weekly for Windows, Mac, Linux, and Android.
spider_fingerprint
Browser fingerprinting library for Rust. Generate realistic browser fingerprints.
auto-encoder
Automatic encoding detection and conversion. Bytes to strings with smart charset detection.
firewall
Firewall utilities for Rust. Block unwanted requests and protect your applications.
proxier
Auto proxy whitelisting for 3rd party services. Manage proxy configurations easily.