Open Source
Spider is built on open source. We believe in transparency, collaboration, and giving back to the community that makes our work possible.
Core Crawlers
The foundation of Spider - high-performance web crawling in multiple languages.
spider
The fastest web crawler written in Rust. High performance, concurrent crawling with smart rate limiting and robot.txt compliance.
spider-py
Spider ported to Python. Get the power of Rust crawling with Python simplicity.
spider-nodejs
Spider ported to Node.js. Native bindings for lightning-fast crawling in JavaScript.
spider-clients
Official Python, JavaScript, and Rust client libraries for the Spider Cloud API.
Browser & Automation
Tools for headless browser control and automation.
headless-browser
Reliable high performance headless browser setup for cloud providers like Fargate, CloudRun, Kubernetes, and more.
Content Processing
Libraries for extracting, transforming, and processing web content.
html2md
56Fast HTML to Markdown conversion for Rust. Perfect for LLM content preparation.
readability
14The readability extraction library for Rust. Extract main content from web pages.
HTML transformation library for Rust. Clean, transform, and process HTML content.
Utilities & Libraries
Supporting libraries that power the Spider ecosystem.
ua_generator
36Pre-compiled random real User-Agents. Updated weekly for Windows, Mac, Linux, and Android.
Browser fingerprinting library for Rust. Generate realistic browser fingerprints.
Automatic encoding detection and conversion. Bytes to strings with smart charset detection.
firewall
1Firewall utilities for Rust. Block unwanted requests and protect your applications.
Resources
Guides, tutorials, and learning resources.
Comprehensive guides on web crawling and scraping. Best practices, tutorials, and examples.
Contribute to Spider
We welcome contributions of all kinds. Whether it's fixing bugs, adding features, improving documentation, or sharing feedback - every contribution matters.