# Open source web crawler in Rust | Spider

spider is an open source web crawler written in Rust. Add the crate with cargo add spider, then find the libraries and clients built around it on GitHub.

Canonical: https://spider.cloud/open-source/

## An open source web crawler in Rust.

Spider's crawler is a public Rust crate, and the libraries and clients around it are public too.

Quick start

```
cargo add spider
```

```
use spider::website::Website;
 
#[tokio::main]
async fn main() {
    let mut website = Website::new("https://example.com");
 
    website.crawl().await;
 
    for page in website.get_pages().unwrap().iter() {
        println!("{} - {} bytes", page.get_url(), page.get_html_bytes_u8().len());
    }
}
```

Copied from the quick start in the [self-hosting guide](https://spider.cloud/guides/self-hosting-spider/). It crawls example.com, follows links on the same domain, and prints each page's URL and size.

The spider crate crawls a site concurrently and hands you each page it fetched.

2.9k stars on GitHub

[Get an API key](https://spider.cloud/register/) [spider-rs/spider on GitHub](https://github.com/spider-rs/spider)

## Crawler core

The crawler, and the libraries that turn a fetched page into text.

- [spider](https://github.com/spider-rs/spider)

  - Rust
  - Low-latency web data collector. Concurrent crawling in Rust. MIT license. Install: cargo add spider

- [html2md](https://github.com/spider-rs/html2md)

  - Rust
  - HTML to Markdown for Rust. Built for LLM content preparation.

- [readability](https://github.com/spider-rs/readability)

  - Rust
  - Readability extraction for Rust. Pulls the main content out of a page.

- [spider_transformations](https://github.com/spider-rs/spider_transformations)

  - Rust
  - HTML transformation library for Rust.

## Browser and rendering

Chrome driven from Rust, and the user agents and profiles a browser presents.

- [headless-browser](https://github.com/spider-rs/headless-browser)

  - Rust
  - Headless browser setup for Fargate, Cloud Run, and Kubernetes.

- [chromey](https://github.com/spider-rs/chromey)

  - Rust
  - Chrome DevTools Protocol implementation for Rust.

- [spider_fingerprint](https://github.com/spider-rs/spider_fingerprint)

  - Rust
  - Consistent browser profile generation for Rust.

- [ua_generator](https://github.com/spider-rs/ua_generator)

  - Rust
  - Pre-compiled real User-Agents for Rust. Updated weekly.

## Clients and SDKs

The crawler from Python and Node.js, and clients for the hosted API.

- [spider-py](https://github.com/spider-rs/spider-py)

  - Python
  - Spider ported to Python. Rust crawler speed from a Python API. Install: pip install spider_rs

- [spider-nodejs](https://github.com/spider-rs/spider-nodejs)

  - JavaScript
  - Spider ported to Node.js. Native bindings for JavaScript. Install: npm install @spider-rs/spider-rs

- [spider-clients](https://github.com/spider-rs/spider-clients)

  - Python, JavaScript, Rust
  - Python, JavaScript, and Rust client libraries for the Spider API.

- [spider-browser](https://github.com/spider-rs/spider-browser)

  - TypeScript
  - Client for Spider's native cloud browser, built for AI agents, with smart retries. Install: npm install spider-browser

- [spider-mcp](https://github.com/spider-rs/spider-cloud-mcp-server)

  - JavaScript
  - Model Context Protocol server for Claude, Cursor, and Windsurf. Install: npx spider-cloud-mcp

## Tools

Smaller pieces that are useful without the crawler.

- [gottem](https://github.com/spider-rs/gottem)

  - Rust
  - One API for every web-scraping vendor. Tiered routing across HTTP, headless, and vendors. Install: cargo install gottem-cli

- [auto-encoder](https://github.com/spider-rs/auto-encoder)

  - Rust
  - Bytes-to-string encoding detection for Rust.

- [async_job](https://github.com/spider-rs/async_job)

  - Rust
  - Simple async cron jobs for Rust.

- [firewall](https://github.com/spider-rs/firewall)

  - Rust
  - Firewall utilities for Rust services.

- [proxier](https://github.com/spider-rs/proxier)

  - Rust
  - Auto proxy whitelisting for third-party services.

## Around the code

The hosted API, and where to ask.

- [Integrations](https://spider.cloud/integrations/)

  Spider inside agent frameworks and automation tools.

- [Docs](https://spider.cloud/docs/overview/)

  The reference for the hosted API.

- [Support](https://spider.cloud/support/)

  Where to ask a question, by email, Discord or GitHub discussions.

- [github.com/spider-rs](https://github.com/spider-rs)

  Every repository on this page, in one place.

## Don't want to run it yourself?

Hosted Spider runs the same crawler with proxies, a browser and scaling handled. 2,500 free credits, worth $0.25, if your signup passes our checks. No card required.

[Open source vs hosted](https://spider.cloud/open-source-vs-cloud/)

[Get an API key](https://spider.cloud/register/)
