# Open Source Web Crawler in Rust | Spider

Open source web crawler and scraper repositories in Rust, MIT licensed. The spider crawler, headless browser, HTML-to-Markdown tools, and clients on GitHub.

Canonical: https://spider.cloud/open-source/

## Open source crawlers, built in public.

Spider runs on open source code. The Rust web crawler, the browser engine, and the scraping and markdown libraries are all on GitHub under the MIT license.

[github.com/spider-rs ↗ ](https://github.com/spider-rs)[Open source vs cloud](https://spider.cloud/open-source-vs-cloud/)

at a glance

- Repositories

  18

- GitHub stars

  3.2k

- License

  MIT

repositories

## Crawlers and clients

[spider Rust](https://github.com/spider-rs/spider)

[Low-latency web data collector. Concurrent crawling in Rust.](https://github.com/spider-rs/spider)

[`$ cargo add spider`](https://github.com/spider-rs/spider)

[2.7k](https://github.com/spider-rs/spider)

[ Rust](https://github.com/spider-rs/gottem)

[One API for every web-scraping vendor. Tiered routing across HTTP, headless, and vendors.](https://github.com/spider-rs/gottem)

[`$ cargo install gottem-cli`](https://github.com/spider-rs/gottem)

[3](https://github.com/spider-rs/gottem)

[spider-browser TypeScript](https://github.com/spider-rs/spider-browser)

[Client for Spider's native cloud browser, built for AI agents, with smart retries.](https://github.com/spider-rs/spider-browser)

[`$ npm install spider-browser`](https://github.com/spider-rs/spider-browser)

[3](https://github.com/spider-rs/spider-browser)

[spider-py Python](https://github.com/spider-rs/spider-py)

[Spider ported to Python. Rust crawler speed from a Python API.](https://github.com/spider-rs/spider-py)

[`$ pip install spider_rs`](https://github.com/spider-rs/spider-py)

[109](https://github.com/spider-rs/spider-py)

[spider-nodejs JavaScript](https://github.com/spider-rs/spider-nodejs)

[Spider ported to Node.js. Native bindings for JavaScript.](https://github.com/spider-rs/spider-nodejs)

[`$ npm install @spider-rs/spider-rs`](https://github.com/spider-rs/spider-nodejs)

[59](https://github.com/spider-rs/spider-nodejs)

[spider-clients Multi](https://github.com/spider-rs/spider-clients)

[Python, JavaScript, and Rust client libraries for the Spider API.](https://github.com/spider-rs/spider-clients)

[26](https://github.com/spider-rs/spider-clients)

repositories

## Browser automation

[headless-browser Rust](https://github.com/spider-rs/headless-browser)

[Headless browser setup for Fargate, Cloud Run, and Kubernetes.](https://github.com/spider-rs/headless-browser)

[87](https://github.com/spider-rs/headless-browser)

[chromey Rust](https://github.com/spider-rs/chromey)

[Chrome DevTools Protocol implementation for Rust.](https://github.com/spider-rs/chromey)

[52](https://github.com/spider-rs/chromey)

[spider-mcp JavaScript](https://github.com/spider-rs/spider-cloud-mcp-server)

[Model Context Protocol server for Claude, Cursor, and Windsurf.](https://github.com/spider-rs/spider-cloud-mcp-server)

[`$ npx spider-cloud-mcp`](https://github.com/spider-rs/spider-cloud-mcp-server)

[2](https://github.com/spider-rs/spider-cloud-mcp-server)

repositories

## Content extraction

[html2md Rust](https://github.com/spider-rs/html2md)

[HTML to Markdown for Rust. Built for LLM content preparation.](https://github.com/spider-rs/html2md)

[79](https://github.com/spider-rs/html2md)

[readability Rust](https://github.com/spider-rs/readability)

[Readability extraction for Rust. Pulls the main content out of a page.](https://github.com/spider-rs/readability)

[17](https://github.com/spider-rs/readability)

[spider_transformations Rust](https://github.com/spider-rs/spider_transformations)

[HTML transformation library for Rust.](https://github.com/spider-rs/spider_transformations)

[2](https://github.com/spider-rs/spider_transformations)

repositories

## Utilities

[ua_generator Rust](https://github.com/spider-rs/ua_generator)

[Pre-compiled real User-Agents for Rust. Updated weekly.](https://github.com/spider-rs/ua_generator)

[40](https://github.com/spider-rs/ua_generator)

[spider_fingerprint Rust](https://github.com/spider-rs/spider_fingerprint)

[Consistent browser profile generation for Rust.](https://github.com/spider-rs/spider_fingerprint)

[14](https://github.com/spider-rs/spider_fingerprint)

[auto-encoder Rust](https://github.com/spider-rs/auto-encoder)

[Bytes-to-string encoding detection for Rust.](https://github.com/spider-rs/auto-encoder)

[1](https://github.com/spider-rs/auto-encoder)

[async_job Rust](https://github.com/spider-rs/async_job)

[Simple async cron jobs for Rust.](https://github.com/spider-rs/async_job)

[2](https://github.com/spider-rs/async_job)

[firewall Rust](https://github.com/spider-rs/firewall)

[Firewall utilities for Rust services.](https://github.com/spider-rs/firewall)

[1](https://github.com/spider-rs/firewall)

[proxier Rust](https://github.com/spider-rs/proxier)

[Auto proxy whitelisting for third-party services.](https://github.com/spider-rs/proxier)

[3](https://github.com/spider-rs/proxier)

managed

## Don't want to run it yourself?

Spider runs the same crawler with proxies, a native browser, and auto-scaling handled. 2,500 free credits, worth $0.25, if your signup passes our checks. No card required. Top up later.

[Get an API key](https://spider.cloud/register/) [See what changes](https://spider.cloud/open-source-vs-cloud/)

See also [Open source vs cloud](https://spider.cloud/open-source-vs-cloud/) [Integrations](https://spider.cloud/integrations/) [Docs](https://spider.cloud/docs/overview/) [Why Spider](https://spider.cloud/why-spider/)
