New Gottem — one API for every web scraping vendor. Read more

Clean Web Data for AI Agents and LLMs

Turn any URL into clean markdown, structured JSON, or a screenshot. One API for crawling, scraping, search, and a headless browser. Built for agents that need fresh web data at runtime.

Get started free Read the docs

Free credits on signup. No card required.

200 OK·https://example.com204ms·1.2 KB·sample

# Example Domain

This domain is for use in illustrative examples in documents.
You may use this domain in literature without prior coordination
or asking for permission.

[More information…](https://www.iana.org/domains/example)

Sample response for scrape. Enter a URL above and Run Test to stream the real one.

POSTapi.spider.cloud/scrapeGet a free API key →

100K+: pages / second
199: proxy countries
99.9%: success rate
10K: req / min

Native integrations with the leading AI frameworks

Features

Give your agents the live web.

Your agents need fresh data. Spider delivers it, even from sites that fight back.

UNMATCHED SPEED

Pages stream back the instant they land. No buffering, no head-of-line blocking. Your pipeline keeps moving even on long crawls.

$ spider crawl example.com --limit 100 --stream

▶ streaming results in parallel

> 100 pages complete

/about

/pricing

/docs/api

/blog/1

/careers

/contact

/faq

/terms

PAY PER USE

Pay for bandwidth and compute, nothing else. Most pages settle for a fraction of a cent. Volume credits unlock at $500+.

Bandwidth: $1 / GB
Compute: $0.001 / min

AI EXTRACTION

Describe the schema in a sentence. Vision models read the rendered DOM and return matching JSON, no selectors required.

prompt “Get every listing with price and rating”

Source <div class="listing"> <h3>MacBook Air M4</h3> <span>$1,099</span> <span>4.8 ★</span> </div>

Structured

[{ "title": "MacBook Air M4",
   "price": "$1,099",
   "rating": 4.8 }]

WEB SEARCH API

One call returns ranked results, fully scraped to markdown, with citations. Skip the search-then-fetch dance.

OpenAI o3 vs Claude 4 benchmarks example

arxiv.org

Benchmarking Large Language Models: O3 and Claude 4

We evaluate performance across MMLU, HumanEval, and GPQA. Claude 4 scores 92.1% on reasoning tasks while o3 leads in…

artificialanalysis.ai

LLM Benchmark Comparison: Speed, Quality, and Price

Side-by-side comparison of output quality, latency, and cost per token across frontier models including o3, Claude 4…

simonwillison.net

Hands-on with o3 and Claude 4: what actually changed

After testing both models extensively on real-world tasks, the practical differences come down to tool use and long…

AI BROWSER

A headless browser on an HTTP API. Click, type, scroll, extract. Stealth headers and proxies handled for you, per request.

protected-site.com/dashboard example

navigate protected-site.com/login

captcha hCaptcha solved automatically

act Click "Export CSV" button

extract structured rows from rendered DOM

observe read dashboard state for next step

done streamed back to your agent

SPEED TEST

tailwindcss.com · 06/2024

Three modes, one API. Smart mode figures out which pages need a browser and which don't, so you don't have to.

HTTP

2s 405 pages

SMART

5s 190 browser, 405 HTTP

BROWSER

8s 595 pages

HTTP → SMART → BROWSER

Community

Teams that ship with Spider.

Engineers building AI agents, RAG pipelines, and research tools on top of Spider.

GitHub discussions Discord chat

@iammerrick

Rust based crawler Spider is next level for crawling & scraping sites. So fast. Their cloud offering is also so easy to use. Good stuff. https://github.com/spider-rs/spider

@WilliamEspegren

Web crawler built in rust, currently the nr1 performance in the world with crazy resource management Aaaaaaand they have a cloud offer, that's wayyyy cheaper than any competitor Name a reason for me to use anything else? github.com/spider-rs/spid…

@gasa

@spider_rs is the best crawling tool i have used. I had a complicated project where i needed to paste url and get the website whole website data. Spider does it in an instant

@Ashpreet Bedi

@spider_rs is THE best crawler out there, give it a try

@Troyusrex

I found a new tool, Spider-rs, which scrapes significantly faster and handles more scenarios than the basic scraper I built did. Our use of Spider-rs and AWS infrastructure reduced the scraping time from four months to under a week.

@Dify.AI

🕷️ Spider @spider_rust can be used as a built-in tool in #Dify Workflow or as an LLM-callable tool in Agent. It allows fast and affordable web scraping and crawling when your AI applications need real-time web data for context.

Why Spider

Built for production.

Rust core. Open source. Engineered for the live web.

100K+ pages per second

Written in Rust with async concurrency from the ground up. The same open-source engine powers every request.

Open source core

The crawler powering this API is on GitHub with 2K+ stars. Audit the code, self-host, or use the managed cloud. No lock-in.

AI-native output

Clean markdown, structured JSON, or screenshots, straight from the rendered DOM. No HTML cleanup, no wasted LLM tokens on navigation chrome.

Bot detection, handled

Stealth headers, residential proxies, and fingerprint rotation on by default. Tune per request when a target demands more.

FAQ

Common questions.

Billing, rate limits, and what happens when a crawl fails.

What is Spider?

Spider is a fast web scraping and crawling API designed for AI agents, RAG pipelines, and LLMs. It supports structured data extraction and multiple output formats including markdown, HTML, JSON, and plain text.

How can I try Spider?

What formats can Spider convert web data into?

Spider outputs HTML, raw, text, and various markdown formats. It supports JSON, JSONL, CSV, and XML for API responses.

Can you crawl all pages?

Yes, Spider accurately crawls all necessary content without needing a sitemap ethically. We rate-limit individual URLs per minute to balance the load on a web server.

Does it respect robots.txt?

Yes, compliance with robots.txt is default, but you can disable this if necessary.

What if a crawl fails?

Failed requests cost nothing. You only pay for successful responses that return data.

What if I get blocked?

Spider includes an unblocker with stealth mode, rotating proxies, and automatic retries. For heavily protected sites, the browser cloud provides full browser sessions with anti-detection built in.

How does billing work?

Each request is billed for bandwidth ($1/GB) plus compute ($0.001/min). Most pages cost a fraction of a cent. You can estimate your costs with the pricing calculator above.