Crawl a website. Get the data.

Follow links across a site or scrape a single page. Spider returns Markdown, HTML, or JSON, rendered in full by a native browser, with proxies built in.

Get started free Try the playground

                   0010                                        1110
              0100001110                                      1001011000
           010111     110                                    101     000111
                       1100                                1101
                        1000                              1010
                          1101                          1101
                           1111                        0010
                            0111                      1101
           011101             1110                  0010             111011
        11010 100001           1110                0011           011000 11001
      1001       0010101        00011    0100    10100        0011100       0101
    1110            1110010       101000011100010111       1000000            0101
  111                   0110010    1101101101011000    0011101                   000
000                        10101001 10110011011011 11010110                        111
                              11001000101001011101011001
                                 10010010100101111010
                                  100101010110010000
             100000000001011100110001010111101100011110010010010010110101
            01011110111010110010101101110011101100110001000010010101011101
          0000                     1111011011100110                     1011
         110                     100100        000010                     101
        111                   111110    001110    111010                   001
       01                   010001    0000001100    111110                   00
     100                  11101     00111110100001     00010                  011
    100                11000       0111101110001110       00111                110
   11                 1100        110110000111001101        0101                 11
  10                  11          011111101010011100          01                  11
 01                  100         11110101000111100001         001                  01
                    011          00001011101110011100          111
                    00           01011111000001010001           11
                   100           00011010101101010000           001
                  101            11010111010011010100            000
                  10             00101100011000101100             11
                 00              00110100111110001100              01
                011               100111011110110100               101
                01                000101100101111110                10
               00                  0000111101010100                  10
               01                   01000110110100                   11
                                      1110000101

curl https://api.spider.cloud/crawl \
  -H "Authorization: Bearer $SPIDER_API_KEY" \
  -H "Content-Type: application/jsonl" \
  -d '{"url": "https://doc.rust-lang.org/book/", "limit": 30,
       "return_format": "markdown", "request": "smart",
       "country_code": "de",
       "whitelist": ["doc\\.rust-lang\\.org/book/"],
       "blacklist": ["print\\.html","/stable/"]}'

Response /ch01-01-installation.html

[
  {
    "url": "https://doc.rust-lang.org/book/ch01-01-installation.html",
    "status": 200,
    "content": "# The Rust Programming Language\nThe first step is to install Rust. We’ll download Rust through `rustup`, a\ncommand line tool for managing Rust versions and associated tools. You’ll need\nan internet connection for the download.\nNote: If you prefer not to use `rustup` for some reason, please see the\n[Other Rust Installation Methods page](https://forge.rust-lang.org/infra/other-installation-methods.html) for more options.\n…"
  }
]

1 request · 30 pages · 319 KB · captured Sep 9, 2026 Open in playground

by the numbers

Pages per second: 100k+
Success rate: 99.9%
Monthly minimum: $0

benchmark

Browser Fidelity Bench V1, 80 tasks.

100% engine pass rate.

Spider passed 80/80 tasks on Sep 18, 2026, each a real page on a protected production site. Other providers retain the March 22, 2026 readings published in the blog; they were not re-measured.

Spider100%

Browser Use Cloud81%

Anchor74%

100%: 999-URL pass 254 domains, 18 categories
0.9s: To connect median, Sep 14, 2026
238ms: Page load, attached example.com, median

Read the March 22, 2026 benchmark

URL in, structured data out.

step 1

Submit a URL

Send a target URL with depth, page limit, and output format.

step 2

Spider walks the site

Follows links, renders JS in a native browser, rotates proxies.

step 3

Clean data back

Receive HTML, Markdown, plain text, JSON, screenshots, or PDF, via stream or batch.

Three rendering strategies. Switch per request.

HTTP

Direct fetch, no browser. Static sites, blogs, docs.

~50ms / page

default

Smart

Detects when JS rendering is needed. Browser only when it has to be.

~200ms / page

Browser

Full headless rendering. SPAs, lazy loading, infinite scroll.

~1s / page

Same payload in any language.

request

POST /crawl

								curl -X POST https://api.spider.cloud/crawl \
  -H "Authorization: Bearer $SPIDER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "limit": 25,
    "return_format": "markdown"
  }'

								import requests

response = requests.post(
    "https://api.spider.cloud/crawl",
    headers={
        "Authorization": f"Bearer {SPIDER_API_KEY}",
    },
    json={
        "url": "https://example.com",
        "limit": 25,
        "return_format": "markdown",
    },
)

pages = response.json()
print(f"Crawled {len(pages)} pages")

								const response = await fetch("https://api.spider.cloud/crawl", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.SPIDER_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    url: "https://example.com",
    limit: 25,
    return_format: "markdown",
  }),
});

const pages = await response.json();
console.log(`Crawled ${pages.length} pages`);

Switch the format with one parameter.

HTML

Raw or cleaned, with optional tag filtering

Markdown

Clean Markdown for LLMs and RAG pipelines

Plain text

Stripped text content, no markup

JSON

Typed extraction via AI-powered schemas

Screenshot

Full-page PNG or viewport capture

PDF

Browser-rendered PDF of any page

Crawling infrastructure that handles millions of pages.

what runs it

Elastic concurrency

Auto-scales concurrent connections to match crawl volume. 10 pages to 10 million, same code.

Proxy rotation

Automatic IP rotation across residential and datacenter pools. Spider selects the proxy type per domain.

Native browser

A real browser engine built for automated work. Pages that need JavaScript render completely on the first request.

HTTP caching

Previously crawled pages are cached and served instantly on repeat requests.

Webhooks

Fires as pages are discovered. Process the stream without waiting for the whole crawl.

Web scraping questions, answered.

What is web crawling?

Automated discovery of pages by following links. A crawler starts from one or more seed URLs, fetches the page, extracts links, and repeats. Spider does the full loop behind one API call: you pass a URL and get structured data back from every reachable page.

Is web scraping legal?

Scraping publicly available data is generally legal in the United States (hiQ Labs v. LinkedIn). Respect robots.txt, terms of service, and rate limits; do not scrape personal data without consent. Spider honours robots.txt by default and includes built-in rate limiting.

How many pages can Spider crawl?

No hard limit. The infrastructure auto-scales from a single page to millions per job. Throughput depends on your plan and concurrency; enterprise users reach 500+ pages per second.

What's the difference between crawling and scraping?

Crawling is discovery: following links to find pages. Scraping is extraction: pulling specific data from those pages. Spider does both: it walks the target site and returns clean output (HTML, Markdown, plain text, JSON, screenshot, or PDF) per page.

Does Spider handle JavaScript-rendered pages?

Yes. Three modes: HTTP (fast static fetch), Smart (auto-detects whether JS rendering is needed), and Browser (full headless rendering for SPAs, lazy loading, and interactions).

How do I scrape a website without getting blocked?

Spider loads pages in its native browser, a real browser engine rather than a patched headless one, so the page renders the way it does for a person. Requests go out through residential and datacenter proxies, with the proxy type picked per domain. Crawls respect robots.txt and rate limits by default, which keeps request volume at a level the site accepts.

Make your first request.

Create an API key and start with one URL.

Get started free Read the quickstart