Flat-rate web scraping vs. pay-as-you-go: which to put where

Metered credits are a good deal right up until your volume gets steady and high. Pay-as-you-go means you pay for exactly what you use, but once you are crawling the same millions of pages every month, you have outgrown the meter. Every request moves the bill, forecasting means modeling per-page costs, and a traffic spike shows up as a spend spike.

Spider’s Unlimited plan is built for that point. It bills by purchased concurrency (the number of requests you can have in flight at once), not per request, so the monthly price is fixed. This post walks through when that step up pays off. Your exact break-even depends on what you’re scraping, since page size, JS rendering, and anti-bot complexity all move the real per-request cost, but the more volume you run, the more the flat rate works in your favor.

How the Unlimited plan works

One unit = one concurrent in-flight request. Buy 25 units and you can have 25 requests running at any moment, all month, for one flat price. These routes meter zero credits. Concurrency bounds your throughput, not a balance.

Three routes are available, plus /v1/unlimited/* aliases:

• POST /unlimited/scrape: same parameters as the standard Scrape API
• POST /unlimited/crawl: same parameters as the standard Crawl API
• POST /unlimited/links: same parameters as the standard Links API

Request parameters and successful response formats match the standard endpoints, so pointing existing code at the Unlimited routes is a URL change, not a rewrite. Full details are in the Unlimited API reference.

What it costs

These are the anchor prices. You can buy any whole number of units between them:

Between the anchors, billing is graduated: each band of units bills at that band’s own per-unit rate and the bands sum, so 4 units is 2 × $20.00 + 2 × $16.25 = $72.50 exactly. Past 50 units, each additional unit is $13/month, the same effective rate as the 50-unit anchor, so the price per unit never climbs back up as you scale. That holds up to the self-serve maximum of 1,538 units at $19,994/month. Beyond that is sales-assisted. Checkout is a standard secure checkout. Pick a number, subscribe, done.

Here is the live picker from the pricing page: choose a preset amount or type your own to see the exact price at any concurrency:

The limitations

Flat-rate pricing comes with real trade-offs. Three you should know before switching:

1. No AI extraction on Unlimited routes. AI and LLM parameters (prompt, extraction_schema, model and vision settings) return a 400 on /unlimited/*. AI extraction runs through /ai/* and is billed separately with credits. If your pipeline depends on LLM extraction, that part stays on pay-as-you-go.

2. No queueing: capacity limits are immediate. When all purchased seats are active, the API returns 429 right away rather than queueing your request:

{
  "error": "concurrency_limit_reached",
  "seats": N,
  "active": n
}

The response includes a Retry-After header, and every response (successful or 429) carries X-Concurrency-Limit and X-Concurrency-Active headers so your client can pace itself. Build backoff into your client; the API will not hold requests for you.

3. A smaller proxy pool. Unlimited traffic routes through a smaller, flat-rate proxy pool. That can affect success rates on the hardest-to-reach targets. The pool is growing, but today pay-as-you-go has the stronger fleet.

Which lane fits your workload

Pay-as-you-go is the right lane at light or bursty volume. Unlimited is the cheaper, calmer option once your traffic is steady and high. Where the crossover lands depends on what you’re scraping, since page size, JS rendering, and anti-bot complexity all move your real per-request cost. Use the scenarios below to find your situation, then check the picker above against your own account usage rather than a generic formula.

Scenario 1: light or bursty volume fits pay-as-you-go. If you’re running a modest, mostly-static crawl (occasional jobs, spiky traffic, or a volume that moves month to month), the metered bill is usually still the cheaper option, and it scales down when your volume does. There’s no reason to commit to a flat rate for concurrency you won’t consistently use.

Scenario 2: steady, high-volume crawling fits Unlimited. Once you’re running the same heavy volume every month, day in, day out, without much variance, the flat rate on Unlimited typically comes out well ahead of the metered equivalent, and that advantage grows the more volume you run. The bill also stops moving with traffic. A heavy month costs exactly what a light month costs, which makes forecasting infrastructure spend easier.

Scenario 3: both lanes at once. A team runs a steady, predictable baseline crawl on /unlimited/crawl at a flat monthly rate (known in advance) plus a smaller, bursty AI-extraction workload on /ai/scrape paid with credits. The AI work has to run on credits anyway, since AI parameters aren’t accepted on Unlimited routes. The flat rate absorbs the predictable volume; credits cover the variable, higher-value calls. This is the setup we designed the two plans for.

Better together: one API key, both lanes

The comparison above could make the plans look like rivals. They aren’t, and the pricing page says the same. One API key runs both lanes:

The split follows directly from the limitation in section three: AI extraction isn’t available on Unlimited, so any pipeline that both crawls at volume and extracts with LLMs uses both lanes. The flat rate makes the big, predictable part of the bill fixed; credits handle everything the flat rate doesn’t cover.

Getting started

If your volume is anywhere near steady and high, this is worth setting up today. It’s a URL change, not a migration:

1. Pick a concurrency amount at /pricing?plan=unlimited, or use the picker above to find the price for your target throughput.
2. Point existing calls at the Unlimited routes. /scrape, /crawl, and /links become /unlimited/scrape, /unlimited/crawl, and /unlimited/links, same parameters, same response shapes, same API key.
3. Keep everything else on credits. /ai/*, search, screenshots, and any other standard endpoint continue to work exactly as they do today.

A first request looks like this:

curl -X POST https://api.spider.cloud/unlimited/scrape \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "return_format": "markdown"
  }'

If the response is a page of clean markdown and no credits moved, you’re on the flat rate. From here, scale the concurrency up as your workload grows: the bill stays predictable, and the per-unit cost only drops, from $20 at the smallest plan down to $13 at scale.

Combining both plans: never stop for a 429

Section three covered the trade-off: Unlimited doesn’t queue, so a request beyond your purchased concurrency gets an immediate 429 instead of waiting in line. That is where the two lanes cover for each other. When Unlimited is momentarily out of seats, fall back to a pay-as-you-go call on the standard endpoint instead of dropping the request or blocking on a retry. The flat rate handles your steady-state throughput; credits absorb the overflow so the pipeline keeps moving.

import requests

API_KEY = "YOUR_API_KEY"
HEADERS = {"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"}

def scrape(url: str) -> dict:
    payload = {"url": url, "return_format": "markdown"}

    # Try the flat-rate lane first: no credits, bounded by purchased concurrency.
    response = requests.post(
        "https://api.spider.cloud/unlimited/scrape", headers=HEADERS, json=payload
    )

    if response.status_code == 429:
        # All seats are busy right now: fall back to the metered endpoint
        # immediately instead of queueing or dropping the request. Credits
        # keep the pipeline moving; Unlimited picks the load back up on its
        # own once a seat frees, so nothing sits idle waiting.
        response = requests.post(
            "https://api.spider.cloud/scrape", headers=HEADERS, json=payload
        )

    response.raise_for_status()
    return response.json()

Your baseline volume runs flat-rate, your burst capacity runs on credits, and a capacity limit on one lane never becomes downtime for the pipeline as a whole.