Flat-Rate Web Scraping: Put Steady Volume on Unlimited, Keep Pay-As-You-Go for the Rest

Metered credits are a good deal right up until your volume gets steady and high. Pay-as-you-go means you pay for exactly what you use, but once you are crawling the same millions of pages every month, you have outgrown the meter: every request moves the bill, forecasting means modeling per-page costs, and a traffic spike shows up as a spend spike.

Spider’s Unlimited plan is built for exactly that point: a flat monthly rate billed by purchased concurrency (the number of requests you can have in flight at once), not per request. Once your volume is steady and high, it’s usually the cheaper, calmer lane, and this post walks through when that step up pays off. Your exact break-even point depends on what you’re scraping. Page size, JS rendering, and anti-bot complexity all move your real per-request cost, but the shape of the conclusion holds: the more volume you run, the more the flat rate works in your favor.

How the Unlimited plan works

One unit = one concurrent in-flight request. Buy 25 units and you can have 25 requests running at any moment, all month, for one flat price. Zero credits are metered on these routes: throughput is bounded by concurrency, not by a balance.

Three routes are available, plus /v1/unlimited/* aliases:

• POST /unlimited/scrape: same parameters as the standard Scrape API
• POST /unlimited/crawl: same parameters as the standard Crawl API
• POST /unlimited/links: same parameters as the standard Links API

Request parameters and successful response formats match the standard endpoints, so pointing existing code at the Unlimited routes is a URL change, not a rewrite. Full details are in the Unlimited API reference.

What it costs

Pricing is anchored at confirmed price points, with any whole number of units purchasable between them:

Past 100 units, each additional unit is $15/month, up to a self-serve cap of $20,000/month at 1,334 units. Beyond that is sales-assisted. Checkout is a standard secure checkout: pick a number, subscribe, done.

Here is the live slider from the pricing page: drag it to see the exact price at any concurrency amount:

The honest limitations

Flat-rate pricing comes with real trade-offs. Three you should know before switching:

1. No AI extraction on Unlimited routes. AI and LLM parameters (prompt, extraction_schema, model and vision settings) return a 400 on /unlimited/*. AI extraction runs through /ai/* and is billed separately with credits. If your pipeline depends on LLM extraction, that part stays on pay-as-you-go.

2. No queueing: capacity limits are immediate. When all purchased seats are active, the API returns 429 right away rather than queueing your request:

{
  "error": "concurrency_limit_reached",
  "seats": N,
  "active": n
}

The response includes a Retry-After header, and every response (successful or 429) carries X-Concurrency-Limit and X-Concurrency-Active headers so your client can pace itself. Build backoff into your client; the API will not hold requests for you.

3. A smaller proxy pool. Unlimited traffic routes through a smaller, flat-rate proxy pool rather than the full proxy fleet that pay-as-you-go uses. That can affect success rates on the hardest-to-reach targets. This is the trade for the flat rate, stated plainly: the pool is expanding over time, but today pay-as-you-go has the stronger fleet.

Which lane fits your workload

The short version: pay-as-you-go is the right lane at light or bursty volume, and Unlimited becomes the cheaper, calmer option once your traffic is steady and high. Exactly where that crossover lands depends on what you’re scraping. Page size, JS rendering, and anti-bot complexity all move your real per-request cost, so use the shapes below to recognize your situation, then check the slider above against your own account usage rather than a generic formula.

Scenario 1: light or bursty volume fits pay-as-you-go. If you’re running a modest, mostly-static crawl (occasional jobs, spiky traffic, or a volume that moves month to month), the metered bill is usually still the cheaper option, and it naturally scales down when your volume does. There’s no reason to commit to a flat rate for concurrency you won’t consistently use.

Scenario 2: steady, high-volume crawling fits Unlimited. Once you’re running the same heavy volume every month, day in, day out, without much variance, the flat rate on Unlimited typically comes out well ahead of the metered equivalent, and that advantage grows the more volume you run. Just as important: the bill stops moving with traffic. A heavy month costs exactly what a light month costs, which makes forecasting infrastructure spend far easier.

Scenario 3: both lanes at once. A team runs a steady, predictable baseline crawl on /unlimited/crawl at a flat monthly rate (known in advance) plus a smaller, bursty AI-extraction workload on /ai/scrape paid with credits. The AI work has to run on credits anyway, since AI parameters aren’t accepted on Unlimited routes. The flat rate absorbs the predictable volume; credits cover the variable, higher-value calls. This is the configuration the two plans were designed for.

Your break-even point is specific to your workload. The fastest way to find it is to compare your account’s real usage against the slider above.

Better together: one API key, both lanes

The comparison table above could make the plans look like rivals. They aren’t. These plans aren’t either-or, and the same message sits on the pricing page itself. One API key runs both lanes:

The split follows directly from the limitation in section three: AI extraction isn’t available on Unlimited, so any pipeline that both crawls at volume and extracts with LLMs naturally uses both lanes. The flat rate makes the big, predictable part of the bill fixed; credits handle everything the flat rate doesn’t cover.

Getting started

If your volume is anywhere near steady and high, this is worth setting up today. It’s a URL change, not a migration:

1. Pick a concurrency amount at /pricing?plan=unlimited, or use the slider above to find the price for your target throughput.
2. Point existing calls at the Unlimited routes. /scrape, /crawl, and /links become /unlimited/scrape, /unlimited/crawl, and /unlimited/links, same parameters, same response shapes, same API key.
3. Keep everything else on credits. /ai/*, search, screenshots, and any other standard endpoint continue to work exactly as they do today.

A first request looks like this:

curl -X POST https://api.spider.cloud/unlimited/scrape \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "return_format": "markdown"
  }'

If the response is a page of clean markdown and no credits moved, you’re on the flat rate. From here, scale the concurrency up as your workload grows: the price stays flat, and so does your bill.

Combining both plans: never stop for a 429

Section three covered the trade-off: Unlimited doesn’t queue, so a request beyond your purchased concurrency gets an immediate 429 instead of waiting in line. That’s also exactly where the two lanes complement each other: when Unlimited is momentarily out of seats, fall back to a pay-as-you-go call on the standard endpoint instead of dropping the request or blocking on a retry. The flat rate handles your steady-state throughput; credits absorb the overflow so the pipeline never actually stops.

import requests

API_KEY = "YOUR_API_KEY"
HEADERS = {"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"}

def scrape(url: str) -> dict:
    payload = {"url": url, "return_format": "markdown"}

    # Try the flat-rate lane first: no credits, bounded by purchased concurrency.
    response = requests.post(
        "https://api.spider.cloud/unlimited/scrape", headers=HEADERS, json=payload
    )

    if response.status_code == 429:
        # All seats are busy right now: fall back to the metered endpoint
        # immediately instead of queueing or dropping the request. Credits
        # keep the pipeline moving; Unlimited picks the load back up on its
        # own once a seat frees, so nothing sits idle waiting.
        response = requests.post(
            "https://api.spider.cloud/scrape", headers=HEADERS, json=payload
        )

    response.raise_for_status()
    return response.json()

The result: your baseline volume runs flat-rate, your burst capacity runs on credits, and a capacity limit on one lane never becomes downtime for the pipeline as a whole.