POST /unlimited/{scrape,crawl,links}

Unlimited

Unlimited lets accounts with an active Unlimited subscription run Scrape, Crawl, and Links requests for one predictable monthly price. Instead of per-request credit metering, you purchase the number of requests that can be in flight at once. See Unlimited pricing for plan details.

Routes

api.spider.cloud

POSThttps://api.spider.cloud/unlimited/scrape

alias /v1/unlimited/scrape, same parameters and response format as the standard scrape endpoint

Same API key as every other Spider endpoint. No per-request credits, billing is your purchased concurrency.

curl -X POST https://api.spider.cloud/unlimited/scrape \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "return_format": "markdown"
  }'

Key capabilities

Predictable monthly billing
Based on purchased concurrency, not per-request credits. The per-unit price drops from $20 to $13 as you add concurrency.
Same parameters
The Scrape, Crawl, and Links request parameters you already use.
Immediate feedback
Concurrency-limit responses come back instantly, with retry guidance.
Same authentication
The API key you use on every other Spider endpoint.

Seat semantics

One concurrency unit is one concurrent in-flight request on /unlimited/scrape, /unlimited/crawl, and /unlimited/links. Buy 25 units and 25 requests can be running at any moment.

Every plan includes at least 10,000 requests per minute. Above 33 units, the per-minute cap is 300 × your concurrency.

A single crawl fetches up to 10 pages in parallel regardless of plan size. More concurrency buys more simultaneous crawls, not faster individual crawls.

Concurrency limits

Unlimited requests are not queued. When all purchased concurrency seats are active, the API immediately returns 429 Too Many Requests. Retry after the Retry-After interval, using backoff in your client.

Seats0 of 4 active

Each request holds a seat while it runs. A request past the limit is refused, not queued.

Responseidle

Send a request to occupy a seat.

Every successful response and every 429 response includes X-Concurrency-Limit and X-Concurrency-Active headers.

AI and authentication

AI parameters are not available 400

promptcustom_promptextraction_promptextraction_schemamodelvisionOpenRouter params

Requests that include them return 400. Use the AI API documentation instead; AI usage is billed separately from Unlimited.

Subscription required 403

Authenticate with the same API key used for other Spider endpoints. The key's account must have an active Unlimited subscription. Otherwise, these routes return 403 with {"error":"unlimited_plan_required"} or {"error":"unlimited_plan_inactive"}.

Tip

Unlimited-plan requests currently route through a smaller, flat-rate proxy pool rather than the full proxy fleet. This may affect success rates on the hardest-to-reach targets compared to pay-as-you-go. We are expanding this pool over time.