Rate limits

Every account gets 10,000 requests per minute on the core API. Enterprise raises it. Unlimited is limited by concurrency instead. AI Studio has its own per-second limit.

Core API limits by plan

Counted per account in a fixed one-minute window. Applies to /crawl, /scrape, /links, /search and the other core endpoints.

Plan	Limit	Notes
Pay as you go	10,000 requests per minute	Every account with an API key, from the first credit.
Unlimited	Your concurrency count	Concurrent in-flight requests, the number you buy. The same 10,000 per minute cap as every plan sits underneath and is rarely reached.
Enterprise	50,000 requests per minute	Higher caps on request, set per account.

Unlimited

An Unlimited plan is limited by concurrency, not by requests per minute. Your concurrency count is how many requests can be in flight at once on /unlimited/scrape, /unlimited/crawl or /unlimited/links. Buy 25 and 25 requests can run at the same time. When all of them are busy the API answers 429 right away with a Retry-After header. Nothing queues. Each response carries X-Concurrency-Limit and X-Concurrency-Active so you can watch usage. The 10,000 requests per minute cap is the same on Unlimited as on every other plan. Size your plan by concurrency, not by the minute.

AI Studio

AI Studio endpoints use a per-second limit set by your subscription tier. Past it, the API returns 429 with Retry-After rounded up to the next second. AI requests still spend credits on your account.

Tier	Requests per second
Starter	1
Lite	5
Standard	10
Scale	25
Enterprise	Custom, beyond 25

Rate limit headers

Every core API response carries these headers. Read them to slow down before you hit the limit.

Header	Description
RateLimit-Limit	Requests allowed per minute on your account.
RateLimit-Remaining	Requests left in the current window.
RateLimit-Reset	Seconds until the window resets.
Retry-After	Seconds to wait before retrying. Only on a 429 Too Many Requests response.

Handle a 429

Past the limit, the API returns 429 Too Many Requests. Back off exponentially and retry.

Python backoff

import requests
import time
import os

def request_with_backoff(url, headers, json_data, max_retries=5):
    """Make an API request with exponential backoff on rate limits."""
    for attempt in range(max_retries):
        response = requests.post(url, headers=headers, json=json_data)

        if response.status_code == 429:
            # Use Retry-After header, or fall back to exponential backoff
            retry_after = response.headers.get('Retry-After')
            wait_time = int(retry_after) if retry_after else 2 ** attempt
            print(f"Rate limited. Retrying in {wait_time}s...")
            time.sleep(wait_time)
            continue

        # Check remaining quota to proactively slow down
        remaining = response.headers.get('RateLimit-Remaining')
        if remaining and int(remaining) < 10:
            reset = int(response.headers.get('RateLimit-Reset', 1))
            time.sleep(reset)

        return response

    raise Exception("Max retries exceeded")

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

result = request_with_backoff(
    'https://api.spider.cloud/crawl',
    headers,
    {"url": "https://example.com", "limit": 5}
)
print(result.json())

Node.js backoff

async function requestWithBackoff(url, options, maxRetries = 5) {
  for (let attempt = 0; attempt < maxRetries; attempt++) {
    const response = await fetch(url, options);

    if (response.status === 429) {
      const retryAfter = response.headers.get('Retry-After');
      const waitTime = retryAfter ? Number(retryAfter) : Math.pow(2, attempt);
      console.log(`Rate limited. Retrying in ${waitTime}s...`);
      await new Promise((r) => setTimeout(r, waitTime * 1000));
      continue;
    }

    // Check remaining quota to proactively slow down
    const remaining = response.headers.get('RateLimit-Remaining');
    if (remaining && Number(remaining) < 10) {
      const reset = Number(response.headers.get('RateLimit-Reset') || 1);
      await new Promise((r) => setTimeout(r, reset * 1000));
    }

    return response;
  }

  throw new Error('Max retries exceeded');
}

const result = await requestWithBackoff('https://api.spider.cloud/crawl', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${process.env.SPIDER_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({ url: 'https://example.com', limit: 5 }),
});
console.log(await result.json());

Tip

Use streaming for a large crawl instead of many single requests. One connection carries the whole crawl, so per-request rate limits do not apply.

Tip

On Unlimited a 429 means all of your concurrency is in use. Wait out Retry-Afteror raise your concurrency count. On AI Studio a 429 clears at the next second.

Stay under the limit

Five habits worth building in before your volume grows.

Batch requests. Use limit to crawl many pages in one request instead of one request per page.
Watch the headers. Check RateLimit-Remaining before the next request and pause when it nears zero.
Use webhooks. For a long crawl, receive results by webhook instead of polling.
Cache results. Store crawl output locally so you never fetch the same page twice.
Stream. For many pages, use concurrent streaming and process results as they arrive.