POST /ai/{crawl,scrape,search,browser,links,unblocker}

AI API

Every AI endpoint takes a plain English prompt alongside the parameters you already use on the standard API. Spider plans the work, cleans the page, and returns data shaped by that prompt. Six routes cover crawling, scraping, search, browser automation, link discovery, and unblocking, all on one request and response contract.

Routes

Six routes, all POST only. Any other method returns 400. Each one also answers on a /v1 alias, so /v1/ai/crawl is equivalent to /ai/crawl.

POST/ai/crawl
Crawl a site from one starting URL and let the prompt decide what matters on each page. Returns an array of pages with structured data attached.
POST/ai/scrape
Fetch a single URL and extract exactly what the prompt asks for. Pair it with an extraction_schema when you need a stable shape.
POST/ai/search
Run a web search driven by the prompt and return ranked results. Pass `search` yourself to skip query generation, or leave it off and the prompt becomes the query.
POST/ai/browser
Drive a real browser session with a natural language prompt, so clicks, toggles and form steps happen before extraction. Best for pages that only reveal data after interaction.
POST/ai/links
Collect the links on a page that match the prompt. Link return is forced on for this route, so every page object carries a `links` array.
POST/ai/unblocker
Fetch a page that is behind bot protection and hand the cleaned result to the model. Use it when a normal scrape returns a challenge page.

How AI endpoints differ

A prompt, not a selector
Describe the data you want in plain language. There is no CSS path to maintain when the markup changes.
Everything the standard route takes
Each AI route accepts the full parameter surface of its non-AI counterpart, plus the AI fields.
Structured output
Results arrive on metadata.extracted_data, shaped by extraction_schema when you supply one.
Always a real browser
AI routes cache, but they never skip the browser, so the model reads live page content.
Separate billing
AI usage draws on an AI Studio subscription, or on credits when no subscription is active.
Model cost passed through
ai_cost is the raw vendor token spend and is excluded from domain markup.

Authentication

Send your API key as a bearer token against https://api.spider.cloud. The same key works across every Spider endpoint. AI usage is billed against an AI Studio subscription, falling back to credits when no subscription is active, so a key that works on /crawl can still be refused here. See Errors for what that looks like.

Authorization header required

Bearer YOUR_API_KEY. Requests without it fail with an empty 400.

Content-Type header required

application/json.

Common AI parameters

These apply across the AI routes. Each route additionally accepts everything its standard counterpart takes, so the Parameters reference covers proxies, geo-location, caching, headers and the rest.

prompt string required

Natural language description of what to do and what to pull back. An empty prompt is rejected with a 400.

extraction_schema object optional

Shape for the structured result. Takes `name` (required), `description` (optional) and `schema`, which may be a JSON object or a JSON encoded string.

cleaning_intent "extraction" | "action" | "general" optional, crawl / scrape / links / unblocker only

How aggressively the HTML is reduced before the model reads it. Use "extraction" for data pulls, "action" to keep interactive elements, "general" for a balanced clean. Not accepted on /ai/search or /ai/browser.

metadata boolean default false

Return page metadata such as title and description. Note that `metadata.extracted_data` is attached to the response either way, so structured output never depends on this flag.

return_format string optional

Content shape for each page, for example "markdown", "raw", "text" or "commonmark".

limit number optional

Maximum pages to process. The value is lifted internally into a crawl budget of `{"*": limit}`.

An extraction_schema pins the output shape. Pass schema as an object, or as a JSON encoded string if that is easier to carry through your client.

{
  "extraction_schema": {
    "name": "pricing_tier",
    "description": "One pricing tier from the plans table.",
    "schema": {
      "type": "object",
      "properties": {
        "tier": { "type": "string" },
        "monthly_cost": { "type": "number" }
      },
      "required": ["tier", "monthly_cost"]
    }
  }
}

Response

Every AI route returns a JSON array of page objects, even when you asked for a single URL. Two fields catch people out, so they are called out below.

url string always

The URL this result came from.

status number always

HTTP status returned by the target page, not by the Spider API.

error string | null always

Per page error. Null on success. A page can fail while the request as a whole returns 200.

duration_elasped_ms number always

Time spent on this page. Note the spelling: "elasped" is part of the public contract, so reading duration_elapsed_ms returns undefined.

costs object | null always

Cost breakdown in USD. It is null, not an object of zeros, whenever the request bills nothing, so check before reading total_cost.

metadata.extracted_data object always

The structured result. It is attached whether or not you passed metadata: true.

content string conditional

Page content in the shape named by return_format. Omitted when no content was requested.

links string[] conditional

Discovered links. Always present on /ai/links, which forces link return on.

AI Crawl

POST/ai/crawl

Crawl a site from one starting URL and let the prompt decide what matters on each page. Returns an array of pages with structured data attached.

depth number optional

How many link levels below the starting URL to follow.

post /ai/crawl

import os

import requests

payload = {
  "url": "https://spider.cloud",
  "prompt": "Collect every pricing tier with its monthly cost and included credits.",
  "cleaning_intent": "extraction",
  "depth": 2,
  "limit": 25,
  "return_format": "markdown",
  "metadata": True
}

response = requests.post(
    "https://api.spider.cloud/ai/crawl",
    headers={
        "Authorization": f"Bearer {os.environ['SPIDER_API_KEY']}",
        "Content-Type": "application/json",
    },
    json=payload,
    timeout=120,
)

print(response.json())

application/json

[
  {
    "url": "https://spider.cloud/pricing",
    "error": null,
    "status": 200,
    "content": "# Pricing\n\nStarter, Pro and Enterprise plans.",
    "metadata": {
      "title": "Pricing",
      "description": "Spider Cloud plans and included credits.",
      "extracted_data": {
        "tiers": [
          { "tier": "Starter", "monthly_cost": 0, "included_credits": 200 },
          { "tier": "Pro", "monthly_cost": 99, "included_credits": 50000 }
        ]
      }
    },
    "costs": {
      "file_cost": 0.00021,
      "transform_cost": 0.00004,
      "compute_cost": 0.00038,
      "ai_cost": 0.00265,
      "bytes_transferred_cost": 0.00002,
      "total_cost": 0.0033,
      "file_cost_formatted": "$0.00021",
      "transform_cost_formatted": "$0.00004",
      "compute_cost_formatted": "$0.00038",
      "ai_cost_formatted": "$0.00265",
      "bytes_transferred_cost_formatted": "$0.00002",
      "total_cost_formatted": "$0.0033"
    },
    "duration_elasped_ms": 4821
  }
]

AI Scrape

POST/ai/scrape

Fetch a single URL and extract exactly what the prompt asks for. Pair it with an extraction_schema when you need a stable shape.

post /ai/scrape

import os

import requests

payload = {
  "url": "https://spider.cloud/pricing",
  "prompt": "Pull the Pro plan name, its monthly cost, and whether annual billing is selected.",
  "cleaning_intent": "extraction",
  "extraction_schema": {
    "name": "pricing_tier",
    "description": "One pricing tier from the plans table.",
    "schema": "{\"type\":\"object\",\"additionalProperties\":false,\"properties\":{\"tier\":{\"type\":\"string\"},\"monthly_cost\":{\"type\":\"number\"},\"is_annual\":{\"type\":\"boolean\",\"default\":true,\"description\":\"Annual billing selected\"}},\"required\":[\"tier\",\"monthly_cost\"]}"
  },
  "return_format": "markdown",
  "metadata": True
}

response = requests.post(
    "https://api.spider.cloud/ai/scrape",
    headers={
        "Authorization": f"Bearer {os.environ['SPIDER_API_KEY']}",
        "Content-Type": "application/json",
    },
    json=payload,
    timeout=120,
)

print(response.json())

application/json

[
  {
    "url": "https://spider.cloud/pricing",
    "error": null,
    "status": 200,
    "content": "# Pricing\n\nStarter, Pro and Enterprise plans.",
    "metadata": {
      "title": "Pricing",
      "extracted_data": {
        "tier": "Pro",
        "monthly_cost": 99,
        "is_annual": false
      }
    },
    "costs": {
      "file_cost": 0.00021,
      "transform_cost": 0.00004,
      "compute_cost": 0.00038,
      "ai_cost": 0.00265,
      "bytes_transferred_cost": 0.00002,
      "total_cost": 0.0033,
      "file_cost_formatted": "$0.00021",
      "transform_cost_formatted": "$0.00004",
      "compute_cost_formatted": "$0.00038",
      "ai_cost_formatted": "$0.00265",
      "bytes_transferred_cost_formatted": "$0.00002",
      "total_cost_formatted": "$0.0033"
    },
    "duration_elasped_ms": 2140
  }
]

AI Search

POST/ai/search

Run a web search driven by the prompt and return ranked results. Pass `search` yourself to skip query generation, or leave it off and the prompt becomes the query.

search string optional, search only

The literal search query. When omitted, a query is generated from the prompt.

fetch_page_content boolean default false, search only

Fetch and return the content of each result page instead of the result listing alone.

num number optional, search only

Number of search results to return.

post /ai/search

import os

import requests

payload = {
  "prompt": "Find recent benchmarks comparing Rust web crawlers on throughput.",
  "search": "rust web crawler throughput benchmark",
  "num": 10,
  "fetch_page_content": True,
  "return_format": "markdown",
  "limit": 10
}

response = requests.post(
    "https://api.spider.cloud/ai/search",
    headers={
        "Authorization": f"Bearer {os.environ['SPIDER_API_KEY']}",
        "Content-Type": "application/json",
    },
    json=payload,
    timeout=120,
)

print(response.json())

application/json

[
  {
    "url": "https://example.com/rust-crawler-benchmarks",
    "error": null,
    "status": 200,
    "content": "# Rust crawler benchmarks\n\nThroughput measured across 12 runs.",
    "metadata": {
      "title": "Rust crawler benchmarks",
      "extracted_data": {
        "results": [
          { "title": "Rust crawler benchmarks", "rank": 1 },
          { "title": "Comparing async crawlers", "rank": 2 }
        ]
      }
    },
    "costs": {
      "file_cost": 0.00021,
      "transform_cost": 0.00004,
      "compute_cost": 0.00038,
      "ai_cost": 0.00265,
      "bytes_transferred_cost": 0.00002,
      "total_cost": 0.0033,
      "file_cost_formatted": "$0.00021",
      "transform_cost_formatted": "$0.00004",
      "compute_cost_formatted": "$0.00038",
      "ai_cost_formatted": "$0.00265",
      "bytes_transferred_cost_formatted": "$0.00002",
      "total_cost_formatted": "$0.0033"
    },
    "duration_elasped_ms": 3675
  }
]

AI Browser

POST/ai/browser

Drive a real browser session with a natural language prompt, so clicks, toggles and form steps happen before extraction. Best for pages that only reveal data after interaction.

wait_for object optional

A WaitFor struct, not a number. Fields are `selector` and `dom` (each `{ timeout, selector }`), `idle_network`, `idle_network0` and `almost_idle_network0` (each `{ timeout }`), `delay` (`{ timeout }`) and the boolean `page_navigations`. Every timeout is a duration object of `{ secs, nanos }`.

post /ai/browser

import os

import requests

payload = {
  "url": "https://spider.cloud/pricing",
  "prompt": "Switch the billing toggle to annual, then read back each plan name with its price.",
  "wait_for": {
    "selector": {
      "timeout": {
        "secs": 10,
        "nanos": 0
      },
      "selector": "#pricing"
    },
    "idle_network": {
      "timeout": {
        "secs": 15,
        "nanos": 0
      }
    },
    "page_navigations": True
  },
  "return_format": "markdown",
  "metadata": True
}

response = requests.post(
    "https://api.spider.cloud/ai/browser",
    headers={
        "Authorization": f"Bearer {os.environ['SPIDER_API_KEY']}",
        "Content-Type": "application/json",
    },
    json=payload,
    timeout=120,
)

print(response.json())

application/json

[
  {
    "url": "https://spider.cloud/pricing",
    "error": null,
    "status": 200,
    "content": "# Pricing\n\nAnnual billing selected.",
    "metadata": {
      "title": "Pricing",
      "extracted_data": {
        "billing_period": "annual",
        "plans": [
          { "plan": "Pro", "price": 79 }
        ]
      }
    },
    "costs": {
      "file_cost": 0.00021,
      "transform_cost": 0.00004,
      "compute_cost": 0.00038,
      "ai_cost": 0.00265,
      "bytes_transferred_cost": 0.00002,
      "total_cost": 0.0033,
      "file_cost_formatted": "$0.00021",
      "transform_cost_formatted": "$0.00004",
      "compute_cost_formatted": "$0.00038",
      "ai_cost_formatted": "$0.00265",
      "bytes_transferred_cost_formatted": "$0.00002",
      "total_cost_formatted": "$0.0033"
    },
    "duration_elasped_ms": 9310
  }
]

AI Links

POST/ai/links

Collect the links on a page that match the prompt. Link return is forced on for this route, so every page object carries a `links` array.

return_page_links boolean forced on

Always enabled for this route. You do not need to send it, and sending false does not disable it.

post /ai/links

import os

import requests

payload = {
  "url": "https://spider.cloud",
  "prompt": "Find every documentation link under the API reference.",
  "cleaning_intent": "general",
  "limit": 50,
  "return_format": "raw"
}

response = requests.post(
    "https://api.spider.cloud/ai/links",
    headers={
        "Authorization": f"Bearer {os.environ['SPIDER_API_KEY']}",
        "Content-Type": "application/json",
    },
    json=payload,
    timeout=120,
)

print(response.json())

application/json

[
  {
    "url": "https://spider.cloud",
    "error": null,
    "status": 200,
    "links": [
      "https://spider.cloud/docs/api",
      "https://spider.cloud/docs/quickstart",
      "https://spider.cloud/docs/ai"
    ],
    "metadata": {
      "title": "Spider Cloud",
      "extracted_data": {
        "docs_links": [
          { "label": "API reference", "url": "https://spider.cloud/docs/api" },
          { "label": "Quickstart", "url": "https://spider.cloud/docs/quickstart" }
        ]
      }
    },
    "costs": {
      "file_cost": 0.00021,
      "transform_cost": 0.00004,
      "compute_cost": 0.00038,
      "ai_cost": 0.00265,
      "bytes_transferred_cost": 0.00002,
      "total_cost": 0.0033,
      "file_cost_formatted": "$0.00021",
      "transform_cost_formatted": "$0.00004",
      "compute_cost_formatted": "$0.00038",
      "ai_cost_formatted": "$0.00265",
      "bytes_transferred_cost_formatted": "$0.00002",
      "total_cost_formatted": "$0.0033"
    },
    "duration_elasped_ms": 1988
  }
]

AI Unblocker

POST/ai/unblocker

Fetch a page that is behind bot protection and hand the cleaned result to the model. Use it when a normal scrape returns a challenge page.

post /ai/unblocker

import os

import requests

payload = {
  "url": "https://example.com/protected-article",
  "prompt": "Return the article headline and body text without navigation or ads.",
  "cleaning_intent": "general",
  "return_format": "markdown",
  "metadata": True
}

response = requests.post(
    "https://api.spider.cloud/ai/unblocker",
    headers={
        "Authorization": f"Bearer {os.environ['SPIDER_API_KEY']}",
        "Content-Type": "application/json",
    },
    json=payload,
    timeout=120,
)

print(response.json())

application/json

[
  {
    "url": "https://example.com/protected-article",
    "error": null,
    "status": 200,
    "content": "# The article title\n\nBody text without navigation or ads.",
    "metadata": {
      "title": "The article title",
      "extracted_data": {
        "headline": "The article title",
        "word_count": 1284
      }
    },
    "costs": {
      "file_cost": 0.00021,
      "transform_cost": 0.00004,
      "compute_cost": 0.00038,
      "ai_cost": 0.00265,
      "bytes_transferred_cost": 0.00002,
      "total_cost": 0.0033,
      "file_cost_formatted": "$0.00021",
      "transform_cost_formatted": "$0.00004",
      "compute_cost_formatted": "$0.00038",
      "ai_cost_formatted": "$0.00265",
      "bytes_transferred_cost_formatted": "$0.00002",
      "total_cost_formatted": "$0.0033"
    },
    "duration_elasped_ms": 6402
  }
]

Errors

Validation failed400

Returned with an empty body. There is no message to parse. It covers malformed JSON, a missing or empty prompt, an unrecognised field, a missing Authorization header, and an unresolvable API key. If a request fails with nothing in the body, check those first.

AI Studio subscription required 402

The account has neither an active AI Studio subscription nor credits to fall back on. Plans are on the AI Studio pricing page.

{
  "error": "AI Studio subscription required",
  "code": "ai_subscription_required",
  "url": "https://spider.cloud/ai/pricing"
}

Rate limit exceeded429

Carries a Retry-After header in seconds, and retry_after_ms in the body. Back off for that long rather than retrying immediately.

{
  "error": "Rate limit exceeded",
  "retry_after_ms": 1000
}

Rate limits

AI routes are limited by requests per second on the account's AI Studio tier, separately from the monthly credit allowance.

Starter1 request per second

600 credits per month

Lite5 requests per second

3,000 credits per month

Standard10 requests per second

12,500 credits per month

Scale25 requests per second

60,000 credits per month

Looking for the non-AI routes? Start at the API reference . For plans and included credits, see AI Studio pricing .