AI API Reference

Spider's AI endpoints enhance standard web data extraction with natural language understanding. All endpoints accept a prompt parameter that guides AI-powered extraction and automation.

Base URL:https://api.spider.cloud

POST/ai/crawl

AI Crawl

Crawl websites intelligently using natural language prompts. The AI analyzes your prompt to determine crawl depth, page filtering, and content extraction strategies.

Parameters

Name	Type	Required	Description
url	string	required	Starting URL to crawl
prompt	string	required	Natural language instruction for what to crawl and extract
limit	number	optional	Maximum pages to crawl
return_format	string	optional	Output format: markdown, html, text, or raw
extraction_schema	object	optional	JSON Schema for structured extraction with name, description, and schema fields
metadata	boolean	optional	Include metadata with extracted_data in response

Example Request

cURL

curl -X POST https://api.spider.cloud/ai/crawl \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://example.com",
  "prompt": "Find all blog posts and extract titles and summaries",
  "limit": 50,
  "return_format": "markdown"
}'

Python

import requests

response = requests.post(
    "https://api.spider.cloud/ai/crawl",
    headers={
        "Authorization": "Bearer YOUR_API_KEY",
        "Content-Type": "application/json"
    },
    json={
        "url": "https://example.com",
        "prompt": "Find all blog posts and extract titles and summaries",
        "limit": 50,
        "return_format": "markdown"
    }
)

print(response.json())

Example Response

{
  "status": 200,
  "data": [
    {
      "url": "https://example.com/blog/post-1",
      "title": "Getting Started with AI",
      "summary": "An introduction to artificial intelligence..."
    }
  ],
  "ai_cost": 0.002
}

POST/ai/scrape

AI Scrape

Extract structured data from any webpage using plain English prompts. AI automatically identifies and extracts the data you describe. Use extraction_schema for typed JSON output.

Parameters

Name	Type	Required	Description
url	string	required	URL to scrape
prompt	string	required	Natural language description of data to extract
return_format	string	optional	Output format: json, markdown, raw, html, text
extraction_schema	object	optional	JSON Schema for structured extraction with name, description, and schema fields
metadata	boolean	optional	Include metadata with extracted_data in response
request	string	optional	Request type: http or chrome for JavaScript rendering

Example Request

cURL

curl -X POST https://api.spider.cloud/ai/scrape \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
  "prompt": "Extract book details",
  "return_format": "raw",
  "extraction_schema": {
    "name": "BookDetails",
    "description": "Product information from a book listing",
    "schema": "{\"type\":\"object\",\"properties\":{\"title\":{\"type\":\"string\"},\"price\":{\"type\":\"string\"},\"availability\":{\"type\":\"string\"},\"description\":{\"type\":\"string\"}},\"required\":[\"title\",\"price\"]}"
  },
  "metadata": true,
  "request": "chrome"
}'

Python

import requests

response = requests.post(
    "https://api.spider.cloud/ai/scrape",
    headers={
        "Authorization": "Bearer YOUR_API_KEY",
        "Content-Type": "application/json"
    },
    json={
        "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
        "prompt": "Extract book details",
        "return_format": "raw",
        "extraction_schema": {
            "name": "BookDetails",
            "description": "Product information from a book listing",
            "schema": "{\"type\":\"object\",\"properties\":{\"title\":{\"type\":\"string\"},\"price\":{\"type\":\"string\"},\"availability\":{\"type\":\"string\"},\"description\":{\"type\":\"string\"}},\"required\":[\"title\",\"price\"]}"
        },
        "metadata": true,
        "request": "chrome"
    }
)

print(response.json())

Example Response

{
  "status": 200,
  "content": "...",
  "metadata": {
    "extracted_data": {
      "title": "A Light in the Attic",
      "price": "£51.77",
      "availability": "In stock (22 available)",
      "description": "It's hard to imagine a world without A Light in the Attic..."
    }
  },
  "costs": {
    "ai_cost": 0.001,
    "total_cost": 0.002
  }
}

POST/ai/search

AI Search

Search the web with AI-enhanced queries. Natural language understanding provides more relevant results than keyword-based search.

Parameters

Name	Type	Required	Description
prompt	string	required	Natural language search query
search_limit	number	optional	Maximum number of search results
return_format	string	optional	Output format for results

Example Request

cURL

curl -X POST https://api.spider.cloud/ai/search \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "prompt": "Find the best Python web scraping libraries released in 2024",
  "search_limit": 10,
  "return_format": "markdown"
}'

Python

import requests

response = requests.post(
    "https://api.spider.cloud/ai/search",
    headers={
        "Authorization": "Bearer YOUR_API_KEY",
        "Content-Type": "application/json"
    },
    json={
        "prompt": "Find the best Python web scraping libraries released in 2024",
        "search_limit": 10,
        "return_format": "markdown"
    }
)

print(response.json())

Example Response

{
  "status": 200,
  "data": [
    {
      "title": "Top Python Scraping Libraries 2024",
      "url": "https://example.com/python-scraping",
      "snippet": "A comprehensive guide to the best scraping tools..."
    }
  ],
  "ai_cost": 0.003
}

POST/ai/browser

AI Browser

Automate browser interactions using natural language. Describe actions in plain English and AI configures the automation.

Parameters

Name	Type	Required	Description
url	string	required	URL to automate
prompt	string	required	Natural language description of browser actions
wait_for	number	optional	Wait time between actions in ms
screenshot	boolean	optional	Capture screenshot after actions

Example Request

cURL

curl -X POST https://api.spider.cloud/ai/browser \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://example.com/login",
  "prompt": "Click the sign in button, wait for the form, fill email with test@example.com",
  "wait_for": 2000
}'

Python

import requests

response = requests.post(
    "https://api.spider.cloud/ai/browser",
    headers={
        "Authorization": "Bearer YOUR_API_KEY",
        "Content-Type": "application/json"
    },
    json={
        "url": "https://example.com/login",
        "prompt": "Click the sign in button, wait for the form, fill email with test@example.com",
        "wait_for": 2000
    }
)

print(response.json())

Example Response

{
  "status": 200,
  "data": {
    "success": true,
    "actions_performed": [
      "clicked: sign in button",
      "waited: 1000ms",
      "filled: email field"
    ],
    "screenshot": "base64..."
  },
  "ai_cost": 0.005
}

POST/ai/links

AI Links

Extract and filter links from webpages using AI guidance. Describe the types of links you want to find and the AI will intelligently filter and categorize them.

Parameters

Name	Type	Required	Description
url	string	required	URL to extract links from
prompt	string	required	Natural language description of what links to find
limit	number	optional	Maximum links to return
depth	number	optional	Crawl depth for finding links

Example Request

cURL

curl -X POST https://api.spider.cloud/ai/links \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://example.com",
  "prompt": "Find all links to product pages and documentation",
  "limit": 100
}'

Python

import requests

response = requests.post(
    "https://api.spider.cloud/ai/links",
    headers={
        "Authorization": "Bearer YOUR_API_KEY",
        "Content-Type": "application/json"
    },
    json={
        "url": "https://example.com",
        "prompt": "Find all links to product pages and documentation",
        "limit": 100
    }
)

print(response.json())

Example Response

{
  "status": 200,
  "data": {
    "links": [
      {
        "url": "https://example.com/products/widget",
        "category": "product",
        "text": "Widget Pro"
      },
      {
        "url": "https://example.com/docs/getting-started",
        "category": "documentation",
        "text": "Getting Started"
      }
    ],
    "total": 2
  },
  "ai_cost": 0.002
}