Realtime search

/search returns live web results. Set fetch_page_content and the same call scrapes every result too.

Search the web

POST a search query to get ranked results. search_limit caps how many come back; page picks the page of results. Each result carries the URL, title, and description.

import requests, os

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

params = {
    "search": "spider web crawling",   # query
    "search_limit": 10,                # results to return
    "page": 1,                         # pagination page
}

response = requests.post(
    'https://api.spider.cloud/search',
    headers=headers,
    json=params,
)

print(response.json())

{
    "content": [
        {
            "description": "Spider is a leading web crawling tool designed for speed and cost-effectiveness, supporting various data formats including LLM-ready markdown.",
            "title": "Spider: The Web Crawler for AI",
            "url": "https://spider.cloud/"
        },
        {
            "description": "Web crawler, sometimes called a spider or spiderbot, is an Internet bot that systematically browses the World Wide Web …",
            "title": "Web crawler",
            "url": "https://en.wikipedia.org/wiki/Web_crawler"
        }
        // more
    ]
}

Tip

Search operators such as site:, intitle: and quoted phrases work inside the search string.

Search and scrape in one call

Set fetch_page_content: true and Spider visits every result URL and returns the body in your chosen format. Raise limit above 1 to crawl deeper from each result.

import requests, os

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

params = {
    "search": "spider web crawling",
    "return_format": "raw",            # also: markdown, text, html
    "fetch_page_content": True,
    "search_limit": 10,
    "page": 1,
    "limit": 1,                        # 1 = scrape only the top hit
}

response = requests.post(
    'https://api.spider.cloud/search',
    headers=headers,
    json=params,
)

print(response.json())

[
    {
        "error": null,
        "status": 200,
        "duration_elasped_ms": 120,
        "costs": {
            "file_cost": 0.000363,
            "ai_cost": 0,
            "compute_cost": 7e-8,
            "transform_cost": 0,
            "total_cost": 0.00036307,
            "bytes_transferred_cost": 0
        },
        "url": "https://en.wikipedia.org/wiki/Web_crawler",
        "content": "<!DOCTYPE html><html><body>content...</body></html>"
    },
    {
        "error": null,
        "costs": { "total_cost": 0.00033178 },
        "status": 200,
        "duration_elasped_ms": 160,
        "url": "https://spider.cloud/",
        "content": "<!DOCTYPE html><html><body>content...</body></html>"
    }
    // more
]

Run several queries in one request

POST an array of query objects. Each query runs on its own and returns its own result set. Handy when an agent needs a handful of answers at once. Batch search does not stream.

import requests, os

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

params = [
    {"search": "latest sports news united states", "search_limit": 5},
    {"search": "latest news around the globe",     "search_limit": 5},
]

response = requests.post(
    'https://api.spider.cloud/search',
    headers=headers,
    json=params,
)

print(response.json())

[
    {
        "content": [
            { "title": "ESPN", "url": "https://www.espn.com/", "description": "Live scores, highlights, sports news…" },
            { "title": "Yahoo Sports", "url": "https://sports.yahoo.com/", "description": "News, scores, video, fantasy games…" }
            // 3 more
        ]
    },
    {
        "content": [
            { "title": "World news | BBC", "url": "https://www.bbc.com/news/world", "description": "Latest world headlines…" },
            { "title": "World news | CNN",  "url": "https://www.cnn.com/world",     "description": "Breaking world news…" }
            // 3 more
        ]
    }
]

Search from a location

Set location, language and country to get the results a user in that region would see.

import requests, os

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

params = {
    "search": "latest sports news",
    "search_limit": 5,
    "language": "en",                 # result language
    "country": "us",                  # prioritize US results
    "location": "San Diego, CA",      # search origin
}

response = requests.post(
    'https://api.spider.cloud/search',
    headers=headers,
    json=params,
)

print(response.json())

Filter by time

tbs restricts results to a recent window, from the past hour to the past year.

import requests, os

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

params = {
    "search": "latest sports news",
    "search_limit": 5,
    "tbs": "qdr:w",                    # past week
}

response = requests.post(
    'https://api.spider.cloud/search',
    headers=headers,
    json=params,
)

print(response.json())

tbs value	Window
qdr:h	Past hour
qdr:d	Past 24 hours
qdr:w	Past week
qdr:m	Past month
qdr:y	Past year