Realtime Search

The /search endpoint returns live web search results with full content extraction. Use it to find relevant pages by keyword, then optionally scrape each result in the same request — combining discovery and data collection in one step.

Basic Web Search

Send a search query to get ranked results from across the web. Set search_limit to control how many results are returned, and page for pagination. Results include the URL, title, and description for each match.

Basic web search

import requests, os

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

params = {
    "search": "spider web crawling", # search query
    "search_limit": 10, # number of search results to return
    "page": 1, # the pagination page for results
}

response = requests.post(
    'https://api.spider.cloud/search', # set to search endpoint
    headers=headers,
    json=params
)

print(response.json())

Response

{
    "content": [
        {
            "description": "What is Spider? Spider is a leading web crawling tool designed for speed and cost-effectiveness, supporting various data formats including LLM-ready markdown.",
            "title": "Spider: The Web Crawler for AI",
            "url": "https://spider.cloud/"
        },
        {
            "description": "Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and ...",
            "title": "Web crawler",
            "url": "https://en.wikipedia.org/wiki/Web_crawler"
        },
      // more
    ]
}

Pro Tip:

Advanced search operators are supported. Learn more about search operators.

Search and Scrape in One Request

Set fetch_page_content: true to fetch the full page content for each search result. Spider visits each URL and returns the content in your chosen format. If limit is greater than 1, Spider will crawl additional pages from each result URL up to the limit.

Search and scrape

import requests, os

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

params = {
    "search": "spider web crawling",
    "return_format": "raw", # return the raw html of the page. Other formats are supported
    "fetch_page_content": True,
    "search_limit": 10,
    "page": 1,
    "limit": 1, # Setting 1 scrapes only the first page of the URL found in each search result
}

response = requests.post(
    'https://api.spider.cloud/search', # set to search endpoint
    headers=headers,
    json=params
)

print(response.json())

Response

[
    {
        "error": null,
        "status": 200,
        "duration_elasped_ms": 120,
        "costs": {
            "file_cost": 0.000363,
            "ai_cost": 0,
            "compute_cost": 7e-8,
            "transform_cost": 0,
            "total_cost": 0.00036307,
            "bytes_transferred_cost": 0
        },
        "url": "https://en.wikipedia.org/wiki/Web_crawler",
        "content": "content..."
    },
    {
        "error": null,
        "costs": {
            "bytes_transferred_cost": 0,
            "total_cost": 0.00033178,
            "file_cost": 0.00032506,
            "ai_cost": 0,
            "transform_cost": 0,
            "compute_cost": 6.72e-6
        },
        "status": 200,
        "duration_elasped_ms": 160,
        "url": "https://spider.cloud/",
        "content": "content..."
    }
        // more results
]

Batch Multiple Queries

Send an array of query objects to execute multiple searches in a single API call. Each query runs independently and returns its own result set. This is especially useful in agentic workflows where you need answers to several questions at once. Note: streaming is not currently supported for batch search requests.

Send multiple queries

import requests, os

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

params = [
	{
	    "search": "latest sports news united states",
	    "search_limit": 5 # 5 results
    },
    {
	    "search": "latest news around the globe",
	    "search_limit": 5 # 5 results
	}
]

response = requests.post(
    'https://api.spider.cloud/search',
    headers=headers,
    json=params
)

print(response.json())

Response

[
    {
        "content": [
            {
                "description": "Visit ESPN for live scores, highlights and sports news. Stream exclusive games on ESPN+ and play fantasy sports.",
                "title": "ESPN - Serving Sports Fans. Anytime. Anywhere.",
                "url": "https://www.espn.com/"
            },
            {
                "description": "Yahoo Sports - NBC Sports Network.",
                "title": "Yahoo Sports: News, Scores, Video, Fantasy Games, Schedules & More - Yahoo Sports",
                "url": "https://sports.yahoo.com/"
            },
            {
                "description": "8 hours ago - Latest sports news from around the world with in-depth analysis, features, photos and videos covering football, tennis, motorsport, golf, rugby, sailing, skiing, horse racing and equestrian.",
                "title": "Latest sports news, videos, interviews and comment | CNN",
                "url": "https://www.cnn.com/sport"
            },
            // 2 more results
        ]
    },
    {
        "content": [
            {
                "description": "Gaza health workers say four killed by Israeli gunfire near aid centre. It is the latest deadly incident to occur near aid distribution points set up by the ...",
                "title": "World | Latest News & Updates",
                "url": "https://www.bbc.com/news/world"
            },
            {
                "description": "Kharkiv hit by 'most powerful attack' of entire war, mayor says, as Russia pounds Ukraine again · Israeli military says it recovered body of Thai hostage from ...",
                "title": "World news - breaking news, video, headlines and opinion",
                "url": "https://www.cnn.com/world"
            },
            {
                "description": "Most viewed in world news · Live · IDF ordered to stop Gaza-bound aid ship carrying Greta Thunberg · Ukraine war briefing: Poland scrambles planes to secure ...",
                "title": "Latest news from around the world",
                "url": "https://www.theguardian.com/world"
            },
            // 2 more results
        ]
    }
]

Geo-Targeted Search

Control where search results originate from using location, language, and country parameters. This is useful for localized queries — retrieving results as a user in a specific region would see them.

Targeted location, language and country

import requests, os

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

params = {
	    "search": "latest sports news",
	    "search_limit": 5,
	    "language": "en", # language of the search results
	    "country": "us", # prioritize search results from this country
	    "location": "San Diego, CA" # Where search originates from
    }

response = requests.post(
    'https://api.spider.cloud/search',
    headers=headers,
    json=params
)

print(response.json())

Time-based search filters by hour, day, week, month, or year

import requests, os

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

params = {
	    "search": "latest sports news",
	    "search_limit": 5,
	    "tbs": "qdr:w" # past week
    }

response = requests.post(
    'https://api.spider.cloud/search',
    headers=headers,
    json=params
)

print(response.json())

TBS Parameter Value	Description
`qdr:h`	Past hour
`qdr:d`	Past 24 hours
`qdr:w`	Past week
`qdr:m`	Past month
`qdr:y`	Past year