Realtime Search

Live results from across the internet with high rate limits.

Perform High-Throughput Web Searches

Do real-time web search using the /search endpoint. In the following example, 10 results are returned in the response. The results in the response are listed according to their original search ranking.

Basic web search

import requests, os

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

params = {
    "search": "spider web crawling", # search query
    "search_limit": 10, # number of search results to return
}

response = requests.post(
    'https://api.spider.cloud/search', # set to search endpoint
    headers=headers,
    json=params
)

print(response.json())

Pro Tip:

Advanced search operators are supported. Learn more about search operators.

Response

{
    "content": [
        {
            "description": "What is Spider? Spider is a leading web crawling tool designed for speed and cost-effectiveness, supporting various data formats including LLM-ready markdown.",
            "title": "Spider: The Web Crawler for AI",
            "url": "https://spider.cloud/"
        },
        {
            "description": "Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and ...",
            "title": "Web crawler",
            "url": "https://en.wikipedia.org/wiki/Web_crawler"
        },
      // more
    ]
}

Combine Search and Scrape

The next example shows how to search and return the page content for each search result. If limit is more than 1, then we'll crawl the site up to this max limit per URL.

Search and scrape

import requests, os

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

params = {
    "search": "spider web crawling",
    "return_format": "raw", # return the raw html of the page. Other formats are supported
    "fetch_page_content": True,
    "search_limit": 10,
    "limit": 1, # Setting 1 scrapes only the first page of the URL found in each search result
}

response = requests.post(
    'https://api.spider.cloud/search', # set to search endpoint
    headers=headers,
    json=params
)

print(response.json())

Response

[
    {
        "error": null,
        "status": 200,
        "costs": {
            "file_cost": 0.000363,
            "ai_cost": 0,
            "compute_cost": 7e-8,
            "transform_cost": 0,
            "total_cost": 0.00036307,
            "bytes_transferred_cost": 0
        },
        "url": "https://en.wikipedia.org/wiki/Web_crawler",
        "content": "content..."
    },
    {
        "error": null,
        "costs": {
            "bytes_transferred_cost": 0,
            "total_cost": 0.00033178,
            "file_cost": 0.00032506,
            "ai_cost": 0,
            "transform_cost": 0,
            "compute_cost": 6.72e-6
        },
        "status": 200,
        "url": "https://spider.cloud/",
        "content": "content..."
    }
        // more results
]

Send multiple queries at the same time

Send multiple queries at the same time by sending a list of queries. This is useful for agentic workflows that require multiple queries to be sent at the same time. Take note that Streaming is not supported at this time.

Send multiple queries

import requests, os

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

params = [
	{
	    "search": "latest sports news united states",
	    "search_limit": 5 # 5 results
    },
    {
	    "search": "latest news around the globe",
	    "search_limit": 5 # 5 results
	}
]

response = requests.post(
    'https://api.spider.cloud/search',
    headers=headers,
    json=params
)

print(response.json())

Response

[
    {
        "content": [
            {
                "description": "Visit ESPN for live scores, highlights and sports news. Stream exclusive games on ESPN+ and play fantasy sports.",
                "title": "ESPN - Serving Sports Fans. Anytime. Anywhere.",
                "url": "https://www.espn.com/"
            },
            {
                "description": "Yahoo Sports - NBC Sports Network.",
                "title": "Yahoo Sports: News, Scores, Video, Fantasy Games, Schedules & More - Yahoo Sports",
                "url": "https://sports.yahoo.com/"
            },
            {
                "description": "8 hours ago - Latest sports news from around the world with in-depth analysis, features, photos and videos covering football, tennis, motorsport, golf, rugby, sailing, skiing, horse racing and equestrian.",
                "title": "Latest sports news, videos, interviews and comment | CNN",
                "url": "https://www.cnn.com/sport"
            },
            // 2 more results
        ]
    },
    {
        "content": [
            {
                "description": "Gaza health workers say four killed by Israeli gunfire near aid centre. It is the latest deadly incident to occur near aid distribution points set up by the ...",
                "title": "World | Latest News & Updates",
                "url": "https://www.bbc.com/news/world"
            },
            {
                "description": "Kharkiv hit by 'most powerful attack' of entire war, mayor says, as Russia pounds Ukraine again · Israeli military says it recovered body of Thai hostage from ...",
                "title": "World news - breaking news, video, headlines and opinion",
                "url": "https://www.cnn.com/world"
            },
            {
                "description": "Most viewed in world news · Live · IDF ordered to stop Gaza-bound aid ship carrying Greta Thunberg · Ukraine war briefing: Poland scrambles planes to secure ...",
                "title": "Latest news from around the world",
                "url": "https://www.theguardian.com/world"
            },
            // 2 more results
        ]
    }
]

Advanced Search Parameters

Set the location, language, and country to target events more precisely on the spot.

Targgeted location, language and country

import requests, os

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

params = {
	    "search": "latest sports news",
	    "search_limit": 5,
	    "language": "en", # language of the search results
	    "country": "us", # prioritize search results from this country
	    "location": "San Diego, CA" # Where search originates from
    }

response = requests.post(
    'https://api.spider.cloud/search',
    headers=headers,
    json=params
)

print(response.json())

Time-based search filters by hour, day, week, month, or year

import requests, os

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

params = {
	    "search": "latest sports news",
	    "search_limit": 5,
	    "tbs": "qdr:w" # past week
    }

response = requests.post(
    'https://api.spider.cloud/search',
    headers=headers,
    json=params
)

print(response.json())

TBS Parameter Value	Description
`qdr:h`	Past hour
`qdr:d`	Past 24 hours
`qdr:w`	Past week
`qdr:m`	Past month
`qdr:y`	Past year