Web search API (SERP) to search and scrape

One endpoint, POST /search, runs a query against the web and hands back the result list. Turn on fetch_page_content and it also scrapes the pages it found, so a research agent gets both the SERP and the text in a single call. The same request accepts a location, a country code, a language, and a time filter. Set location to London and tbs to qdr:w to target that city and hold results to the past week instead of taking the default index.

Search and scrape in one request

Spider’s web search API returns SERP results in under 2 seconds. Set fetch_page_content and the same call scrapes every result it finds, so discovery and extraction happen in one round trip. Common use cases:

• Feeding real-time content into large language models (LLMs)
• Building intelligent agents and data pipelines
• Crawling and collecting fresh, targeted data

Search endpoint usage

POST /search Use this endpoint to compile a list of relevant websites for crawling and resource collection.

Search parameters

Time-based search filters (tbs)

Crawl parameters (when fetch_page_content is true)

When fetch_page_content is enabled, all standard crawl parameters are available alongside search parameters. These let you control how Spider processes each result URL. Common options include:

For the full list of crawl parameters, see the Scraping and Crawling docs.

Example request (Python)

import requests, os

headers = {
  'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
  'Content-Type': 'application/json',
}

json_data = {
  "search": "sports news today",
  "search_limit": 5
}

response = requests.post('https://api.spider.cloud/search', headers=headers, json=json_data)
print(response.json())

Search results format

The API returns structured results as an array of objects:

[
	{
		"title": "ESPN – Serving Sports Fans. Anytime. Anywhere.",
		"description": "Visit ESPN for live scores, highlights and sports news. Stream exclusive games on ESPN+ and play fantasy sports.",
		"url": "https://www.espn.com/"
	},
	{
		"title": "Sports Illustrated",
		"description": "SI.com provides sports news, expert analysis, highlights, stats and scores for the NFL, NBA, MLB, NHL, college football, soccer...",
		"url": "https://www.si.com/"
	}
]

Scrape every result with fetch_page_content

Set fetch_page_content: true and Spider crawls each URL it found. All standard crawl parameters work alongside search parameters, so you can control output format, depth, and proxy settings.

import requests, os

headers = {
  'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
  'Content-Type': 'application/json',
}

json_data = {
  "search": "spider web crawling",
  "return_format": "raw",
  "fetch_page_content": True,
  "search_limit": 10,
  "limit": 1
}

response = requests.post('https://api.spider.cloud/search', headers=headers, json=json_data)
print(response.json())

When fetch_page_content is enabled, the response format changes to include full crawl data:

[
	{
		"error": null,
		"status": 200,
		"duration_elasped_ms": 120,
		"costs": {
			"file_cost": 0.000363,
			"ai_cost": 0,
			"compute_cost": 7e-8,
			"transform_cost": 0,
			"total_cost": 0.00036307,
			"bytes_transferred_cost": 0
		},
		"url": "https://en.wikipedia.org/wiki/Web_crawler",
		"content": "<!DOCTYPE html><html><body>content...</body></html>"
	}
]

This is useful for:

• Extracting full content from top search results
• Automated research and summarization pipelines
• Reducing round-trips in data collection workflows

Geo-targeted search

Use location, country, and language to get results as a user in a specific region would see them.

import requests, os

headers = {
  'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
  'Content-Type': 'application/json',
}

json_data = {
  "search": "latest sports news",
  "search_limit": 5,
  "language": "en",
  "country": "us",
  "location": "San Diego, CA"
}

response = requests.post('https://api.spider.cloud/search', headers=headers, json=json_data)
print(response.json())

Time-filtered search

Restrict results to a specific time period with the tbs parameter.

import requests, os

headers = {
  'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
  'Content-Type': 'application/json',
}

json_data = {
  "search": "AI breakthroughs",
  "search_limit": 10,
  "tbs": "qdr:w"
}

response = requests.post('https://api.spider.cloud/search', headers=headers, json=json_data)
print(response.json())

Auto-pagination

Set auto_pagination: true to automatically collect results across multiple search pages until your search_limit is reached.

import requests, os

headers = {
  'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
  'Content-Type': 'application/json',
}

json_data = {
  "search": "machine learning tutorials",
  "search_limit": 50,
  "auto_pagination": True
}

response = requests.post('https://api.spider.cloud/search', headers=headers, json=json_data)
print(response.json())

Batch multiple queries

Send an array of query objects to execute multiple searches in a single API call. Each query runs independently and returns its own result set. Streaming is not supported for batch requests.

import requests, os

headers = {
  'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
  'Content-Type': 'application/json',
}

json_data = [
  {
    "search": "latest sports news united states",
    "search_limit": 5
  },
  {
    "search": "latest news around the globe",
    "search_limit": 5
  }
]

response = requests.post('https://api.spider.cloud/search', headers=headers, json=json_data)
print(response.json())

Rate limits

• Up to 10,000 search requests per minute
• Multiple search providers for redundancy
• Distributed crawling and parsing for fetched content