Search query in. Crawled pages out.
Spider runs the query against the major search engines, walks the top results, and returns extracted content. One call, one bill, one JSON shape.
One query, many results.
A single search request fans out across the web and converges back to structured content.
Four steps from query to data.
You send a query
A plain-text search query — the same kind you'd type into a search engine. Set search_limit to control how many results to crawl.
Spider searches the web
Queries major search engines and collects the top result URLs. Use country_code for geo-targeted results.
Each result is crawled
Every discovered page is loaded, rendered with a real browser if needed, and its content extracted in your chosen format.
Clean content returned
Structured content from every result page delivered as markdown, text, HTML, or raw bytes — ready for your pipeline.
What you can configure.
Configurable result count
Set search_limit to control how many search results Spider crawls — from 1 to dozens. Balance thoroughness against cost and latency.
Deep crawl results
Combine search_limit with limit to not just scrape result pages but crawl deeper into each discovered site.
Geo-targeted search
Use country_code to get localized results. See what users in different regions find for the same query.
All output formats
Get search-sourced content as markdown, text, HTML, or raw bytes — the same formatting and cleaning as crawl and scrape.
Metadata enrichment
Enable metadata to get page titles, descriptions, and keywords alongside extracted content for building search indexes.
Streaming support
Use JSONL content type to stream results as each page is crawled and processed. Start consuming data immediately.
cURL, Python, Node.
from spider import Spider
client = Spider()
# Search for a topic and get content from top results
results = client.search(
"RAG best practices for LLMs",
params={
"search_limit": 5,
"return_format": "markdown",
"metadata": True,
},
)
for page in results:
print(f"{page['url']}: {page['metadata']['title']}")Run a real search.
Type any search query. Spider will search the web, crawl the top results, and return clean content. No sign-up required.
Type a query and hit Search to see live results from Spider's Search API.
No sign-in required. 25 free searches per day.
Built for production AI workloads.
No rate limit walls
Most search APIs cap at 100-500 concurrent requests. Spider handles 10,000 per minute. Your agent keeps working during traffic spikes.
Batch multiple queries
Send an array of search queries in a single API call. Spider runs them all in parallel and returns grouped results. One round trip instead of N.
Auto-pagination
Set search_limit to 50 and auto_pagination to true. Spider pages through search results automatically. No manual page incrementing.
How Spider's Search compares.
Stacked against alternatives for AI and LLM use cases.
| Feature | Spider | Firecrawl | Jina | SerpAPI |
|---|---|---|---|---|
| Search + scrape | One call | One call | Snippets only | Search only |
| Batch queries | Yes | No | No | No |
| Auto-pagination | Yes | No | No | Manual |
| Rate limit | 10K/min | 2-150 | 500 RPM | 5K/mo |
| Geo-targeting | City-level | Limited | No | Country |
| Time filters | 5 levels | Yes | No | Yes |
| Output formats | 4+ | Markdown | Markdown | JSON |
| Avg cost/search | ~$0.003 | ~$0.01 | Free* | $0.01 |
*Jina free tier is rate-limited to 500 RPM with 10M token cap.
Describe what you need in plain English.
Spider's AI models optimize your query, search the web, extract relevant content, and return structured results. No manual parameter tuning needed.
# Natural language search
curl -X POST https://api.spider.cloud/ai/search \
-H "Authorization: Bearer $KEY" \
-d '{
"prompt": "Find recent benchmarks comparing RAG frameworks",
"search_limit": 5,
"return_format": "markdown"
}'Where teams reach for it.
RAG with live web data
Feed a user's question into the search API, retrieve relevant pages, and pass the content to an LLM for grounded, up-to-date answers. No need to maintain a static corpus.
Market research
Search for competitor names, product categories, or industry terms and automatically collect the latest information from multiple sources in a single request.
Content curation
Build automated pipelines that discover and aggregate the best articles on specific topics for newsletters, research reports, or knowledge bases.
SERP monitoring
Track how search results change over time for keywords that matter to your business. Compare results across regions using country_code targeting.
More from the API.
Search the web programmatically.
Combine search discovery with content extraction in a single API call.