Crawl, search, and extract market data through one API.
Crawl competitor sites, search industry news, and pull structured data from any page. Returns clean markdown or extraction results so your team spends time on analysis instead of data wrangling.
- Stripe launches embedded finance platformtechcrunch.com · full text returned
- Global fintech funding Q1 2026: $12.8B across 340 dealscbinsights.com · full text returned
- Plaid pricing changes signal shift in BaaS economicsfinextra.com · full text returned
- Revolut expands crypto offering to 15 new marketsreuters.com · full text returned
Search, crawl, extract.
Hit the search endpoint with a query like "fintech pricing changes Q1 2026". Spider returns URLs, titles, and snippets from multiple engines. Add fetch_page_content: true to get full text in the same call.
Feed those URLs into the crawl endpoint. Spider renders JavaScript and returns clean markdown or JSON. Set limit to control depth, or crawl entire competitor domains to map their full product landscape.
Turn raw pages into structured data. Set extra_ai_data: true with a custom_prompt and Spider extracts the data you describe. Pipe the results into your warehouse, Slack, or an LLM for summarization.
Search results plus structured extraction.
import spider
client = spider.Spider()
# Search for competitor intelligence
results = client.search(
"fintech competitor pricing 2026",
params={
"num": 30,
"fetch_page_content": True,
"return_format": "markdown",
}
)
for page in results:
print(page["url"], page["title"])# Extract structured pricing data
pricing = client.crawl_url(
"https://competitor.com/pricing",
params={
"limit": 1,
"return_format": "markdown",
"extra_ai_data": True,
"custom_prompt": "Extract plan names, prices, features",
}
)
# extra_ai_data field contains extracted JSON
# pricing[0]["extra_ai_data"]What you can do with it.
Track competitor pricing
Point Spider at a pricing page and get structured plan data back. Run it weekly to spot changes before your customers do.
Monitor industry news
Use the search endpoint with fetch_page_content: true to get full article text, not just snippets. Schedule via your own cron or use Spider's built-in daily/weekly/monthly scheduling.
Check geo-specific data
Set country_code on crawl requests to route through proxies in different regions. See how a competitor prices in Germany vs. Japan without managing proxy infrastructure.
AI-powered extraction
Set extra_ai_data: true and describe the data you need in custom_prompt. Spider reads the page and extracts structured data. Pull pricing tables, feature lists, or any repeating patterns.
Keep reading.
Start collecting competitive intelligence today.
Search and crawl through one API. No contracts, no minimum commitment.