Business Intelligence
Gather Market Intelligence
at Scale
Crawl competitor sites, search industry news, and pull structured data from any page. The API returns clean markdown or structured extraction results so your team spends time on analysis, not data wrangling.
A typical research pipeline
SEARCH
Hit the search endpoint with a query like "fintech pricing changes Q1 2026". Spider
returns URLs, titles, and snippets from multiple engines. Add
fetch_page_content: true
to get full text in the same call.
CRAWL
Feed those URLs into the crawl endpoint. Spider renders JavaScript and returns clean
markdown or JSON. Set limit to control depth, or crawl entire
competitor domains to map their full product landscape.
EXTRACT
Turn raw pages into structured data. Set
extra_ai_data: true
with a custom_prompt and Spider extracts the data you
describe. Pipe the results into your warehouse, Slack, or an LLM for summarization.
Two calls, full pipeline
import spider
client = spider.Spider()
# Search for competitor intelligence
results = client.search(
"fintech competitor pricing 2026",
params={
"num": 30,
"fetch_page_content": True,
"return_format": "markdown",
}
)
for page in results:
print(page["url"], page["title"])# Extract structured pricing data
pricing = client.crawl_url(
"https://competitor.com/pricing",
params={
"limit": 1,
"return_format": "markdown",
"extra_ai_data": True,
"custom_prompt": "Extract plan names, prices, features",
}
)
# extra_ai_data field contains extracted JSON
# pricing[0]["extra_ai_data"]What you can do with it
Track competitor pricing
Point Spider at a pricing page and get structured plan data back. Run it weekly to spot changes before your customers do.
Monitor industry news
Use the search endpoint with fetch_page_content: true to get
full article text, not just snippets. Schedule searches via your own cron job or use Spider's
built-in daily/weekly/monthly scheduling.
Check geo-specific data
Set country_code on crawl requests to route through proxies
in different regions. See how a competitor prices in Germany vs. Japan without managing your
own proxy infrastructure.
AI-powered extraction
Set extra_ai_data: true and describe the data you need in
custom_prompt
. Spider reads the page and extracts structured data. Pull pricing tables,
feature lists, or any repeating patterns.