AI Crawl

Intelligent website crawling guided by natural language

AI Crawl combines Spider's high-performance crawler with AI understanding. Describe what you want to find, and the AI will guide the crawler to discover relevant pages, filter content, and extract structured data.

POST/ai/crawl

Full API Docs

FEATURES

Capabilities

Natural language crawl guidance

Smart page relevance filtering

Automatic depth optimization

Structured data extraction

Concurrent crawling at scale

JavaScript rendering support

USE CASES

Example Prompts

Find blog posts

prompt: "Crawl all blog posts and extract titles, dates, and summaries"

Product discovery

prompt: "Find all product pages with prices under $100"

Documentation scraping

prompt: "Collect all API documentation pages with code examples"

News aggregation

prompt: "Find recent news articles about artificial intelligence"

API

Parameters

Name	Type	Status	Description
url	string	required	Starting URL to crawl
prompt	string	required	Natural language crawl instructions
limit	number	optional	Maximum pages to crawl (default: 25)
depth	number	optional	Maximum crawl depth
return_format	string	optional	Output format: markdown, html, text, or raw. Defaults to empty (only extracted data returned)

CODE

Code Examples

curl -X POST https://api.spider.cloud/ai/crawl \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "prompt": "Find all product pages and extract names and prices",
    "limit": 50
  }'

Ready to use AI Crawl?

Start extracting data with natural language prompts. No complex selectors or scripts needed.

Create Account Full Documentation