AI Crawl
Intelligent website crawling guided by natural language
AI Crawl combines Spider's high-performance crawler with AI understanding. Describe what you want to find, and the AI will guide the crawler to discover relevant pages, filter content, and extract structured data.
POST /ai/crawlFull API DocsFeatures
Natural language crawl guidance
Smart page relevance filtering
Automatic depth optimization
Structured data extraction
Concurrent crawling at scale
JavaScript rendering support
Example Use Cases
Find blog posts
"Crawl all blog posts and extract titles, dates, and summaries"
Product discovery
"Find all product pages with prices under $100"
Documentation scraping
"Collect all API documentation pages with code examples"
News aggregation
"Find recent news articles about artificial intelligence"
Parameters
| Name | Type | Required | Description |
|---|---|---|---|
| url | string | required | Starting URL to crawl |
| prompt | string | required | Natural language crawl instructions |
| limit | number | optional | Maximum pages to crawl (default: 25) |
| depth | number | optional | Maximum crawl depth |
| return_format | string | optional | Output format: markdown, html, text |
Code Examples
cURL
curl -X POST https://api.spider.cloud/ai/crawl \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com",
"prompt": "Find all product pages and extract names and prices",
"limit": 50
}'Python
from spider import Spider
spider = Spider()
result = spider.ai_crawl(
url="https://example.com",
prompt="Find all product pages and extract names and prices",
limit=50
)
for page in result:
print(page)JavaScript
import { Spider } from '@spider-cloud/spider-client';
const spider = new Spider();
const result = await spider.aiCrawl({
url: 'https://example.com',
prompt: 'Find all product pages and extract names and prices',
limit: 50
});
console.log(result);