Concurrent streaming

Read each page as it finishes instead of waiting for the whole crawl. Use it for any crawl past a few dozen pages: less memory, no HTTP timeout on a long job, and the first result arrives in under a second.

How it works

Set Content-Type to application/jsonl and turn on streaming in your HTTP client. Spider writes each page as one JSON line the moment it finishes. The crawler still runs at full concurrency, so pages arrive as fast as they are crawled, not in any fixed order.

When to stream

Three cases where it pays off.

Large crawls. Any limit over 50 pages. Without streaming the response buffers until the crawl ends, which can hit an HTTP timeout or a memory limit.
Live pipelines. Embed or store each page as it lands, for example feeding a vector database during the crawl.
Progress. Each line names the page and its status, so you can show progress or log metrics as you go.

Python with requests

Pass stream=True and iterate over lines as they arrive.

Python with requests

import requests, json, os

def process_page(page: dict):
    url = page.get("url", "unknown")
    status = page.get("status", 0)
    content = page.get("content", "")
    print(f"Crawled: {url} ({status}) - {len(content)} chars")

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/jsonl',
}

response = requests.post(
    'https://api.spider.cloud/crawl',
    headers=headers,
    json={
        "url": "https://www.example.com",
        "limit": 100,
        "depth": 3,
        "request": "smart",
        "return_format": "markdown"
    },
    stream=True,
    timeout=120
)

response.raise_for_status()

for line in response.iter_lines(decode_unicode=True):
    if line:
        page = json.loads(line)
        process_page(page)

Python with the SDK

The Python SDK streams through the stream and callback parameters.

Python SDK

from spider import Spider

app = Spider()

def handle_page(page: dict) -> None:
    print(f"Crawled: {page['url']} ({page['status']})")

result = app.crawl_url(
    "https://www.example.com",
    params={
        "limit": 100,
        "depth": 3,
        "request": "smart",
        "return_format": "markdown"
    },
    stream=True,
    callback=handle_page,
)

Node.js

Read the fetch body as a stream and split it on newlines.

Node.js

const response = await fetch('https://api.spider.cloud/crawl', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${process.env.SPIDER_API_KEY}`,
    'Content-Type': 'application/jsonl',
  },
  body: JSON.stringify({
    url: 'https://www.example.com',
    limit: 100,
    depth: 3,
    request: 'smart',
    return_format: 'markdown',
  }),
});

const reader = response.body.getReader();
const decoder = new TextDecoder();
let buffer = '';

while (true) {
  const { done, value } = await reader.read();
  if (done) break;

  buffer += decoder.decode(value, { stream: true });
  const lines = buffer.split('\n');
  buffer = lines.pop(); // Keep incomplete line in buffer

  for (const line of lines) {
    if (line.trim()) {
      const page = JSON.parse(line);
      console.log(`Crawled: ${page.url} (${page.status})`);
    }
  }
}

Tip

One HTTP connection carries the whole crawl, so per-request rate limits do not apply.