Reliability

Streams drop. Servers restart. Deploys kill long-running connections. When crawling thousands of pages, you need results to land even when your client doesn't. Data connectorsand webhooksrun server-side, so your data is safe regardless of what happens to your connection. Below are four patterns for combining them with streaming.

Stream + data connector

Stream JSONL so you can process pages as they arrive, and attach a data connector so Spider Cloud writes every page server-side too. If your connection drops, the connector already has the data.

Stream JSONL with S3 backup

import requests, os, json headers = { "Authorization": f"Bearer {os.getenv('SPIDER_API_KEY')}", "Content-Type": "application/jsonl", } response = requests.post("https://api.spider.cloud/crawl", headers=headers, json={ "url": "https://example.com", "limit": 100, "return_format": "markdown", "data_connectors": { "s3": { "bucket": "my-crawl-data", "access_key_id": os.getenv("AWS_ACCESS_KEY_ID"), "secret_access_key": os.getenv("AWS_SECRET_ACCESS_KEY"), "region": "us-west-2", "prefix": "crawls/" }, "on_find": True } }, stream=True) # Process in real time while S3 has your backup for line in response.iter_lines(): if line: page = json.loads(line) print(f"Got {page['url']}")

Tip

You can Ctrl+C the client and the connector keeps writing. Delivery is server-side, independent of your stream. See Data Connectorsfor provider setup.

Background crawl with webhook

Set run_in_background: true, point a connector at your storage, and add an on_website_status webhook. The API returns immediately. Pages accumulate in your bucket or database while Spider crawls. When it finishes, you get a webhook. Best for cron jobs and large batch crawls.

Background crawl with Supabase and webhook

import requests, os headers = { "Authorization": f"Bearer {os.getenv('SPIDER_API_KEY')}", "Content-Type": "application/json", } response = requests.post("https://api.spider.cloud/crawl", headers=headers, json={ "url": "https://example.com", "limit": 500, "return_format": "markdown", "run_in_background": True, "data_connectors": { "supabase": { "url": "https://your-project.supabase.co", "anon_key": os.getenv("SUPABASE_ANON_KEY"), "table": "crawled_pages" }, "on_find": True }, "webhook": { "url": "https://your-server.com/crawl-done", "on_website_status": True } }) # Returns immediately with a crawl_id print(response.json())

JSONL checkpointing

For smaller crawls, you can track progress client-side with blacklist. Collect URLs as you consume the stream. If the connection drops, pass them back so Spider skips pages you already have. This works well up to a few hundred URLs, but beyond that the request payload gets too large, and you should use a data connector instead (Spider tracks delivery server-side, so there's nothing to replay on reconnect).

Client-side checkpointing with blacklist

import requests, os, json API = "https://api.spider.cloud/crawl" HEADERS = { "Authorization": f"Bearer {os.getenv('SPIDER_API_KEY')}", "Content-Type": "application/jsonl", } def crawl_with_checkpoint(url: str, limit: int): processed = set() while len(processed) < limit: try: body = { "url": url, "limit": limit - len(processed), "return_format": "markdown", } if processed: body["blacklist"] = list(processed) resp = requests.post(API, headers=HEADERS, json=body, stream=True) for line in resp.iter_lines(): if line: page = json.loads(line) processed.add(page["url"]) yield page except requests.exceptions.ConnectionError: if not processed: raise print(f"Disconnected after {len(processed)}/{limit}, resuming...") continue crawl_with_checkpoint("https://example.com", limit=100)

Tip

For large crawls, skip the blacklist and use a data connectorinstead. Spider delivers pages server-side regardless of your connection, so there's nothing to resume.

Webhook queue pipeline

Enable the on_find webhook and push each page into your existing queue (SQS, Redis Streams, RabbitMQ, etc). Spider discovers pages, your queue buffers them, workers consume at their own pace.

Crawl with on_find webhook

Webhook receiver pushing to SQS

from fastapi import FastAPI, Request import boto3, json app = FastAPI() sqs = boto3.client("sqs") QUEUE_URL = "https://sqs.us-east-1.amazonaws.com/123456789/spider-pages" @app.post("/spider-webhook") async def handle_webhook(request: Request): payload = await request.json() sqs.send_message( QueueUrl=QUEUE_URL, MessageBody=json.dumps(payload), ) return {"ok": True} # return 200 fast

Tip

Return 200 immediately after enqueuing. Heavy inline processing will cause Spider to time out waiting for your response.

Choosing a pattern

These patterns combine freely. A common setup is streaming with a connector for real-time output plus a backup, then adding a webhook queue for async post-processing.

Need	Pattern
Real-time with guaranteed delivery	Stream + Connector
Large batch or cron jobs	Background + Connector + Webhook
Small crawls with disconnect recovery	JSONL Checkpointing
Event-driven architecture	Webhook Queue

Tip

New to these primitives? Start with the Data Connectors, Webhooks, and Streamingguides.