Skip to main content
NEW AI Studio is now available Try it now

Webhooks

Receive real-time HTTP POST notifications when events occur during a crawl, including page discovery, metadata extraction, credit usage alerts, and crawl status changes. Eliminates the need for polling.

Available Events

Each event corresponds to a boolean flag on the webhook object. Set the relevant flags to true to subscribe.

EventKeyDescription
Page Foundon_findFires when a page is discovered. Includes full page content.
Metadata Foundon_find_metadataFires when metadata is extracted. Returns only the URL, status code, and headers.
Crawl Statuson_website_statusFires on crawl status changes (started, completed, errored).
Credits Depletedon_credits_depletedFires when your credit balance hits zero.
Credits Half Depletedon_credits_half_depletedFires when your credits drop below 50%.

Inline Webhook (Per-Request)

Pass a webhook object in any crawl, scrape, search, or screenshot request. Spider sends an HTTP POST to your URL whenever the enabled events fire.

Crawl with webhook

import requests, os headers = { "Authorization": f"Bearer {os.getenv('SPIDER_API_KEY')}", "Content-Type": "application/json", } response = requests.post("https://api.spider.cloud/crawl", headers=headers, json={ "url": "https://example.com", "limit": 50, "return_format": "markdown", "webhook": { "url": "https://your-server.com/spider-webhook", "on_find": True, "on_website_status": True } }) print(response.json())

Webhook Payload

When an event fires, Spider sends an HTTP POST to your URL with a JSON body. The shape depends on the event. Below is an example on_find payload.

on_find payload

{ "crawl_id": "d4e5f6a7-b8c9-0d1e-2f3a-4b5c6d7e8f9a", "url": "https://example.com/page", "status_code": 200, "content": "...", "domain": "example.com", "pathname": "/page" }

on_website_status payload

{ "crawl_id": "d4e5f6a7-b8c9-0d1e-2f3a-4b5c6d7e8f9a", "url": "https://example.com", "mode": "complete", "domain": "example.com", "links": 42, "crawl_duration": 15230 }

Verifying Signatures

Every webhook includes an X-Spider-Signature header for payload verification. The header format is t=timestamp,v1=signature where the signature is an HMAC-SHA256 hex digest computed using your API key. To verify, reconstruct the signed payload as timestamp.body and compare the HMAC using your Spider API key as the secret.

Verify webhook signature

import hmac, hashlib, time, os def verify_spider_signature(payload: bytes, header: str, tolerance: int = 300) -> bool: """Verify an X-Spider-Signature header using your API key.""" secret = os.getenv("SPIDER_API_KEY") parts = dict(p.split("=", 1) for p in header.split(",")) timestamp = parts["t"] expected = parts["v1"] if abs(time.time() - int(timestamp)) > tolerance: return False # Reject stale signatures signed = f"{timestamp}.{payload.decode()}" computed = hmac.new(secret.encode(), signed.encode(), hashlib.sha256).hexdigest() return hmac.compare_digest(computed, expected)

Discord Webhooks

Spider automatically detects Discord webhook URLs (matching discord.com/api/webhooks/) and formats the payload as a rich embed with file attachments for the HTML content and screenshot. Provide your Discord webhook URL in the url field.

Discord webhook example

{ "url": "https://example.com", "limit": 5, "webhook": { "url": "https://discord.com/api/webhooks/123456/abcdef", "on_find": true } }