Webhooks

Spider POSTs to your URL as a crawl runs: when it finds a page, extracts metadata, changes status, or your credits run low. No polling.

Events

Each event is a boolean flag on the webhook object. Set it to true to subscribe.

Event	Key	Description
Page found	on_find	Fires when a page is found. Includes the full page content.
Metadata found	on_find_metadata	Fires when metadata is extracted. Carries only the URL, status code and headers.
Crawl status	on_website_status	Fires when the crawl starts, completes or errors.
Credits depleted	on_credits_depleted	Fires when your credit balance hits zero.
Credits half depleted	on_credits_half_depleted	Fires when your credits drop below 50%.

Per-request webhook

Pass a webhook object in any crawl, scrape, search, or screenshot request. Spider POSTs to your URL whenever an enabled event fires.

Crawl with webhook

import requests, os

headers = {
    "Authorization": f"Bearer {os.getenv('SPIDER_API_KEY')}",
    "Content-Type": "application/json",
}

response = requests.post("https://api.spider.cloud/crawl", headers=headers, json={
    "url": "https://example.com",
    "limit": 50,
    "return_format": "markdown",
    "webhook": {
        "url": "https://your-server.com/spider-webhook",
        "on_find": True,
        "on_website_status": True
    }
})

print(response.json())

Payload

Each event POSTs a JSON body whose shape depends on the event. An on_find payload and an on_website_status payload:

on_find payload

{
  "crawl_id": "d4e5f6a7-b8c9-0d1e-2f3a-4b5c6d7e8f9a",
  "url": "https://example.com/page",
  "status_code": 200,
  "content": "...",
  "domain": "example.com",
  "pathname": "/page"
}

on_website_status payload

{
  "crawl_id": "d4e5f6a7-b8c9-0d1e-2f3a-4b5c6d7e8f9a",
  "url": "https://example.com",
  "mode": "complete",
  "domain": "example.com",
  "links": 42,
  "crawl_duration": 15230
}

Verify the signature

Every delivery carries an X-Spider-Signature header of the form t=timestamp,v1=signature, where the signature is an HMAC-SHA256 hex digest keyed with your API key. Rebuild timestamp.body, compute the HMAC with your key as the secret, and compare.

Verify webhook signature

import hmac, hashlib, time, os

def verify_spider_signature(payload: bytes, header: str, tolerance: int = 300) -> bool:
    """Verify an X-Spider-Signature header using your API key."""
    secret = os.getenv("SPIDER_API_KEY")
    parts = dict(p.split("=", 1) for p in header.split(","))
    timestamp = parts["t"]
    expected = parts["v1"]

    if abs(time.time() - int(timestamp)) > tolerance:
        return False  # Reject stale signatures

    signed = f"{timestamp}.{payload.decode()}"
    computed = hmac.new(secret.encode(), signed.encode(), hashlib.sha256).hexdigest()
    return hmac.compare_digest(computed, expected)

Discord

A url under discord.com/api/webhooks/ gets a Discord embed instead of raw JSON, with the HTML and the screenshot attached as files.

Discord webhook example

{
  "url": "https://example.com",
  "limit": 5,
  "webhook": {
    "url": "https://discord.com/api/webhooks/123456/abcdef",
    "on_find": true
  }
}

Tip

Every event defaults to false. Enable at least one or nothing fires. Deliveries are batched and asynchronous, so return 200 fast or the delivery times out. Webhooks work on every endpoint: crawl, scrape, search, screenshot, transform, and unblocker. Add data connectors to push results to storage at the same time.