Webhooks

Receive real-time HTTP POST notifications when events occur during a crawl, including page discovery, metadata extraction, credit usage alerts, and crawl status changes. Eliminates the need for polling.

Available Events

Each event corresponds to a boolean flag on the webhook object. Set the relevant flags to true to subscribe.

Event	Key	Description
Page Found	`on_find`	Fires when a page is discovered. Includes full page content.
Metadata Found	`on_find_metadata`	Fires when metadata is extracted. Returns only the URL, status code, and headers.
Crawl Status	`on_website_status`	Fires on crawl status changes (started, completed, errored).
Credits Depleted	`on_credits_depleted`	Fires when your credit balance hits zero.
Credits Half Depleted	`on_credits_half_depleted`	Fires when your credits drop below 50%.

Inline Webhook (Per-Request)

Pass a webhook object in any crawl, scrape, search, or screenshot request. Spider sends an HTTP POST to your URL whenever the enabled events fire.

Crawl with webhook

import requests, os

headers = {
    "Authorization": f"Bearer {os.getenv('SPIDER_API_KEY')}",
    "Content-Type": "application/json",
}

response = requests.post("https://api.spider.cloud/crawl", headers=headers, json={
    "url": "https://example.com",
    "limit": 50,
    "return_format": "markdown",
    "webhook": {
        "url": "https://your-server.com/spider-webhook",
        "on_find": True,
        "on_website_status": True
    }
})

print(response.json())

Webhook Payload

When an event fires, Spider sends an HTTP POST to your URL with a JSON body. The shape depends on the event. Below is an example on_find payload.

on_find payload

{
  "crawl_id": "d4e5f6a7-b8c9-0d1e-2f3a-4b5c6d7e8f9a",
  "url": "https://example.com/page",
  "status_code": 200,
  "content": "...",
  "domain": "example.com",
  "pathname": "/page"
}

on_website_status payload

{
  "crawl_id": "d4e5f6a7-b8c9-0d1e-2f3a-4b5c6d7e8f9a",
  "url": "https://example.com",
  "mode": "complete",
  "domain": "example.com",
  "links": 42,
  "crawl_duration": 15230
}

Discord Webhooks

Spider automatically detects Discord webhook URLs (matching discord.com/api/webhooks/) and formats the payload as a rich embed with file attachments for the HTML content and screenshot. Provide your Discord webhook URL in the url field.

Discord webhook example

{
  "url": "https://example.com",
  "limit": 5,
  "webhook": {
    "url": "https://discord.com/api/webhooks/123456/abcdef",
    "on_find": true
  }
}

Pro Tip:

All events default to false. You must enable at least one or the webhook will not fire. Webhooks are batched and delivered asynchronously, so your endpoint should return a 200 response quickly to avoid timeouts. Webhooks work on all endpoints: crawl, scrape, search, screenshot, transform, and unblocker. Combine them with data connectors to push results to cloud storage at the same time.