Authentication

The Spider API uses Bearer token authentication. Every request must include your API key in the Authorization header.

Getting Your API Key

Sign up or log in to the Spider Dashboard to get your API key. Your key is available on the dashboard home page under your account settings.

Using Your API Key

Include your API key as a Bearer token in the Authorization header of every request. All API requests must be made over HTTPS.

cURL

curl 'https://api.spider.cloud/crawl' \
  -H 'Authorization: Bearer YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{"url": "https://example.com", "limit": 1}'

Python

import requests
import os

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

response = requests.post(
    'https://api.spider.cloud/crawl',
    headers=headers,
    json={"url": "https://example.com", "limit": 1}
)

print(response.json())

Node.js

const response = await fetch('https://api.spider.cloud/crawl', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${process.env.SPIDER_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({ url: 'https://example.com', limit: 1 }),
});

const data = await response.json();
console.log(data);

Using the SDK

Our SDK libraries handle authentication automatically. Set your API key as the SPIDER_API_KEY environment variable and the SDK will use it.

Python SDK

# pip install spider_client
from spider import Spider

# Reads SPIDER_API_KEY from environment automatically
app = Spider()
result = app.crawl_url('https://example.com', params={'limit': 1})
print(result)

Node.js SDK

// npm install @spider-cloud/spider-client
import { Spider } from "@spider-cloud/spider-client";

// Reads SPIDER_API_KEY from environment automatically
const app = new Spider();
const result = await app.crawlUrl("https://example.com", { limit: 1 });
console.log(result);

Pro Tip:

Never hard-code your API key in source code. Use environment variables or a secrets manager to keep your key secure.

Security Best Practices

Follow these guidelines to keep your API key safe.

Use environment variables: Store your key in SPIDER_API_KEY and reference it at runtime.
Never commit keys to version control: Add .env to your .gitignore file.
Rotate keys if compromised: Regenerate your API key from the dashboard immediately if you suspect it has been exposed.
Use separate keys per environment: Use different keys for development, staging, and production.
Restrict access: Only share API keys with team members who need them.

Regenerating Your API Key

You can regenerate your API key at any time from the Spider Dashboard. When you regenerate a key, the old key is immediately invalidated. Update all your applications with the new key before regenerating to avoid service disruption.

Authentication Errors

If your API key is missing, invalid, or expired, the API returns a 401 Unauthorized response. See the Error Codes page for details on all error responses.

401 Unauthorized Response

{
  "error": "Unauthorized",
  "status": 401
}