Error Codes

The Spider API uses standard HTTP status codes to indicate success or failure. Errors include a JSON response body with details. See Authentication for auth-related errors and Rate Limits for throttling details.

JSON Error Response Format

When an error occurs, the API returns a JSON object with the following structure.

Error Response Body

{
  "error": "Rate limit exceeded",
  "status": 429
}

Response Body Status Codes (for pages)

These status codes appear in the response body for each page in the crawl results. They reflect the HTTP status of the target page, not the Spider API request itself.

Status Code	Description	Consumes Credits	Cause	Resolution
2xx	Success	Yes	The target page was fetched successfully.	No action needed.
404	Target page not found	Yes	The URL does not exist on the target website.	Verify the URL is correct and the page exists.
403	Forbidden	Yes	The target website blocked the request.	Try enabling proxy_enabled or using request mode "chrome" for JavaScript rendering.
401	Unauthorized	Yes	The target page requires authentication.	The target site requires login credentials. Consider using automation_scripts to handle login flows.
400	Bad request	Yes	The target server rejected the request.	Check if the URL is properly formatted and accessible.
429	Rate limit exceeded	Yes	The target website is rate limiting requests.	Add a delay parameter between requests or reduce concurrency.
500	Server error	No	The target server encountered an internal error.	Retry the request. If persistent, the target site may be experiencing issues.
503	Service unavailable	No	The target server is temporarily unavailable.	Retry after a short delay. The target site may be under maintenance.

HTTP Status Codes for Spider Request

These status codes are returned as the HTTP response status when you send a request to the Spider API. They indicate whether your API request was accepted and processed.

Status Code	Description	Cause	Resolution
200	Success	Request was processed successfully.	No action needed.
204	No content returned	The crawl completed but no content was found.	Check if the URL returns content. Try a different return_format.
400	Bad request, check request parameters	Invalid request parameters or malformed JSON body.	Review the request body for missing required fields (e.g., url) or invalid parameter values.
401	Authentication error	Missing, invalid, or expired API key.	Check your Authorization header. See the authentication docs for correct format.
402	Payment required	Insufficient credits or no active subscription.	Add credits to your account or upgrade your plan from the dashboard.
413	Payload too large	The request body exceeds the maximum size limit.	Reduce the number of URLs or parameters in a single request.
429	Rate limit exceeded	Too many requests in the current time window.	Implement exponential backoff. Check the rate limits page for your plan's limits.
500	Server error	An unexpected error occurred on Spider's servers.	Retry the request. If the issue persists, contact support.
503	Service unavailable	Spider service is temporarily unavailable.	Retry after a short delay. Check the status page for any ongoing incidents.