Developer Quickstart

The Spider API provides a simple interface for web crawling, web scraping, and web screenshots to support your data-driven applications. Follow this quick start guide to get up and running quickly.

Create and Export an API Key

Create an API key in the dashboard, then store it securely as an environment variable. This key will allow you to access the Spider API securely. Store the key in a safe location, like a .zshrc file or another text file on your computer. Once you’ve generated an API key, export it as an environment variable in your terminal.

Export an envrionment variable on *nix systems

export SPIDER_API_KEY="your_api_key_here"

Make Your First API Request

Once you have your API key, you can start making requests to Spider API. Here are examples for web crawling, web scraping, and capturing web screenshots:

Example request

from spider import Spider
app = Spider()
params = {"limit":50}
crawl_result = app.crawl_url('https://mywebsite.com', params=params)