Libraries

Set up your development environment to use the Spider API with an SDK in your preferred language.

Python Library

To use the Spider API in Python, you can use the official Spider SDK for Python. Get started by installing the SDK using pip:

pip install spider_client

Crawl website with Python

from spider import Spider
app = Spider()
params = {"limit":5}
crawl_result = app.crawl_url('https://example.com', params=params)

Javascript Library

Use the official TypeScript and JavaScript Spider SDK to integrate the Spider API into server-side JavaScript environments such as Node.js, Deno, or Bun. Get started by installing the SDK using npm or your preferred package manager:

npm install @spider-cloud/spider-client

Crawl website with Javascript

import { Spider } from "@spider-cloud/spider-client";
const app = new Spider();
const params = {"limit":5};
const result = await app.crawlUrl("https://example.com", params)

Rust Library

To use the Spider API in Rust, you can use the official Spider SDK for Rust. Get started by installing the SDK using cargo:

cargo add spider-client

Crawl website with Rust

let spider = spider_client::Spider::new(None).expect("API required");
let crawler_params = RequestParams { limit: Some(5), ..Default::default() };
let crawl_result = spider.crawl_url("https://example.com", Some(crawler_params), false, "application/json", None).await.expect("Failed to crawl the URL");

CLI Library

Get the CLI library, which you can install by running with cargo:

cargo install spider-cloud-cli

spider-cloud-cli auth --api_key YOUR_API_KEY

Now after authentication run any of the commands:

spider-cloud-cli crawl --url http://example.com --limit 5

Use spider-cloud-cli --help in terminal to see all available commands.