Libraries

We have a couple of libraries to use to interact with the API.

Python Library

We provide a Python library, which you can install by running:

pip install spider-client

Once installed, you can use the library and your secret key to run the following:

Example request

from spider import Spider
app = Spider()
params = {"limit":50}
crawl_result = app.crawl_url('https://mywebsite.com', params=params)

Javascript Library

We provide a Javascript library, which you can install by running:

npm install @spider-cloud/spider-client

Once installed, you can use the library and your secret key to run the following:

Example request

import { Spider } from "@spider-cloud/spider-client";
const app = new Spider();
const params = {"limit":50};
const result = await app.crawlUrl("https://spider.cloud/scraping", params)

Rust Library

We provide a Rust library, which you can install by running:

cargo add spider-client

Once installed, you can use the library and your secret key to run the following:

Example request

let spider = spider_client::Spider::new(None).expect("API required");
    let crawler_params = RequestParams { limit: Some(50), ..Default::default() };
    let crawl_result = spider.crawl_url("https://spider.cloud/scraping", Some(crawler_params), false, "application/json", None).await.expect("Failed to crawl the URL");

CLI Library

We provide a CLI library, which you can install by running with cargo:

cargo install spider-cloud-cli

Once installed, you can use the library and your secret key to run the following:

spider-cloud-cli auth --api_key YOUR_API_KEY

Now you can run any of the commands:

spider-cloud-cli crawl --url http://example.com --limit 10

Use the following: spider-cloud-cli --help to see all available commands.