LlamaIndex integration

Use Spider as a LlamaIndex web reader. Crawled pages load straight into an index. Python only.

Install

Install LlamaIndex and the Spider client.

Install the Spider client

pip install spider_client

Put your API key in SPIDER_API_KEY.

Usage

SpiderWebReader turns web pages into LlamaIndex documents.

Scrape a single page

from llama_index.readers.web import SpiderWebReader

spider_reader = SpiderWebReader(
    api_key="YOUR_API_KEY",
    mode="scrape",
    # params={} # Optional parameters
)

documents = spider_reader.load_data(url="https://spider.cloud")
print(documents)

Crawl multiple pages

from llama_index.readers.web import SpiderWebReader

spider_reader = SpiderWebReader(
    api_key="YOUR_API_KEY",
    mode="crawl",
    # params={} # Optional parameters
)

documents = spider_reader.load_data(url="https://spider.cloud")
print(documents)

Pass extra request parameters in the params dictionary. The API reference lists every option.