RAG Pipelines

Real-Time Web Data for Smarter AI

Retrieval-augmented generation is only as good as your data. Spider keeps your knowledge base current by crawling documentation, websites, and knowledge sources so your AI always has the latest information.

Get Started Read the Guide

$ spider pipeline --rag

Crawl docs.example.com ...342 pages

Clean markdown + metadata done

Chunk semantic splitting 1,847 chunks

Embed vector store indexed

Query ready

> pipeline complete in 4.2s

THE PROBLEM

Why most RAG systems produce stale or inaccurate results.

LLMs hallucinate when they lack current information

Documentation changes frequently and goes stale

Manual data updates don't scale

Building reliable crawlers is a distraction from core product

THE FIX

Spider handles the data pipeline so you can focus on your AI.

Ground your AI in real, current web data

Webhook delivery for real-time updates

Incremental updates: only fetch what changed

Native integrations with LangChain and LlamaIndex

Integration

LangChain Document Loader

Spider is a first-class LangChain document loader. Crawl any site and get back chunked, metadata-rich documents ready for your vector store. Just a few lines of Python.

LangChain LlamaIndex Python Node.js

from langchain_community.document_loaders import SpiderLoader

# Load documents from a website
loader = SpiderLoader(
    url="https://docs.example.com",
    api_key="your-api-key",
    mode="crawl",  # or "scrape" for single page
)

documents = loader.load()

# Documents are ready for your vector store
for doc in documents:
    print(doc.page_content[:100])
    print(doc.metadata["source"])

Capabilities

Built for RAG Workflows

Everything you need to keep your retrieval-augmented generation pipeline fed with fresh, structured web data.

Vector-Ready Output

Clean markdown with proper chunking for embedding models. Optimized for semantic search.

Incremental Crawling

Only fetch pages that changed since your last crawl. Save time and reduce costs.

Batch Processing

Process multiple URLs in a single request. Efficient bulk data collection.

Low Latency

Fast crawling means fresher data. Get results in milliseconds, not minutes.

Source Attribution

Every chunk includes source URL and metadata for proper citations.

Webhook Delivery

Push new content directly to your vector database via webhooks.

How it Works

From Web to Vector Store

Spider fits into your RAG stack wherever you need it. Data flows from the open web through Spider into your embedding pipeline and vector database.

CRAWL

Point Spider at your target sites. It handles JavaScript rendering, anti-bot measures, and pagination automatically.

spider.crawl(url)

CLEAN

Raw HTML is converted to clean markdown with metadata. Navigation, ads, and boilerplate stripped out.

return_format: "markdown"

CHUNK

Content is semantically chunked and optimized for embedding models. Each chunk keeps its source URL.

chunking: "semantic"

DELIVER

Push results to your vector database via webhooks, or pull them through the API. Streaming supported.

webhook: "https://..."

Resources

Go Deeper

GUIDE