Case study: how a RAG pipeline went from 6 hours to 15 minutes

The problem

An AI research assistant needed fresh documentation from hundreds of technical sites. Its answers were only as good as the docs, knowledge bases, and API references behind them, and those pages change constantly.

The crawler was the bottleneck.

Three Python microservices did the work. One found URLs, one rendered JavaScript pages with Playwright, and one turned HTML into markdown. A proxy rotation provider sat in front of all of it to dodge rate limits and bot detection. Keeping the stack alive took about half of one senior engineer’s week.

What broke

Anti-bot protections changed on a handful of the documentation sites that mattered most. Pages that used to return clean HTML started serving CAPTCHAs or refusing the request outright. Failure rates on those sites hit 30%.

Paying customers got stale search results for two days before anyone noticed. Every patch was written against one site’s specific protection. A week later a different set of sites changed theirs, and the team wrote the patches again.

The proxy provider offered a residential tier at 3x the price. The team went looking for something else instead.

The old architecture

The pipeline ran nightly and a full crawl of every target site took about 6 hours. A failed page got one retry, then waited until the next run. Success rate sat around 92% on a good night and under 80% on a bad one.

The migration

The team replaced all three microservices with Spider’s crawl API over one weekend. The pipeline collapsed into a single function:

from spider import Spider
import os

client = Spider(api_key=os.getenv("SPIDER_API_KEY"))

def crawl_docs(urls: list[str]) -> list[dict]:
    results = []
    for url in urls:
        pages = client.crawl_url(url, params={
            "return_format": "markdown",
            "limit": 200,
            "request": "smart",
            "readability": True,
        })
        results.extend(pages)
    return results

No proxy configuration, no browser management, no extraction logic. Spider does the JavaScript rendering, the bot protection bypass, and the markdown conversion inside the same call.

smart mode is doing most of the work there. Spider decides per page whether a lightweight HTTP fetch is enough or the page needs a full headless render. Static documentation takes the fast path. Client-rendered sites get a browser. The caller never has to know which one ran.

The new architecture

The following week the team cancelled the $2,400/month proxy contract, archived three repos, and terminated the EC2 instances.

Performance

The page count went up because Spider crawled pages the old stack had been quietly skipping after failed retries.

Why it got faster

Spider crawls pages concurrently on its own machines. The old pipeline was capped by how many EC2 instances the team was willing to run and how much memory each Playwright browser took.

It also skips the browser when a page doesn’t need one. The old pipeline pushed everything through Playwright, including the static documentation that makes up most of the target list.

And there is no proxy hop. Rotation added latency to every request. Bot protection now happens inside Spider’s own infrastructure instead.

What it did to RAG quality

Faster crawls with fewer failures kept the vector database current. Before the migration, some chunks were a week stale, because a failed crawl meant a skipped update. Now every target page refreshes daily.

Answer relevance scores, judged with GPT-4, improved 12% after the migration. That came from fresher and more complete source data.

What they would do differently

Turn on readability from the start. The first integration left it off. The raw markdown carried navigation menus, footers, and sidebars into the embeddings, which watered down chunk relevance.

Stream the large crawls. The first version waited for the whole response, so a 200-page site sat in memory in one piece. Switching to application/jsonl cut memory use and let the embedding step start while pages were still coming in.

Try it

If your team is patching scrapers every week, the arithmetic tends to land in the same place. Spider’s API does the rendering, the bot bypass, and the extraction, which leaves your engineers on the part of the product only they can build.

Start with a free account and the quickstart guide.