Web search API for AI agents: search and extract in one call

I’ve talked to hundreds of teams building AI agents over the past year. The ones that ship something that works in production all hit the same bottleneck early: the agent needs to know things that happened after the model was trained.

The training cutoff is a hard limit. Your agent can reason, plan, and write good code, but ask it about last week and it either hallucinates or admits it doesn’t know. Neither is acceptable if people depend on the answer.

The obvious fix is web search. Let the agent Google things. What is less obvious is how painful that is to implement well.

Three services duct-taped together

Most teams end up with a stack that looks something like this:

# Step 1: Hit a SERP API
serp_results = serp_client.search("latest AI regulations 2026")
urls = [r["link"] for r in serp_results["organic_results"]]

# Step 2: Scrape each result
pages = []
for url in urls:
    try:
        html = scraper.fetch(url, render_js=True)
        text = html_to_markdown(html)
        pages.append({"url": url, "content": text})
    except Exception:
        continue  # hope for the best

# Step 3: Feed to LLM
context = "\n\n".join([p["content"] for p in pages])
response = llm.chat(f"Based on this context:\n{context}\n\nAnswer: ...")

That is three API keys, three billing models, and three sets of error handling, plus an N+1 request pattern where every scrape waits on the search because you don’t have the URLs until it finishes.

It works in a demo notebook. In production, the cracks show up fast.

The SERP call takes 1-2 seconds. Each scrape takes 2-5 seconds. Five results means your user is staring at a loading spinner for 12-27 seconds before the LLM even begins generating. By then they’ve already switched tabs.

Then there’s the failure cascade. The SERP API returns URLs. One of them is behind Cloudflare. Another uses heavy client-side rendering your basic scraper can’t handle. A third returns a 403 because you’re hammering it from the same IP as every other developer using the same scraping service. Your agent gets back three results instead of five, and two of those are half-parsed nav menus mixed with article text.

You end up writing more glue code than agent logic. Retry wrappers, HTML cleaning pipelines, fallback scraping strategies, timeout handling. All of it just to answer the question “what happened today?”

One call instead of six

We built Spider’s Search API to kill this problem. One POST request does everything: searches the web, crawls every result, handles rendering and anti-bot, cleans the content, and returns structured data.

from spider import Spider

client = Spider()

results = client.search(
    "latest AI regulations 2026",
    params={
        "search_limit": 5,
        "return_format": "markdown",
        "fetch_page_content": True,
    }
)

That’s the entire search-and-scrape pipeline. fetch_page_content is the parameter that matters. It tells Spider to visit each result URL and extract the content instead of returning titles and snippets.

The response comes back with clean markdown from each page, ready to drop into an LLM context window:

[
  {
    "url": "https://example.com/ai-regulation-tracker",
    "status": 200,
    "content": "# AI Regulation Tracker\n\nThe EU AI Act entered...",
    "costs": {
      "total_cost": 0.00036
    }
  }
]

Proxy rotation, JavaScript rendering, CAPTCHA solving, and HTML-to-markdown conversion all happen server-side. You don’t manage any of it.

What this looks like in a real agent

This is the pattern we see working best. It is simple on purpose: the fewer moving parts between the user’s question and the LLM’s answer, the more reliable the system.

from spider import Spider
from openai import OpenAI
import os

spider = Spider(api_key=os.getenv("SPIDER_API_KEY"))
openai_client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

def answer_with_web(question: str) -> str:
    # Search and scrape in one call
    results = spider.search(
        question,
        params={
            "search_limit": 5,
            "return_format": "markdown",
            "fetch_page_content": True,
            "readability": True,
        }
    )

    # Build context from results
    context_parts = []
    for r in results:
        if r.get("content"):
            content = r["content"][:3000]
            context_parts.append(f"Source: {r['url']}\n{content}")

    context = "\n\n---\n\n".join(context_parts)

    # Generate a grounded answer
    response = openai_client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "system",
                "content": "Answer using only the provided web sources. Cite sources by URL."
            },
            {
                "role": "user",
                "content": f"## Web Sources\n\n{context}\n\n## Question\n\n{question}"
            }
        ]
    )

    return response.choices[0].message.content

Total latency: 3-5 seconds. The Spider search call takes 1-2 seconds (it searches and scrapes all results in parallel internally), and the LLM generates the answer in another 1-3 seconds. That’s a grounded, cited response faster than most people can read the question.

The readability flag strips navigation, sidebars, footers, and cookie banners from the scraped content. Without it you burn context window tokens on boilerplate that confuses the model.

Features that matter for agents specifically

A few capabilities that get less attention than they deserve and change how you design the agent:

Batch queries. Most search APIs process one query at a time. Spider accepts an array. If your agent needs to research three topics to answer one question, you send all three as a batch and Spider runs them in parallel. One HTTP call, three searches, all results back together.

queries = [
    {"search": "company X earnings Q1 2026", "search_limit": 3, "fetch_page_content": True, "return_format": "markdown"},
    {"search": "company X competitor landscape", "search_limit": 3, "fetch_page_content": True, "return_format": "markdown"},
    {"search": "company X SEC filings recent", "search_limit": 3, "fetch_page_content": True, "return_format": "markdown"}
]

response = requests.post("https://api.spider.cloud/search", headers=headers, json=queries)

That’s nine pages searched and scraped in a single round trip. Try doing that with three sequential SERP API calls plus twenty-seven individual scrape requests.

Time filters. When someone asks “what happened today,” you don’t want results from six months ago cluttering the context. The tbs parameter restricts results to the past hour (qdr:h), day (qdr:d), week (qdr:w), month (qdr:m), or year (qdr:y). This is the difference between a useful answer and a hallucinated one.

City-level geo-targeting. Pass location: "San Francisco, CA" along with country and language, and search results reflect what a user in that location would see. This matters for any agent handling local queries. “Best restaurants nearby” means different things in Tokyo and Toronto.

Auto-pagination. Set auto_pagination: true with a high search_limit, and Spider pages through the search results for you. You don’t increment pages by hand. Useful when your agent needs deep research across 30 or 50 results instead of the usual 5.

10,000 requests per minute. If you’re building an agent that serves real traffic, rate limits matter. Firecrawl caps concurrency at 2-150 depending on your plan. Jina’s free tier is 500 RPM. Spider handles 10K. You don’t want your agent failing because twenty users asked questions at the same time.

How this compares

What each search API gives you:

Firecrawl is the closest feature match. The main practical differences are batch queries, the rate limit ceiling, and cost. Jina is good for a quick single-page read but doesn’t combine search and scrape. SerpAPI returns search results and leaves fetching the content to you.

Streaming for faster time-to-first-token

If you’re streaming the LLM response to the user, you don’t need all search results before you start. Use JSONL content type and Spider streams each result back as soon as it’s scraped:

curl -X POST https://api.spider.cloud/search \
  -H "Authorization: Bearer $SPIDER_API_KEY" \
  -H "Content-Type: application/jsonl" \
  -d '{"search": "web scraping best practices", "search_limit": 10, "fetch_page_content": true, "return_format": "markdown"}'

You can start feeding the first two or three results to the LLM while results four through ten are still being fetched, which cuts the wait the user sees.

When to use Search vs. Crawl vs. Scrape

Spider has three main endpoints:

Search is for when you don’t have URLs. You have a question or topic and need to discover relevant pages on the open web. This is the default for AI agents answering user questions.

Crawl is for when you have a starting URL and want to go deep. Follow links, index an entire site, build a knowledge base from documentation.

Scrape is for when you have specific URLs and want their content. Monitoring known pages, refreshing previously crawled data.

Most AI agents start every interaction with Search, then use Crawl or Scrape for follow-up tasks once they’ve identified the right sources.

Start searching

from spider import Spider

client = Spider()
results = client.search("your query", params={"search_limit": 5, "fetch_page_content": True, "return_format": "markdown"})

for r in results:
    print(r["url"], len(r.get("content", "")), "chars")

There’s no subscription. You pay per request and credits never expire. The average search + scrape costs less than $0.003 per query.

Create an account | Search API docs | Full search guide