Building AI agents that browse the web

Production AI agents browse the web. The hard parts are not the LLM reasoning. They are the infrastructure: fetching pages reliably at the speed the agent needs, keeping token costs under control, and handling the inevitable failures when target sites block you or return garbage.

This post covers three agent patterns that hold up past the demo stage, with implementations in CrewAI and AutoGen. Each pattern uses Spider Cloud as the scraping backend, though the architecture applies regardless of which API you use.

Why agents need a different kind of scraper

A traditional scraper fetches one page, parses it, and stores the result. An agent works differently. It decides which pages to fetch based on intermediate reasoning, issues requests in tight loops, and feeds the content straight into an LLM. That changes the requirements.

Latency matters more than you think. An agent making 40 requests per task cannot afford 30 seconds per page. At that rate, a single research question takes 20 minutes of wall-clock time before the LLM even starts reasoning. Spider’s HTTP mode returns cached pages in roughly 10-15ms. Uncached pages requiring Chrome rendering take 1-5 seconds each. With batching (sending all 40 URLs in one request), the scraping step drops from minutes to seconds. Your scraper should not be the bottleneck in the agent loop.

Token cost is a function of content quality. Raw HTML from a typical page runs 10,000 to 80,000 tokens depending on page complexity. Clean markdown of the same page runs 1,000 to 5,000 tokens. When every page goes through an LLM, the difference is often 10x or more in token cost. Spider strips navigation, ads, footers, and boilerplate and returns markdown you can put in a context window without post-processing.

Batch support reduces round trips. Spider accepts multiple URLs in a single API call. Instead of issuing 20 sequential HTTP requests, an agent can send all 20 URLs at once and get results back in a single response. This matters for agents that collect a list of URLs from a search step and then need to scrape all of them.

Structured extraction without selectors. Spider’s prompt-based extraction lets you describe what you want in natural language and get JSON back. For data extraction agents, this removes brittle parsing logic. The trade-off is that prompt-based extraction is slower and less deterministic than a well-maintained CSS selector. For sites you control or scrape frequently, selectors are still the better choice.

Pattern 1: Research agent

A research agent takes a question, searches the web, crawls relevant pages, synthesizes the content, and produces an answer. The hard part is deciding when to stop searching and start answering.

Architecture

User question
    |
    v
[Query planner] --> generates search queries
    |
    v
[Spider search] --> returns ranked URLs
    |
    v
[Spider scrape]  --> fetches pages as markdown (batch)
    |
    v
[Evaluator]     --> checks if content answers the question
    |                    |
    | (insufficient)     | (sufficient)
    v                    v
[Refine query]      [Synthesizer] --> final answer

The agent loops between searching, scraping, and evaluating until it has enough material. A hard cap on iterations prevents runaway costs.

CrewAI implementation

import os
from spider import Spider
from crewai import Agent, Task, Crew, Process
from crewai_tools import tool

spider = Spider(api_key=os.getenv("SPIDER_API_KEY"))

@tool("web_search")
def web_search(query: str) -> str:
    """Search the web and return titles, URLs, and descriptions."""
    results = spider.search(query, params={"limit": 10, "fetch_page_content": False})
    entries = results.get("content", results) if isinstance(results, dict) else results
    output = []
    for r in entries:
        title = r.get("title", "")
        url = r.get("url", "")
        desc = r.get("description", "")
        output.append(f"- [{title}]({url}): {desc}")
    return "\n".join(output)

@tool("scrape_urls")
def scrape_urls(urls: str) -> str:
    """Scrape one or more URLs (comma-separated) and return their content as markdown."""
    url_list = [u.strip() for u in urls.split(",")]
    results = []
    for url in url_list[:10]:  # cap at 10 pages per call
        data = spider.scrape_url(url, params={"return_format": "markdown"})
        if data and len(data) > 0:
            content = data[0].get("content", "")
            results.append(f"## {url}\n\n{content[:8000]}")
    return "\n\n---\n\n".join(results)

researcher = Agent(
    role="Senior Research Analyst",
    goal="Find accurate, comprehensive information to answer: {question}",
    backstory=(
        "You are a meticulous researcher. You search the web, read sources, "
        "and cross-reference claims before drawing conclusions. You never "
        "fabricate information. If sources conflict, you note the disagreement."
    ),
    tools=[web_search, scrape_urls],
    verbose=True,
    memory=True,
    allow_delegation=False,
)

synthesizer = Agent(
    role="Technical Writer",
    goal="Write a clear, well-sourced answer to: {question}",
    backstory=(
        "You turn raw research into polished answers. You cite sources, "
        "highlight key findings, and flag uncertainties. You never add "
        "information beyond what the research provides."
    ),
    verbose=True,
    memory=True,
    allow_delegation=False,
)

research_task = Task(
    description=(
        "Research the following question thoroughly: {question}\n\n"
        "1. Search the web for relevant sources.\n"
        "2. Scrape the top 5-8 most relevant pages.\n"
        "3. Extract key facts, data points, and expert opinions.\n"
        "4. If initial results are insufficient, refine your query and search again.\n"
        "5. Compile your findings with source URLs."
    ),
    expected_output="Detailed research notes with source URLs and key findings.",
    tools=[web_search, scrape_urls],
    agent=researcher,
)

synthesis_task = Task(
    description=(
        "Using the research notes provided, write a comprehensive answer "
        "to the question: {question}\n\n"
        "Include source citations. Flag any conflicting information."
    ),
    expected_output="A well-structured answer with citations, 4-8 paragraphs.",
    agent=synthesizer,
)

crew = Crew(
    agents=[researcher, synthesizer],
    tasks=[research_task, synthesis_task],
    process=Process.sequential,
    memory=True,
    cache=True,
    max_rpm=100,
)

result = crew.kickoff(inputs={
    "question": "What are the current best practices for fine-tuning LLMs on domain-specific data?"
})
print(result)

The researcher scrapes pages in batches and caps content at 8,000 characters per page to stay within context limits. The synthesizer only works with material the researcher collected, which reduces hallucination.

AutoGen implementation

import os
from spider import Spider
from autogen import ConversableAgent, register_function
from typing import List, Dict, Any
from typing_extensions import Annotated

config_list = [
    {"model": "gpt-4o", "api_key": os.getenv("OPENAI_API_KEY")}
]

spider = Spider(api_key=os.getenv("SPIDER_API_KEY"))

def search_web(
    query: Annotated[str, "The search query"]
) -> Annotated[str, "Search results with titles and URLs"]:
    """Search the web for information on a topic."""
    results = spider.search(query, params={"limit": 8, "fetch_page_content": False})
    entries = results.get("content", results) if isinstance(results, dict) else results
    output = []
    for r in entries:
        output.append(f"- {r.get('title', '')}: {r.get('url', '')}")
    return "\n".join(output)

def scrape_pages(
    urls: Annotated[str, "Comma-separated URLs to scrape"]
) -> Annotated[str, "Markdown content from scraped pages"]:
    """Scrape web pages and return their content as markdown."""
    url_list = [u.strip() for u in urls.split(",")]
    results = []
    for url in url_list[:8]:
        data = spider.scrape_url(url, params={"return_format": "markdown"})
        if data and len(data) > 0:
            content = data[0].get("content", "")[:6000]
            results.append(f"## Source: {url}\n{content}")
    return "\n\n".join(results)

research_agent = ConversableAgent(
    "Researcher",
    llm_config={"config_list": config_list},
    system_message=(
        "You are a research agent. Search the web, scrape relevant pages, "
        "and compile findings. When you have enough information to answer "
        "the question thoroughly, write your final answer and end with TERMINATE."
    ),
)

user_proxy = ConversableAgent(
    "UserProxy",
    llm_config=False,
    human_input_mode="NEVER",
    code_execution_config=False,
    is_termination_msg=lambda x: x.get("content", "") is not None
        and "terminate" in x.get("content", "").lower(),
    default_auto_reply="Continue researching if not done. Otherwise TERMINATE.",
)

register_function(search_web, caller=research_agent, executor=user_proxy,
    name="search_web", description="Search the web for a topic.")
register_function(scrape_pages, caller=research_agent, executor=user_proxy,
    name="scrape_pages", description="Scrape URLs and return markdown content.")

result = user_proxy.initiate_chat(
    research_agent,
    message="Research: What are the performance differences between PostgreSQL and CockroachDB for OLTP workloads?",
    summary_method="reflection_with_llm",
    max_turns=10,
)
print(result.summary)

The max_turns parameter is the cost ceiling. The agent can search and scrape multiple times within that budget, but it cannot loop indefinitely.

Pattern 2: Monitoring agent

A monitoring agent watches a set of URLs for changes, compares new content against a baseline, and triggers alerts when something meaningful shifts. This is useful for tracking competitor pricing, regulatory updates, documentation changes, or news on a specific topic.

Architecture

Scheduled trigger (cron / interval)
    |
    v
[Spider scrape] --> fetches current page content
    |
    v
[Diff engine]   --> compares against stored baseline
    |
    v
[LLM classifier] --> determines if change is meaningful
    |                    |
    | (noise)            | (meaningful)
    v                    v
[Skip]              [Alert + update baseline]

The LLM classifier is what separates this from a simple diff. Copyright year changes, minor formatting tweaks, and rotated ad blocks are noise. New product launches, price changes, and policy updates are meaningful. The LLM makes that judgment call.

Implementation

import os
import json
import hashlib
import difflib
from datetime import datetime
from spider import Spider
from openai import OpenAI

spider = Spider(api_key=os.getenv("SPIDER_API_KEY"))
openai_client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
BASELINE_FILE = "baselines.json"

def load_baselines() -> dict:
    if os.path.exists(BASELINE_FILE):
        with open(BASELINE_FILE, "r") as f:
            return json.load(f)
    return {}

def save_baselines(baselines: dict):
    with open(BASELINE_FILE, "w") as f:
        json.dump(baselines, f, indent=2)

def content_hash(text: str) -> str:
    return hashlib.sha256(text.encode()).hexdigest()

def get_diff(old: str, new: str) -> str:
    old_lines = old.splitlines(keepends=True)
    new_lines = new.splitlines(keepends=True)
    diff = difflib.unified_diff(old_lines, new_lines, lineterm="")
    return "".join(list(diff)[:200])  # cap diff size for LLM context

def classify_change(url: str, diff_text: str, context: str) -> dict:
    response = openai_client.chat.completions.create(
        model="gpt-4o",
        response_format={"type": "json_object"},
        messages=[
            {
                "role": "system",
                "content": (
                    "You classify website changes. Respond with JSON: "
                    '{"meaningful": true/false, "category": "...", "summary": "..."}\n'
                    "Categories: pricing, product, policy, content, cosmetic, error.\n"
                    "Cosmetic changes (formatting, copyright year, ad rotation) are not meaningful."
                ),
            },
            {
                "role": "user",
                "content": f"URL: {url}\n\nDiff:\n{diff_text}\n\nContext:\n{context[:3000]}",
            },
        ],
    )
    return json.loads(response.choices[0].message.content)

def send_alert(url: str, classification: dict):
    # Replace with your preferred notification method:
    # Slack webhook, email, PagerDuty, database insert, etc.
    print(f"ALERT [{classification['category'].upper()}] {url}")
    print(f"  Summary: {classification['summary']}")

def monitor(urls: list[str]):
    baselines = load_baselines()

    for url in urls:
        data = spider.scrape_url(url, params={"return_format": "markdown"})
        if not data or len(data) == 0:
            print(f"  Failed to fetch {url}, skipping.")
            continue

        current_content = data[0].get("content", "")
        current_hash = content_hash(current_content)

        if url not in baselines:
            baselines[url] = {
                "hash": current_hash,
                "content": current_content,
                "last_checked": datetime.now().isoformat(),
            }
            print(f"  Baseline set for {url}")
            continue

        if current_hash == baselines[url]["hash"]:
            baselines[url]["last_checked"] = datetime.now().isoformat()
            continue

        # Content changed. Classify the change.
        diff_text = get_diff(baselines[url]["content"], current_content)
        classification = classify_change(url, diff_text, current_content)

        if classification.get("meaningful", False):
            send_alert(url, classification)

        # Update baseline regardless of whether the change was meaningful.
        baselines[url] = {
            "hash": current_hash,
            "content": current_content,
            "last_checked": datetime.now().isoformat(),
            "last_change": classification,
        }

    save_baselines(baselines)

# Monitor these URLs on a schedule (cron, Celery beat, etc.)
watch_list = [
    "https://openai.com/pricing",
    "https://docs.anthropic.com/en/docs/about-claude/models",
    "https://cloud.google.com/vertex-ai/generative-ai/pricing",
]

monitor(watch_list)

Run this on a schedule (every hour, every 15 minutes, whatever fits the use case). Set "cache": False when you need guaranteed fresh content. Leave caching on to cut costs when near-real-time freshness is acceptable.

Scaling the watch list

The naive implementation above scrapes URLs sequentially. For larger watch lists (hundreds or thousands of URLs), use Spider’s batch endpoint and Python’s concurrent.futures to parallelize.

from concurrent.futures import ThreadPoolExecutor, as_completed

def monitor_batch(urls: list[str], max_workers: int = 10):
    baselines = load_baselines()

    def check_url(url):
        data = spider.scrape_url(url, params={
            "return_format": "markdown",
            "cache": False,
        })
        if not data or len(data) == 0:
            return url, None
        return url, data[0].get("content", "")

    with ThreadPoolExecutor(max_workers=max_workers) as executor:
        futures = {executor.submit(check_url, url): url for url in urls}
        for future in as_completed(futures):
            url, content = future.result()
            if content is None:
                continue
            # Same diff/classify/alert logic as above
            # ...

    save_baselines(baselines)

With batching you can monitor thousands of pages in a single API call and keep each monitoring cycle under a minute of wall-clock time.

Pattern 3: Data extraction agent

A data extraction agent visits pages and pulls structured data into a predefined schema. Unlike traditional scraping with CSS selectors or XPath, the agent uses natural language prompts to describe what it wants, and Spider’s extraction layer returns JSON.

This pattern fits product catalogs, job listings, event schedules, contact directories, or any semi-structured content whose layout varies from site to site.

Architecture

Input: list of URLs + target schema
    |
    v
[Spider scrape with extraction prompt]
    |
    v
[Schema validator] --> checks extracted JSON against schema
    |                    |
    | (valid)            | (invalid / partial)
    v                    v
[Store result]       [Retry with refined prompt or fallback scrape]

Implementation with Spider’s extraction

Spider supports prompt-based extraction natively. You send a natural language prompt describing the data you want, and Spider returns structured JSON, with no selectors or parsing code on your side.

import os
import json
from spider import Spider

spider = Spider(api_key=os.getenv("SPIDER_API_KEY"))

def extract_structured_data(urls: list[str], extraction_prompt: str) -> list[dict]:
    """
    Extract structured data from a list of URLs using Spider's
    prompt-based extraction.
    """
    results = []

    for url in urls:
        data = spider.scrape_url(url, params={
            "return_format": "markdown",
            "extra_ai_data": True,
            "prompt": extraction_prompt,
        })

        if data and len(data) > 0:
            page = data[0]
            extracted = page.get("extra_ai_data", page.get("content", ""))
            results.append({
                "url": url,
                "data": extracted,
            })

    return results

# Example: extract product data from e-commerce pages
product_prompt = (
    "Extract product information as JSON with these fields: "
    "name (string), price (number), currency (string), "
    "description (string, max 200 chars), in_stock (boolean), "
    "rating (number, 0-5), review_count (number). "
    "If a field is not found, use null."
)

urls = [
    "https://example.com/product/widget-pro",
    "https://example.com/product/widget-lite",
    "https://example.com/product/widget-max",
]

products = extract_structured_data(urls, product_prompt)
for p in products:
    print(json.dumps(p, indent=2))

CrewAI data extraction agent

For more complex extraction tasks where the agent needs to discover URLs, navigate pagination, or handle multiple page types, CrewAI provides the orchestration layer.

import os
import json
from spider import Spider
from crewai import Agent, Task, Crew, Process
from crewai_tools import tool

spider = Spider(api_key=os.getenv("SPIDER_API_KEY"))

@tool("crawl_site")
def crawl_site(url: str) -> str:
    """Crawl a website and return a list of discovered page URLs."""
    data = spider.crawl_url(url, params={
        "return_format": "markdown",
        "limit": 50,
    })
    if not data:
        return "No pages found."
    urls = [page.get("url", "") for page in data if page.get("url")]
    return "\n".join(urls)

@tool("extract_data")
def extract_data(url_and_prompt: str) -> str:
    """Extract structured data from a URL.
    Input format: URL ||| extraction prompt"""
    parts = url_and_prompt.split("|||")
    if len(parts) != 2:
        return "Error: provide input as 'URL ||| extraction prompt'"

    url = parts[0].strip()
    prompt = parts[1].strip()

    data = spider.scrape_url(url, params={
        "return_format": "markdown",
        "extra_ai_data": True,
        "prompt": prompt,
    })

    if data and len(data) > 0:
        extracted = data[0].get("extra_ai_data", data[0].get("content", ""))
        return json.dumps({"url": url, "data": extracted}, indent=2)
    return json.dumps({"url": url, "error": "extraction failed"})

extractor = Agent(
    role="Data Extraction Specialist",
    goal="Extract structured {data_type} data from {target_site}",
    backstory=(
        "You are an expert at extracting structured data from websites. "
        "You first crawl the site to discover relevant pages, then extract "
        "data from each page using specific prompts. You validate that "
        "extracted data matches the expected schema before including it."
    ),
    tools=[crawl_site, extract_data],
    verbose=True,
    memory=True,
)

extraction_task = Task(
    description=(
        "Extract {data_type} data from {target_site}.\n\n"
        "Steps:\n"
        "1. Crawl the site to discover relevant pages.\n"
        "2. Identify pages that contain {data_type} information.\n"
        "3. Extract data from each page using this prompt: {extraction_prompt}\n"
        "4. Compile all extracted data into a single JSON array.\n"
        "5. Remove duplicates and validate completeness."
    ),
    expected_output="A JSON array of extracted {data_type} records.",
    agent=extractor,
)

crew = Crew(
    agents=[extractor],
    tasks=[extraction_task],
    process=Process.sequential,
    max_rpm=100,
)

result = crew.kickoff(inputs={
    "target_site": "https://example.com/jobs",
    "data_type": "job listing",
    "extraction_prompt": (
        "Extract job listing as JSON: title (string), company (string), "
        "location (string), salary_range (string or null), "
        "posted_date (string), requirements (list of strings)."
    ),
})
print(result)

The agent discovers pages through crawling, then applies extraction prompts to each relevant page. Spider handles both the crawling and the structured extraction, so the agent logic stays focused on orchestration.

Common failure modes

Building agents that work in demos is straightforward. Building agents that work at 3 AM on a Saturday without supervision is harder. These are the failure modes that show up in production.

Rate limiting and throttling

Most scraping APIs impose rate limits that agents hit fast. Most providers will throttle an agent that fans out to 50 URLs in a tight loop, and then you get retries, backoff delays, and task completion times you cannot predict.

Spider’s rate limit is high enough that most agent workloads never hit it. If you do approach the limit, the API returns standard rate limit headers (X-RateLimit-Remaining, Retry-After) so your agent can back off instead of blindly retrying.

Stale data from caching

Caching helps with cost but can cause problems for monitoring agents. If your agent checks a pricing page and gets a cached version from 6 hours ago, the price change you are watching for will be invisible.

Use "cache": False in Spider requests when freshness matters. For research agents where slight staleness is acceptable, leave caching enabled to reduce costs and improve response times.

Hallucination from bad scrapes

When a scrape fails silently (returns a login wall, a CAPTCHA page, or a cookie consent overlay instead of the actual content), the agent feeds garbage into the LLM. The LLM then hallucinates an answer based on whatever fragments it received. The user sees a confident, well-formatted response built on nothing.

Defend against this at two levels. First, check the scraped content before passing it to the LLM. A page that contains fewer than 100 characters, or that contains strings like “please enable JavaScript” or “access denied,” is probably not real content. Second, use Spider’s smart mode, which detects when a page needs JavaScript rendering and switches to headless Chrome.

def validate_content(content: str) -> bool:
    """Check if scraped content looks like real page content."""
    if not content or len(content.strip()) < 100:
        return False
    noise_signals = [
        "please enable javascript",
        "access denied",
        "captcha",
        "cookies must be enabled",
        "403 forbidden",
    ]
    content_lower = content.lower()
    return not any(signal in content_lower for signal in noise_signals)

Cost runaway

An agent with a search-scrape-evaluate loop and no iteration cap can burn through API credits fast. Each loop iteration costs LLM tokens (for evaluation) plus scraping credits (for new pages). Without guardrails, a vague question can trigger dozens of iterations.

Set hard limits at multiple levels:

# Cap iterations in the agent loop
MAX_ITERATIONS = 5

# Cap pages per scrape call
MAX_PAGES_PER_REQUEST = 10

# Cap total content length sent to LLM
MAX_CONTEXT_CHARS = 50000

# Use Spider's cost controls
params = {
    "return_format": "markdown",
    "max_credits_per_page": 10,   # cap per-page cost
    "max_credits_allowed": 500,   # cap total cost per request
}

With these limits in place the worst case is a number you chose, not an open-ended bill.

Brittle selectors when layouts change

Traditional scrapers break when a site changes its HTML structure. CSS selectors and XPath expressions are coupled to specific DOM layouts. When a site redesigns, you update every selector.

Spider’s prompt-based extraction sidesteps this. Instead of div.price > span.amount, you write “extract the product price as a number.” The extraction layer understands the page semantically, so layout changes rarely break it. That matters most for extraction agents that target many sites with different page structures.

The path to fully autonomous browsing

The patterns above still rely on external LLMs (OpenAI, Anthropic, etc.) for reasoning. Every LLM call adds latency, cost, and a dependency on a third-party API.

Spider already runs its own extraction models that handle most structured HTML-to-JSON conversion without calling an external LLM. For agent builders, that means the extraction step adds negligible latency and zero per-token cost. Scraping and extraction happen in one call.

Getting started

The code above is a starting point. In production, you will need to add: retry logic for transient API failures, token counting to prevent context overflow, cost tracking per agent run, and evaluation harnesses to measure whether your agent actually answers questions correctly. Start with the research agent pattern, as it is the most general and the easiest to evaluate.

One thing the examples above do not address: how do you know the agent is producing good answers? Agent evaluation is the hardest unsolved problem in this space. At minimum, build a test set of 20-30 questions with known answers and run your agent against it after every change. Without that, you are shipping blind.

A note on framework versions: AutoGen is in the middle of an API rework (the AG2 fork and autogen-agentchat 0.4+). Pin your dependency version and check the migration guide before upgrading. CrewAI’s tool-calling behavior can be unpredictable with complex tool schemas, so test it with your real tools before trusting it in production.