Give a Microsoft Agent Framework agent the web

Microsoft Agent Framework is the successor to AutoGen and Semantic Kernel. It runs on Python, .NET and Go, and both migration guides live in the repo. If you are moving an AutoGen agent over, the AutoGen guide still describes the older setup.

An agent there has no way to read a page until you give it one. Spider fills that in through either route below. The MCP server takes a minute and needs no Spider SDK. The @tool function takes more code and gives you control over what comes back.

Before you start

Install the framework and get a key from the API keys page:

pip install agent-framework --pre
export SPIDER_API_KEY=<your-api-key-here>

Route 1: the hosted MCP server

Agent Framework speaks MCP, so it can load Spider’s tools over HTTP with nothing installed. MCPStreamableHTTPTool connects on async with, reads the server’s tool list and hands the whole thing to the model.

import asyncio
import os

from agent_framework import Agent, MCPStreamableHTTPTool
from agent_framework.openai import OpenAIChatClient


async def main() -> None:
    spider = MCPStreamableHTTPTool(
        name="spider",
        url="https://mcp.spider.cloud/mcp",
        static_headers={"Authorization": f"Bearer {os.environ['SPIDER_API_KEY']}"},
        allowed_tools=["spider_scrape", "spider_search"],
    )

    async with spider:
        agent = Agent(
            client=OpenAIChatClient(),
            instructions="Answer from pages you fetch. Quote the source URL.",
            tools=spider,
        )
        result = await agent.run("What does spider.cloud charge per credit?")
        print(result.text)


asyncio.run(main())

The server carries 22 tools: crawl, scrape, search, links, screenshots, transforms, the unblocker, the AI variants and nine browser controls. That is a lot of schema for a model to read on every turn, so allowed_tools narrows it to the ones your agent needs. Drop the argument to load them all. The full list is on the MCP integration page.

Route 2: a @tool function

Write the tool yourself when you want to fix the return format, trim the payload before it reaches the model, or bill the call against your own limits.

pip install spider-client

@tool reads the signature and builds a Pydantic model from it, so the parameter descriptions come from Annotated and the tool description comes from the docstring.

import asyncio
import os
from typing import Annotated

from agent_framework import Agent, tool
from agent_framework.openai import OpenAIChatClient
from spider import Spider

client = Spider(os.environ["SPIDER_API_KEY"])


@tool
def scrape(
    url: Annotated[str, "The page to fetch."],
) -> str:
    """Fetch one page through Spider and return it as markdown."""
    pages = client.scrape_url(url, {"return_format": "markdown"})
    return pages[0]["content"]


async def main() -> None:
    agent = Agent(
        client=OpenAIChatClient(),
        instructions="Answer from pages you fetch. Quote the source URL.",
        tools=[scrape],
    )
    result = await agent.run("Summarize https://spider.cloud/docs/overview/")
    print(result.text)


asyncio.run(main())

ai_function is an alias for @tool that older samples still use. Both give you the same FunctionTool.

A crawl tool is the same shape with client.crawl_url and a page limit, and search is client.search(query, {"limit": 5}). Keep the limit low. A crawl of a few hundred pages will outrun the model’s context long before it runs out of pages.

Which one to reach for

Start with MCP. One object, no SDK, and new Spider tools show up without a release on your side. Move to @tool when the model needs a smaller or differently shaped payload than the server sends, or when the tool has to do something before or after the fetch, such as caching a page or counting it against a per-user budget.