CrewAI integration

Give CrewAI agents a SpiderTool that scrapes and crawls the web. The agent gets markdown from any URL.

Install

Install the Spider client and the CrewAI tools package.

Install Spider client and CrewAI

pip install spider_client 'crewai[tools]'

Put your API key in SPIDER_API_KEY.

Usage

Create a SpiderTool and hand it to the agent. It returns markdown by default.

Create a SpiderTool

from crewai_tools import SpiderTool

# To enable scraping any website it finds during its execution
spider_tool = SpiderTool(api_key='YOUR_API_KEY')

SpiderTool arguments

What the constructor accepts.

Argument	Type	Description
api_key	string, optional	Your Spider API key. When omitted the tool reads `SPIDER_API_KEY` from the environment.
website_url	string	The site to scrape. Used as the fallback URL when set at init.
log_failures	bool	Log scrape failures instead of failing silently. Defaults to `true`.
custom_params	object, optional	Extra request parameters, passed through to the API.

Agent setup

Create an agent with the tool and give it a task.

An agent with SpiderTool

from crewai import Agent, Task

# Create a researcher agent
research_agent = Agent(
    role="Web Researcher",
    goal="Find and summarize information about the contents of a website URL",
    backstory='You are an expert web researcher tasked with analyzing website content and extracting valuable insights.',
    tools=[spider_tool()],
    verbose=True  # Enable logging for debugging
)

# Example task for the agent
task = Task(
    description='Analyze the website content and provide key insights',
    agent=research_agent
)

Next steps

The CrewAI tasks docs cover running several agents together.