CrewAI Integration

CrewAI is a cutting-edge framework for orchestrating autonomous AI agents. CrewAI enables you to create AI teams where each agent has specific roles, tools, and goals, working together to accomplish complex tasks.

Install Spider client and CrewAI

pip install spider-client 'crewai[tools]'

Create an API key, then store it as an environment variable. This key will allow you to access the Spider API securely. If no API key is provided it looks for SPIDER_API_KEY in the env.

Usage

There are various ways you can use the SpiderTool to enable your agent to scrape and crawl websites. The data returned from the Spider API is LLM-ready.

Simple Instantiation of SpiderTool

from crewai_tools import SpiderTool

# To enable scraping any website it finds during its execution
spider_tool = SpiderTool(api_key='YOUR_API_KEY')

SpiderTool Arguments

api_key (string, optional): Specifies Spider API key. If not specified, it looks for SPIDER_API_KEY in environment variables.
website_url (string): The website URL. Will be used as a fallback if passed when the tool is initialized.
log_failures (bool): Log scrape failures or fail silently. Defaults to true.
custom_params (object, optional): Optional parameters for the request.

See custom API parameters that can be used with this integration at https://spider.cloud/docs/api.

Agent Setup

Agent Setup with SpiderTool

from crewai import Agent, Task

# Create a researcher agent
research_agent = Agent(
    role="Web Researcher",
    goal="Find and summarize information about the contents of a website URL",
    backstory='You are an expert web researcher tasked with analyzing website content and extracting valuable insights.',
    tools=[spider_tool()],
    verbose=True  # Enable logging for debugging
)

# Example task for the agent
task = Task(
    description='Analyze the website content and provide key insights',
    agent=research_agent
)

Check out the CrewAI docs on how to setup and run agent tasks https://docs.crewai.com/concepts/tasks.