CrewAI Integration

CrewAI is a cutting-edge framework for orchestrating autonomous AI agents. It enables the creation of AI teams where each agent has a specific role, tools, and goals, working collaboratively to accomplish complex tasks.

Install Spider client and CrewAI

Use pip to install the official Spider and CrewAI clients.

Install Spider client and CrewAI

pip install spider_client 'crewai[tools]'

Create an API key, then store it as an environment variable. This key will allow you to access the Spider API securely. If no API key is provided it looks for SPIDER_API_KEY in the env.

Usage

There are various ways you can use the SpiderTool to enable your agent to scrape and crawl websites. The data returned from the Spider API is LLM-ready.

Simple Instantiation of SpiderTool

from crewai_tools import SpiderTool

# To enable scraping any website it finds during its execution
spider_tool = SpiderTool(api_key='YOUR_API_KEY')

SpiderTool Arguments

Use arguments to adjust the way you get data.

api_key (string, optional): Specifies Spider API key. If not specified, it looks for SPIDER_API_KEY in environment variables.
website_url (string): The website URL. Will be used as a fallback if passed when the tool is initialized.
log_failures (bool): Log scrape failures or fail silently. Defaults to true.
custom_params (object, optional): Optional parameters for the request.

Agent Setup

Use a CrewAI agent with Spider to research, automate, and more.

Agent Setup with SpiderTool

from crewai import Agent, Task

# Create a researcher agent
research_agent = Agent(
    role="Web Researcher",
    goal="Find and summarize information about the contents of a website URL",
    backstory='You are an expert web researcher tasked with analyzing website content and extracting valuable insights.',
    tools=[spider_tool()],
    verbose=True  # Enable logging for debugging
)

# Example task for the agent
task = Task(
    description='Analyze the website content and provide key insights',
    agent=research_agent
)

Setup more agents with CrewAI

Check out the CrewAI docs on how to setup and run agent tasks https://docs.crewai.com/concepts/tasks.