Skip to main content

CrewAI Integration

Give CrewAIagents the ability to scrape and crawl the web using Spider's SpiderTool. Agents get LLM-ready content from any URL.

Install

Install Spider and CrewAI tools:

Install Spider client and CrewAI

pip install spider_client 'crewai[tools]'

Set your API keyas SPIDER_API_KEY in your environment.

Usage

Initialize SpiderTool and pass it to your agent. Data returned is LLM-ready markdown by default.

Simple Instantiation of SpiderTool

from crewai_tools import SpiderTool # To enable scraping any website it finds during its execution spider_tool = SpiderTool(api_key='YOUR_API_KEY')

SpiderTool Arguments

Configuration options for the SpiderTool.

  1. api_key (string, optional): Specifies Spider API key. If not specified, it looks for SPIDER_API_KEY in environment variables.
  2. website_url (string): The website URL. Will be used as a fallback if passed when the tool is initialized.
  3. log_failures (bool): Log scrape failures or fail silently. Defaults to true.
  4. custom_params (object, optional): Optional parameters for the request.

Agent Setup

Create an agent with SpiderTool to scrape and research web content.

Agent Setup with SpiderTool

from crewai import Agent, Task # Create a researcher agent research_agent = Agent( role="Web Researcher", goal="Find and summarize information about the contents of a website URL", backstory='You are an expert web researcher tasked with analyzing website content and extracting valuable insights.', tools=[spider_tool()], verbose=True # Enable logging for debugging ) # Example task for the agent task = Task( description='Analyze the website content and provide key insights', agent=research_agent )

Next steps

See the CrewAI tasks docsfor running multi-agent workflows.