Agent CLI
spider-agent is the command line tool for the Spider Cloud API. It picks plain HTTP or a browser for each page, escalates when a site refuses, stops at a budget you set, and returns only the fields you asked for. It calls no language model. Every decision is a rule or a sum against the budget.
Install
One line downloads the build for your machine, checks it against the release checksums, and puts it in ~/.local/bin. Builds cover macOS and Linux on arm64 and x86_64. Anywhere else, install it from crates.io with a Rust toolchain from rustup.rs.
Install the release build
curl -fsSL https://spider.cloud/install/spider-agent.sh | sh spider-agent --version
Or build from crates.io
cargo install spider-agent-cli
Your API key
The tool reads SPIDER_API_KEY first, then the keychain, then ~/.spider/credentials. There is no flag for the key, because a flag shows up in the process list. Keys come from the API keys page.
Set it in the environment
export SPIDER_API_KEY="sk-..."
Or sign in through a browser and store it
spider-agent login
From Claude Code or Codex
Add one line to the file your agent reads at launch. Claude Code reads CLAUDE.md and Codex reads AGENTS.md. The line points at a skill file that tells the agent how to install the tool, which command fits which job, and what the exit codes mean. Worked jobs are on the agent examples page.
CLAUDE.md or AGENTS.md
For web pages use spider-agent. Read https://spider.cloud/agent-skill/spider-agent/SKILL.md first.
Keep the skill on disk
Both clients also load skills from a directory, so the agent never has to fetch the file. Claude Code looks in ~/.claude/skills and Codex looks in ~/.agents/skills.
Claude Code
mkdir -p ~/.claude/skills/spider-agent && curl -fsSL https://spider.cloud/agent-skill/spider-agent/SKILL.md -o ~/.claude/skills/spider-agent/SKILL.md
Codex
mkdir -p ~/.agents/skills/spider-agent && curl -fsSL https://spider.cloud/agent-skill/spider-agent/SKILL.md -o ~/.agents/skills/spider-agent/SKILL.md
Three commands worth copying
One page as JSON, a site streamed to disk under a credit cap, and named fields off a list of addresses piped in. With fields, the page body never crosses the wire.
One page as JSON
spider-agent scrape https://example.com --goal markdown --json | jq -r '.items[0].body'
A site to NDJSON on disk, streamed as it arrives
spider-agent crawl https://example.com --limit 50 --ndjson -o pages.ndjson --budget 25
Named fields off a list of addresses
cat urls.txt | spider-agent extract --selectors fields.json --ndjson
fields.json. Several selectors under one name are tried in order
{"price": ".price", "title": ["h1", ".product-title"]}Commands
The ones you will reach for. spider-agent --help lists the rest, including the account reads, which fetch no page and cost no credits.
| Command | What it does |
|---|---|
| scrape | one page or a list of them |
| crawl | a site, following its links |
| extract | named fields, and no page bytes |
| links | the links on a page |
| search | a query |
| screenshot | a picture, into a file |
| run | work a goal over addresses until a cap stops it |
| credits | the balance |
| login | sign in through a browser and store the key |
| schema | the command tree and the record contract, as JSON |
Built to be called by another program
Results go to stdout and diagnostics go to stderr. Nothing prints a spinner or asks a question. --ndjson writes one object per line and flushes each one, so a caller can read the first page of a crawl while the tool is still fetching the last. The last line is always a report that says what the run served, refused, and cost. --json writes one document instead and buffers until the run ends.
Exit codes
A site that refused is code 5 and a key that was refused is code 3. Retrying the first from another address can work. Retrying the second buys the same refusal.
| Code | Meaning |
|---|---|
| 0 | done |
| 1 | failed, and none of the others describes it |
| 2 | usage, or an input that could not be read |
| 3 | auth: no key, a refused key, or a balance of zero |
| 4 | budget: a cap stopped the run |
| 5 | the site refused every attempt |
| 6 | transport: the call never reached the service, or it failed |
| 7 | output: a destination could not be written |
Source
The binary ships as spider-agent-cli and the same client is a Rust library, spider-cloud-agent. Both come from spider-rs/spider-cloud-agent on GitHub, MIT licensed.