---
name: spider-agent
description: >-
  Fetch web pages with the spider-agent command line tool. Use it whenever a
  task needs the content, links, fields or a screenshot of a web page. It picks
  HTTP or a browser per page, escalates when a site refuses, stops at a credit
  budget, and exits with a code you can branch on. It calls no model.
---

## Install

Run this if `spider-agent --version` fails:

    curl -fsSL https://spider.cloud/install/spider-agent.sh | sh

It downloads the release build for this machine, checks it against the release's SHA256SUMS.txt, and puts it in ~/.local/bin. With a Rust toolchain, `cargo install spider-agent-cli` builds it instead.

## Key

    export SPIDER_API_KEY="sk-..."

The key comes from SPIDER_API_KEY, then SPIDER_CLOUD_API_KEY, then the keychain when the binary was built with the keyring feature, then ~/.spider/credentials. The first non-empty value wins. There is no flag for the key. `spider-agent login` signs in through a browser and stores the key instead.

## Which command

| Command | What it does |
|---|---|
| scrape | one page or a list of them |
| crawl | a site, following its links |
| extract | named fields, and no page bytes |
| links | the links on a page |
| search | a query |
| screenshot | a picture, into a file |
| run | work a goal over addresses until a cap stops it |
| route | the transport it would choose, worked out locally with no call and no spend |
| schema | the command tree and the record contract, as JSON |

`fetch` also exists. It asks for a config the service already stored for a path, and the first fetch on a new path is slow and can fail. Use scrape unless you want that stored config.

## Always pass --json or --ndjson

    spider-agent scrape https://example.com --goal markdown --json | jq -r '.items[0].body'
    spider-agent crawl https://example.com --limit 50 --ndjson -o pages.ndjson --budget 25
    cat urls.txt | spider-agent extract --selectors fields.json --ndjson

fields.json is `{"price": ".price", "title": ["h1", ".product-title"]}`. Results go to stdout and diagnostics go to stderr. The last NDJSON line is a report of what the run served, refused and cost. With fields, the page body never crosses the wire.

## Budget

    spider-agent run https://example.com --goal markdown --expand 20 --budget 40 --ndjson

Before every page, the run checks the cap against what is left. A cap that stops the run is exit 4.

## Exit codes

| Code | Meaning |
|---|---|
| 0 | done |
| 1 | failed, and none of the others describes it |
| 2 | usage, or an input that could not be read |
| 3 | auth: no key, a refused key, or a balance of zero |
| 4 | budget: a cap stopped the run |
| 5 | the site refused every attempt |
| 6 | transport: the call never reached the service, or it failed |
| 7 | output: a destination could not be written |

Retrying exit 5 from another address can work. Retrying exit 3 buys the same refusal.

## Free checks

`spider-agent route <url>` works locally, makes no call and spends nothing. `spider-agent schema` prints the command tree, the plan shape and the exit codes as JSON.

## Why

Nothing in it calls a large model. The decisions are rules, a local classifier over the shape of a URL, and arithmetic against a budget. The crate's request planner measured the bytes below over a recorded 7.8KB product page. The baseline is the 9021 bytes you get by asking for nothing in particular. The numbers measure the crate, not the service.

| Asked for | Bytes back | Saved |
|---|---|---|
| Need::fields(...) | 187 | 97.9% |
| Need::Metadata | 350 | 96.1% |
| Need::Links | 505 | 94.4% |
| Need::Text | 2156 | 76.1% |
| Need::Markdown | 2249 | 75.1% |
| Need::Html | 4342 | 51.9% |

## Docs

https://spider.cloud/docs/integrations/agent-cli/

Worked examples in Claude Code, Codex and the shell: https://spider.cloud/docs/integrations/agent-examples/