Agent examples

Each example is a job someone hands to Claude Code or Codex, the spider-agent command the agent runs for it, and what the command prints. The shell and Rust sections do the same work with no agent in the loop. For every flag and exit code, see the Agent CLI reference.

You

Fill in title, price and stock in data/books.csv for every address in urls.txt. Use spider-agent and keep it under 2 credits.

Claude

cat > fields.json <<'EOF'
{"title":"h1","price":".price_color","stock":".instock.availability"}
EOF
spider-agent extract --urls-from urls.txt --selectors fields.json --budget 2

stdout

{"body":null,"bytes":62,"call_elapsed_ms":1497,"content":"fields","cost_credits":0.03091265,"duration_ms":994,"fields":{"price":["£51.77"],"stock":["In stock (22 available)"],"title":["A Light in the Attic"]},"status":200,"type":"page","url":"https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"}
{"approx_tokens_in":0,"approx_tokens_out":23,"attempts":1,"cost_credits":0.03091265,"elapsed_ms":1497,"refused":0,"returned_bytes":90,"served":1,"stopped":null,"targets":1,"type":"report","wire_bytes":713}

Claude writes the selectors to a file, runs extract through its Bash tool and reads the two NDJSON lines back: one page record, then the report.

Reportserved1refused0attempts1cost_credits0.03091265

Before the examples

Once per machine

Install the tool and sign in once. The agent then finds the stored key on its own, so no example below passes one. To tell the agent the tool exists, add the line and the skill file from From Claude Code or Codex on the Agent CLI page.

Install, then sign in

curl -fsSL https://spider.cloud/install/spider-agent.sh | sh
spider-agent login

Claude Code

Runs through the Bash tool

You type the request in a Claude Code session. Claude picks the command, runs it through its Bash tool, and reads what comes back. The requests here are examples. The commands run as pasted.

Fill a CSV from product pages

Claude Code

urls.txt holds one product address per line. Claude writes the selectors to a file and runs extract, which returns the named fields and none of the page. Extract writes NDJSON, one page record per address and a report last, so Claude appends one CSV row per record and checks served against targets in the report.

1You type
The request in the Claude Code tab at the top, which asks for the three columns under 2 credits.

2Claude runs

cat > fields.json <<'EOF'
{"title":"h1","price":".price_color","stock":".instock.availability"}
EOF
spider-agent extract --urls-from urls.txt --selectors fields.json --budget 2

3Comes back
The two lines in that tab: one page record with the three fields, then the report with served at 1 for 1 target.

Read a library's docs before writing code

Claude Code

Claude crawls the docs site as markdown instead of guessing at the API from memory. --limit and --depth bound the pages, and --max-tokens bounds what Claude has to read. The pages are written together when the crawl finishes, so Claude waits for the command to exit before it reads. If the crawl would pass --budget, the tool stops and exits 4, and Claude tells you rather than raising the cap.

1You type
Read the docs at https://example.com/docs/ before you write the client for that API.

2Claude runs

spider-agent crawl https://example.com/docs/ --limit 20 --depth 2 --goal markdown --max-tokens 20000 --budget 5

Retry a page that failed

Claude Code

A failed record carries error, billed and hint. Claude reads the hint and changes the one flag it names, instead of sending the same command again. $PAGE holds the address that failed, and this run came back with try_browser.

1You type
The scrape of our competitor's pricing page came back failed. Find out why and get the page.

2Claude reads

spider-agent scrape "$PAGE" --goal markdown --ndjson | jq -r 'select(.type == "failed") | .hint'

3Comes back
```
try_browser
```

4Claude runs next

spider-agent scrape "$PAGE" --goal markdown --mode browser --budget 2

Hint to flag

One change each, or a reason to stop

try_browser
one flag
Run again with --mode browser.
try_residential_proxy
one flag
Run again with --proxy residential.
try_different_country
one flag
Run again with --country and another country code.
needs_session
stop, ask
The page is behind a login. Stop and ask you for a session.
permanent
stop
Stop. No flag changes the answer.
slow_down
wait
Wait, then run the same command again. Raising the budget makes it worse.
unknown
read error
No hint fits. Read error before changing a flag.

Run Claude Code from a script

Claude Code

claude -p runs one request and exits, and --allowedTools lets Claude run spider-agent without stopping to ask.

1Headless

claude -p "Read https://example.com with spider-agent and summarise it" --allowedTools 'Bash(spider-agent:*)'

2Flag order
Tip
Put the prompt before --allowedTools. That flag takes several values, so a prompt placed after it is read as one more tool name.

Codex

Runs through the shell tool

Codex runs the same commands through its shell tool. These three jobs differ from the Claude Code ones, and either agent can do any of the six.

Answer a question from the web

Codex

search writes NDJSON by default. --fetch-pages reads the pages the results point at, so Codex answers from those pages and writes the answer into the repo with a link to each source.

1You type
Find out how crawlers treat the crawl-delay line in robots.txt and write it up in docs/crawl-delay.md.

2Codex runs

spider-agent search "robots.txt crawl-delay directive" --limit 5 --fetch-pages --max-tokens 8000 --budget 2

Check release notes before a bump

Codex

scrape writes text by default, so the markdown lands on stdout where Codex reads it. --max-tokens keeps a long changelog from filling the context. Codex then compares the notes with the calls your code makes.

1You type
Before you bump the HTTP client, read its release notes at https://example.com/releases and list anything that breaks our code.

2Codex runs

spider-agent scrape https://example.com/releases --goal markdown --max-tokens 4000

Compare the links on two versions of a page

Codex

links prints one address per line on stdout and its summary on stderr, so the redirect keeps only the addresses and diff shows what the new page dropped or added.

1You type
Check that the redesigned page at https://example.com/new/ keeps every link the current https://example.com page has.

2Codex runs

spider-agent links https://example.com > links-current.txt
spider-agent links https://example.com/new/ > links-new.txt
diff links-current.txt links-new.txt

3First call prints

https://iana.org/domains/example
1 served, 0 refused, 1 calls, 0.001 credits, 1096 ms

The first line above is stdout. The second is the summary on stderr, which the redirect leaves out of the file.

Run Codex from a script

Codex

codex exec runs one request and exits. The -c override turns on network access inside the workspace-write sandbox.

1Headless

codex exec -s workspace-write -c sandbox_workspace_write.network_access=true "List the links on https://example.com with spider-agent"

2Sandbox
Tip
The workspace-write sandbox has network access off by default. Without the override, spider-agent never reaches the API, gives up after its attempt limit and exits 4 with budget exceeded: attempts. That is the attempt cap, not your credits. Add -c sandbox_workspace_write.network_access=true and run it again.

3Network off prints

0 served, 0 refused, 5 calls, 0.000 credits, 981 ms, stopped on budget
budget exceeded: attempts
exit 4

On its own

No agent in the loop

Cron, a shell script or a Makefile can call spider-agent directly and branch on its exit code.

A nightly crawl in cron

Shell

The script appends each night's records to one NDJSON file with -o and --append, and -q keeps cron mail quiet. Exit 4 means a budget stopped the crawl, and exit 3 means the key is missing or refused, so the script says which one happened.

1crontab -e
```
0 2 * * * $HOME/bin/books-nightly.sh
```

2~/bin/books-nightly.sh

#!/bin/sh
spider-agent crawl https://books.toscrape.com/ --goal markdown --limit 50 --budget 5 \
  -o "$HOME/data/books.ndjson" --append -q
code=$?
case $code in
  0) ;;
  4) echo "books: stopped on the budget, raise --budget or lower --limit" >&2 ;;
  3) echo "books: no usable key, run spider-agent login" >&2 ;;
  *) echo "books: spider-agent exited $code" >&2 ;;
esac
exit $code

3A budget stop, by hand

$ spider-agent crawl https://books.toscrape.com/ --limit 50 --budget 0.01 > /dev/null
stopped on budget, after 0 page(s) and 0.000 credits
$ echo $?
4

Exit codes

Marked: the two the script names

0done
1failed
2usage
3auth
4budget
5refused
6transport
7output

A job from a plan file

Shell

A plan file keeps the goal, addresses, selectors and caps in version control. This one reads the start page, follows its links one step, and stops at 3 pages or 2 credits. Flags on the command line win over the plan. run writes NDJSON, and in the report stopped is null because no cap was reached.

1plan.json

{
  "goal": "fields",
  "urls": ["https://books.toscrape.com/"],
  "expand": 1,
  "max_pages": 3,
  "budget": 2,
  "selectors": { "title": "h1", "price": ".price_color" }
}

2Run it
```
spider-agent run --plan plan.json
```

3The last line

{"approx_tokens_in":0,"approx_tokens_out":2460,"attempts":2,"cost_credits":0.1208323,"elapsed_ms":1301,"refused":0,"returned_bytes":9838,"served":2,"stopped":null,"targets":1,"type":"report","wire_bytes":6920}

Screenshots into a directory

Shell

With -d, each page gets its own file named from its address, and --mkdir creates the directory. A screenshot needs -o, -d or a redirected stdout.

1Addresses

Two pages

spider-agent screenshot https://example.com https://books.toscrape.com/ -d shots --mkdir

Convert HTML you already have

Shell

transform fetches nothing. It converts the markup you give it, from a file or from stdin with --input=-.

1page.html

<html><body><h1>Release 2.4</h1><p>Adds <b>retries</b>.</p></body></html>

2Run it
Read page.html
spider-agent transform --input page.html
3What it prints
```
# Release 2.4
Adds **retries**.
```

Check the route and the balance first

Shell

route works out locally which transport and proxy a URL would get. It makes no call and costs nothing. Before a large crawl, credits prints the balance.

1One address

spider-agent route https://books.toscrape.com/

2Address, mode, proxy
```
https://books.toscrape.com/	smart	isp
```
3The balance
```
spider-agent credits
```

The three columns of the route line are tab separated.

In a Rust program

spider-cloud-agent 0.4

This program crawls a few pages of a catalogue under a client budget and asks for three named fields with Need::fields, so no page body comes back. A crawl that hits a cap returns an error, and the program tells a budget stop, an empty account and an exhausted walk apart before it exits.

A crawl under a credit cap

Rust

Two files. The budget on the client applies to every call it makes, and the match at the end sorts the ways a crawl can stop.

1Cargo.toml

[dependencies]
spider-cloud-agent = "0.4"
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }

2src/main.rs

use std::time::Duration;

use spider_cloud_agent::{Budget, Credits, Error, Need, Spider};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Every operation on this client stops at 2 credits, 3 calls or 90 seconds.
    let budget = Budget::default()
        .with_credits(Credits::new(2.0))
        .with_attempts(3)
        .with_wall(Duration::from_secs(90));
    let spider = Spider::builder().budget(budget).build()?;

    let crawl = spider
        .crawl("https://books.toscrape.com/")
        .limit(3)
        .depth(1)
        .need(Need::fields([
            ("title", "h1"),
            ("price", ".price_color"),
            ("stock", ".instock.availability"),
        ]))
        .send_all()
        .await;

    let pages = match crawl {
        Ok(pages) => pages,
        Err(error) => {
            match error.cause() {
                Error::BudgetExceeded { kind, .. } => eprintln!("stopped by the {kind} cap"),
                Error::InsufficientCredits => eprintln!("no credits left, do not retry"),
                Error::Exhausted { reason, .. } => eprintln!("no usable page: {reason}"),
                other => eprintln!("{other}"),
            }
            eprintln!("spent before the stop: {}", error.spent());
            std::process::exit(1);
        }
    };

    for page in pages.ok() {
        println!("{}", page.url);
        if let Some(fields) = page.body.fields() {
            for (name, value) in fields {
                println!("  {name}: {value}");
            }
        }
    }
    for failed in pages.failed() {
        println!("{} failed, next change: {:?}", failed.url, failed.hint);
    }
    println!("{} in {} call(s)", pages.cost, pages.attempt_count());

    Ok(())
}

Where to go next

The Agent CLI reference lists every command, flag and exit code. The MCP page covers the hosted server, for clients that call tools rather than a shell. The skill file is what Claude Code and Codex load to learn the tool.