Skip to main content

Web scraping for AI agents: what each call adds to the context

Web scraping for AI agents, measured November 17, 2026. Tokens, seconds and cost for a search with page content at 3, 5 and 10 results, one scrape and one crawl.

Jeff Mendez 5 min read

Web scraping for AI agents has two bills. The agent pays once to fetch a page, and it pays again in tokens each time the page goes to the model as context. The three calls an agent uses to read the web differ in tokens far more than they differ in price. On November 17, 2026 we made each call and counted what came back.

Agent stepCallTokens returnedTimeCost
A question, no URL yet/search with fetch_page_content, 3 results10,6872.7 s$0.0019
The same question/search, 5 results32,6918.1 s$0.0032
The same question/search, 10 results35,6505.6 s$0.0039
A URL the agent already has/scrape, one page2,6321.4 s$0.00017
A whole site, pages unknown/crawl, 63 pages295,4083.5 s$0.0138

Run of 2026-11-17, one call per row. Every call asked for markdown with the cache off, and every page in every answer had status 200. The scrape and the crawl were plain HTTP fetches of one documentation site. Tokens are the content fields counted with the o200k_base encoding. Time is the wall time of the call at the client. Cost is the sum of the costs.total_cost fields on the pay as you go meter. The requests printed below leave the cache at its default.

The scrape row is the same call our web scraper page makes for one URL. The rest of this post takes the rows in the order an agent meets them.

Ask for markdown before anything else

On ten public pages we fetched on November 5, 2026, the median page was 72,093 tokens as raw HTML and 4,384 as markdown. Page by page, markdown held 15 times fewer tokens at the median, and the ratio ran from 2.9 to 47.

So every request below sets return_format: "markdown". HTML is for an agent that needs a selector or a form field.

A question with no URL: search with page content

When a task starts as a sentence and not an address, one search call can return the results and their pages together:

curl -s https://api.spider.cloud/search \
  -H "Authorization: Bearer $SPIDER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"search": "kubernetes pod disruption budget explained", "search_limit": 3,
       "fetch_page_content": true, "return_format": "markdown"}'

We sent this query three times, with search_limit at 3, 5 and 10. Each returned exactly the number of results it asked for, every one with status 200 and its page as markdown in the content field.

The token totals did not grow in step with the limit. Three results came to 10,687 tokens. Five came to 32,691, about three times as much, because one of the five was a long page of 17,915 tokens. Ten came to 35,650. The set of ten did not include that long page, and its largest result was 5,464 tokens. Across the three calls a single result ran from 907 tokens to 17,915.

So in this run the number of results was a loose control on context, and the length of each page was the tight one. Two rules follow for an agent with a small window. Start at search_limit 3 and ask for more only when the first pages do not answer. And cap the tokens you keep from any one result in your own code, since one long page can outweigh the rest of the answer.

The times are one call each. We would not read 2.7, 8.1 and 5.6 seconds as a curve.

A URL the agent already has: scrape

When the user pastes a link, or a search result names the page that matters, the agent needs one page and nothing around it:

curl -s https://api.spider.cloud/scrape \
  -H "Authorization: Bearer $SPIDER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://sqlmodel.tiangolo.com/tutorial/connect/create-connected-tables/",
       "request": "http", "return_format": "markdown"}'

This call returned the page in 1.4 seconds as 2,632 tokens of markdown, for $0.00017. It is the cheapest step in the table on both bills, which is the reason to scrape a known URL and not search for it again.

request: "http" skips JavaScript, which suits documentation and articles. For a page that only renders in a browser, leave request out. The default, smart, moves to a browser only when the page needs it.

A whole site: crawl, then keep it out of the prompt

A task like “answer questions about this library’s docs” covers pages the agent cannot list. A crawl starts at one URL and follows the links:

curl -s https://api.spider.cloud/crawl \
  -H "Authorization: Bearer $SPIDER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://sqlmodel.tiangolo.com/", "limit": 200, "return_format": "markdown",
       "request": "http"}'

One call returned 63 pages of the SQLModel documentation in 3.5 seconds for $0.0138. As a fetch that is cheap, about $0.00022 a page. As context it is the largest answer in the table by far. The 63 pages came to 295,408 tokens, with a median page of 3,233 and a largest page of 39,452.

We would not put that in a prompt. Write the pages to files or chunk them into an index, then let the agent search that store and read the few pages a question needs. Each read is then one stored page, and the median page here is 3,233 tokens. Set limit to what the task needs. Another post times a crawl against a scrape per URL on one site.

The same calls through MCP

An agent in Claude Code, Codex or another MCP client can make these calls without any HTTP code. The hosted MCP server exposes them as spider_search, spider_scrape and spider_crawl. In Claude Code it is one command:

claude mcp add --transport http spider https://mcp.spider.cloud/mcp

The hosted server registers 27 tools. ChatGPT lists the seven core tools. The same instructions fit a model that holds all three tools. Tell it to search for questions, scrape known URLs and crawl into storage.

What these numbers do not show

Every row is one call on one day. The search rows are one query, and another query will rank pages of other lengths. The scrape and the crawl are one documentation site that needs no browser. Nothing here measures a site that renders in JavaScript, a page behind a login, or many agents calling at once. Run the three requests on the pages your agent will read and count the tokens yourself.

To give your agent these calls, create an API key.

Frequently asked questions

How many tokens does a web search add to an agent's context?

In our run it depended on which pages came back more than on how many. On November 17, 2026 three search calls that also fetched their results as markdown returned 10,687 tokens for 3 results, 32,691 for 5 and 35,650 for 10. A single long page in the set of 5 was 17,915 tokens. Start at 3 results and cap the tokens you keep from each page.

Should an agent search first or scrape first?

Search first when the agent has a question and no URL. Scrape first when it already has the URL, from the user or from an earlier step. In our run one scrape call returned a documentation page as 2,632 tokens of markdown in 1.4 seconds.

When does an agent need a crawl?

When the task covers a whole site and the agent does not know its pages. One crawl call returned 63 pages of a documentation site in 3.5 seconds on November 17, 2026. As markdown they came to 295,408 tokens, so a crawl result belongs in files or an index the agent searches, not in the prompt.

Can an agent use these calls through MCP?

Yes. The hosted Spider MCP server exposes search, scrape and crawl as the spider_search, spider_scrape and spider_crawl tools, among the 27 it registers.

Sources

  1. Measured run, 2026-11-17, 10:30 to 10:36 UTC: three search calls with fetch_page_content, one scrape call and one crawl call through the Spider API, tokens counted with tiktoken 0.14.0, o200k_base encoding
  2. Measured run, 2026-11-05: tokens in ten pages as raw HTML and as markdown, o200k_base encoding
  3. Spider OpenAPI file, the search, scrape and crawl routes and the fetch_page_content, search_limit, limit, request and return_format parameters
  4. Spider MCP server documentation, the hosted endpoint and its tools

Keep reading

Built-in proxies for web scraping: median $0.0025 a complete page

Proxies for web scraping come built into every Spider request. On November 17, 2026, a complete category or listing page cost a median of $0.0025, proxy included.

Web scraping to markdown: a 63-page docs site in one call

Web scraping to markdown, measured on November 17, 2026. One crawl call returned a 63-page docs site as markdown in 3.5 seconds for $0.0138.

Web scraping with JavaScript: a Node.js script that streams each page to disk

Web scraping with JavaScript in Node.js and the Spider client. One script scrapes a page, reads a field by CSS selector and streams a 25 page crawl to a file. Run against the API on November 17, 2026.

Run this on a page you care about

The playground sends the request this page describes and shows you the response. Keyless runs work without an account and are rate limited.