Fetching the web is the cheap part of an agent's research bill
We priced the fetch and the page input for five agent research tasks on 2026-11-18. Fetching ten pages was the small line. Reading them as raw HTML took 6.5x to 22.5x the tokens of markdown on four tasks and 1.8x on the fifth.
An agent that researches the web runs up two bills. On 2026-11-18 we measured the web content on both for one task, a database question: a search and the ten pages it returned. Reading those pages cost $0.14 as markdown and $2.87 as raw HTML, with the model input computed at $3 per million tokens.
Fetching the pages was the small line and reading them was the large one. That held on all five tasks we ran. The same pages took 6.5x to 22.5x more tokens as raw HTML than as markdown on four of them, and 1.8x on the fifth.
Those costs land on two invoices. The web data bill is itemized by page. The model bill says “input tokens” and a total. To see what the web content in one task costs you have to add them up yourself, so we did.
Which five tasks did we measure?
We wrote the five questions down before the run and dropped none of them afterwards:
- how does HTTP/3 differ from HTTP/2
- EU AI Act obligations for general purpose AI models
- postgres vs sqlite for a small web app
- how do heat pumps perform in cold climates
- what is retrieval augmented generation and when to use it
Each question went to Spider’s search endpoint as one call. That call searches, fetches ten results and returns them as markdown. Then we fetched the same ten addresses again as raw HTML, which is what a model gets when nothing converts the page first, and counted the tokens on both sides with gpt-tokenizer’s o200k_base encoding. A page is compared when both formats came back with status 200 and more than 50 tokens. 48 of the 50 pages met that rule. One answered 403 in both formats and one came back with almost no markdown, so both are left out. One result in task 5 was a PDF, which is not HTML, so it is counted on its own further down.
| Task | Spider fetch | Raw HTML tokens | Markdown tokens | Ratio | Input as raw HTML | Input as markdown |
|---|---|---|---|---|---|---|
| 1 | $0.0068 | 1,167,789 | 175,798 | 6.6x | $3.50 | $0.53 |
| 2 | $0.0031 | 323,554 | 49,485 | 6.5x | $0.97 | $0.15 |
| 3 | $0.0056 | 956,209 | 46,730 | 20.5x | $2.87 | $0.14 |
| 4 | $0.0051 | 817,493 | 36,297 | 22.5x | $2.45 | $0.11 |
| 5 | $0.0215 | 978,408 | 540,506 | 1.8x | $2.94 | $1.62 |
Measured 2026-11-18. Task 1 compares nine pages, task 5 eight and the others ten. The fetch column covers all ten results of each task.
The fetch column is Spider’s side of the task: the page charges the call returned, plus about $0.001 for the search at list price. Cost varies by site.
The two input columns are computed, not measured. They are the token counts multiplied by $3 per million input tokens, a round figure we chose. Swap in the rate you pay. The dollars will move and the ratio will not. They cover the page content only. The prompt, the model’s output and any reasoning it bills for are not counted here.
We charged the raw HTML side nothing for fetching. In the run that fetch cost about the same as the markdown one, so the table is kind to it.
Where does the money go?
Read the fetch column, then the two on the right. Task 3 is the middle one by raw size. With Spider’s fetch added to the markdown side, it costs about $0.15 one way and $2.87 the other.
The ten pages were the same on both sides. Only the format changed, and the format moved the reading from 46,730 tokens to 956,209.
On four tasks the gap ran from 6.5x to 22.5x. Across all 47 HTML pages it was 5.0x, and page by page the median was 10.5x. An earlier post measures single pages in four output formats. What this run adds is the fetch charge beside the tokens, and at these sizes the fetch charge is a rounding error. Someone hunting for savings on the web data line is looking at half a cent. The $2.73 sits on the other invoice.
The table also hides a second cost. The raw HTML passed 200,000 tokens in all five tasks, so no task’s raw pages fit in one 200,000-token context window, a common size. As markdown, four of the five fit. When the reading does not fit, the agent has to split it, summarize it or drop pages before it can answer.
Now run task 3 a thousand times. The computed input is $2,869 as raw HTML and $140 as markdown. Spider’s share is $4.61 in page charges and $1.00 in search fees on pay as you go. This call is a search, and search stays on the meter on every plan, at the page charges plus about $0.001 a search. Unlimited is $40 a month for two request slots and covers scrape and crawl calls, so it suits an agent that fetches addresses it already has. On either plan, the token line is the one to manage.
What happens when a result is a whole book?
Task 5 is the row with the small ratio. The question was ordinary. Three of its results were long index pages that are nearly all text, 149,000 to 180,000 tokens each as markdown. Markdown has little to remove from a page like that, and the task came to 1.8x.
A fourth result was a PDF of a whole book. It is not in the table, because a PDF is not HTML. As markdown it was 865,414 tokens, more than every other result of the task together. That is $2.60 of reading at the assumed rate, for a question that never needed it.
Nobody plans for the book. It turns up in the results of an ordinary question, and an agent told to read the results will read it. A cleaner format only makes the book smaller. A limit on the task is what keeps it out.
What should you ask your team on Monday?
Three questions will tell you most of what you need.
- When our agent reads a web page, what does the model receive? Raw HTML, or text with the markup removed?
- What is the most one task can spend? Is there a cap on results per search and on the size of a page?
- What share of last month’s input tokens was web content? If nobody knows, measure that first.
How do you run it on your own questions?
This is the call from the run:
curl -s https://api.spider.cloud/search \
-H "Authorization: Bearer $SPIDER_API_KEY" \
-H "Content-Type: application/json" \
-d '{"search": "postgres vs sqlite for a small web app", "search_limit": 10, "fetch_page_content": true, "return_format": "markdown"}'search_limit caps how many results are fetched, so ask for fewer when fewer will do. It does not count tokens, so check the length of each page in your own code before it reaches a prompt. That check is what stops the book.
If the agent speaks MCP, one line adds Spider’s search as a tool:
claude mcp add --transport http spider https://mcp.spider.cloud/mcpTake one question your agent asked this week and run it both ways. Count the tokens with your model’s tokenizer and multiply by your rate. This was five questions on one day, and your pages will differ. Two numbers from one task will tell you which line of your bill is the large one.
Frequently asked questions
What does one agent research task cost?
In a run on 2026-11-18, a task was one search plus the ten pages it returned. Spider's side came to $0.0031 to $0.0215 a task: the page charges it returned plus about $0.001 for the search at list price. Cost varies by site. Model input, computed at $3 per million tokens, came to $0.11 to $1.62 a task as markdown and $0.97 to $3.50 as raw HTML, with one book PDF counted apart. Swap in the rate you pay.
How many fewer tokens does markdown use than raw HTML?
On the 47 HTML pages compared in the 2026-11-18 run, raw HTML held 6.5x to 22.5x the tokens of markdown on four tasks and 1.8x on the fifth, whose results were long pages of plain text. The median page was 10.5x. One page was 1.1x and one was 142.5x, so measure your own pages. Tokens were counted with gpt-tokenizer and the o200k_base encoding.
When does markdown not keep a task cheap?
When a result is very long. One task in the run pulled in a PDF of a whole book, 865,414 tokens as markdown. That is far too much to hand a model. Ask for fewer results with search_limit, and check the length of each page before it goes into a prompt.
What does Spider charge to fetch the pages?
From $0.16 per 1,000 pages on light static pages. Cost varies by site. A search that returns results adds about $0.001. The Unlimited plan is $40 a month for two request slots and covers scrape and crawl calls. Search stays on the meter.
Sources
Keep reading
What a web page costs in tokens, measured on ten public pages
Raw HTML, markdown, readability and text output for the same ten pages, counted in o200k tokens. Markdown cut the bill 15x at the median, and readability mode left four index pages with under 60 tokens.
Proxies for web scraping, built into every request
Proxies for web scraping come built into every Spider request and rotate on their own. What they cost, and how to pick the exit country or use your own.
Web scraping to markdown: a whole docs site into files
Web scraping to markdown with one crawl call and a short Python script that writes a file per page, then a check of the code blocks and tables it kept.
Run this on a page you care about
The playground sends the request this page describes and shows you the response. Keyless runs work without an account and are rate limited.