What a web page costs in tokens, measured on ten public pages

An agent that reads the web pays for every page twice. Once to fetch it, and again every time the page sits in the context window. The second bill is the one that grows, and it depends on which form of the page you kept. I wanted a number instead of a feeling, so on 2026-11-05 I fetched ten public pages through our scrape endpoint in four output formats and counted the tokens.

The run

Ten pages, chosen for shape rather than fame: a Wikipedia article, four documentation pages (Python, MDN, Kubernetes, React), a GitHub repository page, two news front pages (BBC, the Guardian), the Hacker News front page and the gov.uk organisations index. Each one fetched four times with return_format set to raw, markdown and text, plus markdown with readability: true. Tokens counted with gpt-tokenizer 4.0.0 using the o200k_base encoding.

Two pages from the original list of twelve are out of the table. Stack Overflow answered with a 403, and the arXiv abstract came back as a 6.8 KB stub whose markdown was 38 bytes. I report both because a benchmark that hides its misses is an ad.

Tokens per page, o200k_base, 2026-11-05. Totals across the ten: 1,032,403 raw, 87,029 markdown, 32,890 text, 14,613 readability.

What the numbers say

Markdown cut the token count by 15x at the median, with a range of 2.9x on Hacker News to 47x on the Kubernetes page. The spread is the interesting part. Hacker News is already mostly text, so there is little to strip. The Kubernetes overview is a short chapter wrapped in a 480 KB shell of navigation, scripts and inline SVG, and almost all of it goes.

Text came in smaller than markdown on 8 of the 10 pages, 19.6x at the median. The difference is link targets and heading marks. That is exactly the material an agent uses to decide what to fetch next, so text is the right format when the page is the answer and the wrong one when the page is a step in a crawl. The two exceptions, MDN and Kubernetes, are pages where the markdown converter dropped boilerplate that the text path kept.

Readability is the one I would warn about. On the article pages it did what it says: the Wikipedia article halved again, the React chapter lost a third, the GitHub page kept the README and dropped the rest. On the four index pages it returned 3, 19, 21 and 59 tokens. A front page has no single body for the Reader algorithm to find, so it keeps almost nothing, and the response still comes back as a 200. If your pipeline turns readability on globally, your agent will read the BBC front page as a single sentence and believe it.

What to do with this

Set the format per page type, not per pipeline. Markdown is the safe default for anything an agent will navigate from. Text is for pages you are summarizing and discarding. Readability is for pages you already know are articles, with a byte-length check on the result so an empty body fails loudly instead of landing in your embeddings.

If you store raw HTML today, the median page in this run is 72,093 tokens raw against 4,384 as markdown. At any per-token price, that gap is the difference between a context window that holds one page and one that holds sixteen.

Reproduce it with your own URLs. The call for one format is a single request:

curl -s https://api.spider.cloud/scrape \
  -H "Authorization: Bearer $SPIDER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://kubernetes.io/docs/concepts/overview/", "return_format": "markdown", "readability": false}'

Swap return_format and readability, count the content field with the tokenizer your model uses, and compare against the raw fetch. For HTML you already hold, the transform endpoint converts it without fetching again. Start with one page you are paying for right now and see what it costs in each form.