Web scraping to markdown: a 63-page docs site in one call
Web scraping to markdown, measured on November 17, 2026. One crawl call returned a 63-page docs site as markdown in 3.5 seconds for $0.0138.
Web scraping to markdown took one request on a 63-page documentation site. On November 17, 2026, at 10:30 UTC, we sent that request to the Spider crawl endpoint for the SQLModel documentation and asked for markdown. It returned 63 pages, all with status 200, in 3.5 seconds, for $0.0138. A second request for the same 63 pages as raw HTML returned 8,635,386 bytes. The markdown came to 1,160,154 bytes and 127,372 words, and a short script wrote it to 63 files, one per page.
The $0.0138 is the pay as you go meter, which works out to $0.22 per 1,000 pages at this site’s page weight. The Unlimited plan is priced by requests in flight, from 2 at $40 a month, and not by page.
For a site like this one, the work is that request plus a loop that writes files. Our guide to web scraping covers the request itself. This post is about what ends up in the files, so we counted it against the HTML.
One request, then a loop that writes files
curl -s https://api.spider.cloud/crawl \
-H "Authorization: Bearer $SPIDER_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://sqlmodel.tiangolo.com/", "limit": 200, "return_format": "markdown", "request": "http", "metadata": true}'limit is a cap on pages. The crawl stopped by itself at 63. "request": "http" asks for a plain fetch with no browser, which is enough when the server sends the text in the first response. The measured run also turned the cache off, so that its 3.5 seconds would be a fresh fetch. The request printed here leaves the cache at its default.
The response is a JSON array with one object per page. The script below uses three of its fields, url, status and content, and names each file after the URL path.
import json
import os
import pathlib
import re
import urllib.parse
import urllib.request
START = "https://sqlmodel.tiangolo.com/"
OUT = pathlib.Path("site")
body = {
"url": START,
"limit": 200,
"return_format": "markdown",
"request": "http",
"metadata": True,
}
req = urllib.request.Request(
"https://api.spider.cloud/crawl",
data=json.dumps(body).encode(),
headers={
"Authorization": "Bearer " + os.environ["SPIDER_API_KEY"],
"Content-Type": "application/json",
},
)
with urllib.request.urlopen(req, timeout=300) as resp:
pages = json.load(resp)
for page in pages:
if page.get("status") != 200 or not page.get("content"):
continue
path = urllib.parse.urlparse(page["url"]).path.strip("/") or "index"
path = re.sub(r"[^A-Za-z0-9._/-]", "_", path).replace("..", "_")
dest = OUT / (path + ".md")
dest.parent.mkdir(parents=True, exist_ok=True)
dest.write_text(page["content"], encoding="utf-8")
print(dest, len(page["content"].encode()), "bytes")It needs Python 3 and nothing installed. We ran it as printed at 10:32 UTC, 81 seconds after the measured run started. It wrote 63 files, site/index.md and site/tutorial/connect/create-connected-tables.md among them, and all 63 matched the run’s pages byte for byte.
The same passage as HTML and as markdown
This is a heading, two paragraphs and the start of a code block from the page on connected tables, as the raw crawl returned them. Through the end of the code block the passage is 2,124 bytes. We cut it here after the first line of code.
<h3 id="create-the-team-table">Create the <code>team</code> Table<a class="headerlink" href="#create-the-team-table" title="Permanent link">¶</a></h3>
<p>Let's start by creating the tables in code.</p>
<p>Import the things we need from <code>sqlmodel</code> and create a new <code>Team</code> model:</p>
<div class="highlight"><pre><span></span><code><span id="__span-1-1"><span class="kn">from</span><span class="w"> </span><span class="nn">sqlmodel</span><span class="w"> </span><span class="kn">import</span> <span class="n">Field</span><span class="p">,</span> <span class="n">SQLModel</span><span class="p">,</span> <span class="n">create_engine</span>The same passage in the markdown file is 382 bytes. This is all of it up to the block’s last comment line and closing fence, which we left out.
### Create the `team` Table
Let's start by creating the tables in code.
Import the things we need from `sqlmodel` and create a new `Team` model:
```
from sqlmodel import Field, SQLModel, create_engine
class Team(SQLModel, table=True):
id: int | None = Field(default=None, primary_key=True)
name: str = Field(index=True)
headquarters: strThe heading keeps its level and the inline code keeps its backticks. The permalink sign that the docs generator puts after every heading is gone. Then read the code. The two blank lines above the class are there, the class body sits four spaces in, and create_engine and primary_key are spelled as the page spells them. The block in the file is the code the page shows.
What survived the conversion, counted
| Element | In the HTML | In the markdown |
|---|---|---|
| Code blocks, all 63 pages | 838 <pre> elements | 838 fenced blocks |
| Code blocks with the same text as the page, all 63 pages | 838 | 838 |
| Headings, 62 pages | 711 | 711 |
| Links other than heading anchors, 62 pages | 1,505 | 1,505 |
| Images, 62 pages | 77 | 77 |
| Tables, 62 pages | 53, with 200 rows | 53, with 200 rows |
Run of 2026-11-17. On the HTML side a regular expression counted elements inside each page’s <article>, and links and images inside a <pre> were left out. On the markdown side the script read lines outside fenced blocks. A heading is a line that starts with one to six # and a space. A link is a ](target) that is not an image. A table is a run of lines that each start and end with a pipe character, and its row count leaves out the separator line under the header. The counts match on each of the 62 pages, not only in total.
For the second row we took the text of each <pre> with its tags removed and its entities decoded. We compared it with the lines between the matching pair of fences, ignoring a newline at either end. The two were equal for all 838 blocks. That is the row we care most about. A count says a block is still a block. This row says the code in it is the code on the page.
The other rows are counts. They say a heading is still a heading and a table still has its rows, and they do not compare the words inside.
The last four rows leave out the home page. Its HTML has a badge link whose <a> tag is never closed, so two links nest there, and a count by regular expression is not reliable on that page. Its six code blocks are in the first two rows.
Half of this site’s HTML sat inside its code blocks
The 838 <pre> elements weigh 4,265,264 bytes, which is 49% of the HTML. The code inside them is 685,873 bytes, so the markup around it is 3,579,391 bytes, or 41% of the HTML. The site’s highlighter wraps each word and each space of code in a <span>, as the excerpt shows. Another 3,650,452 bytes, 42%, sit outside the <article> element in the head, the navigation tree and the scripts, and each page carries them again.
So 8,635,386 bytes to 1,160,154, or 7.4 to 1, mostly describes this site’s highlighter and its navigation. Across the 63 pages it ran from 2.5 to 1 on the release notes, which are nearly all prose and links, to 183 to 1 on the about page, which went from 51,838 bytes to 283.
That is bytes on one site. We used no tokenizer, so it is not a token count. A separate run on ten pages counted four formats in tokens.
Pick the format by what the page is for
Markdown fits pages where headings, links, tables and code carry the structure, as they do on this site. In this run the code arrived with the same text on all 63 pages, and the other elements with the same counts on the 62 pages we counted, in 13% of the bytes.
Ask for raw HTML when you need something markdown has no place for. An attribute is one case, and an element you want to select by CSS is another. The raw crawl of the same 63 pages took 3.3 seconds and cost $0.0132, so on a site this size you can keep both.
The API also has a text format. This run did not request it. The web scraper page returns one URL as markdown in the browser.
What we check before the files go into an index
We do these on any converted site.
- The first two lines of a file. A converter can put the page title above the first heading. Decide whether your chunks should carry both.
- Lines that repeat. Count each line across all files. A line found in most of them is worth a look before you index. Here “Back to top” was in all 63 files.
- One code block against the page. Open a file beside its page and compare the indentation in the first block. All 838 matched here, and on a new site the check takes a minute.
- Relative links. A link such as
../ai-development/is relative to the page’s URL, not to your folder.
What this run does not show
- More than one site. Its generator tag reads
zensical-0.0.73. A site built with another generator or a heavier theme will give different counts. - Tokens.
- Whether the words inside headings, links and tables match. For those we compared counts. For code we compared the text.
- Pages that need a browser. The request was a plain fetch.
- A second day. Each crawl ran once. The script’s run 81 seconds later returned the same bytes.
Point the script at a documentation site you already read, then open three of the files next to the pages. Create an API key to run it.
Frequently asked questions
How do I convert a whole website to markdown?
Send one crawl request with return_format set to markdown and write each page's content field to a file. On November 17, 2026 one such request returned all 63 pages of the SQLModel documentation in 3.5 seconds for $0.0138 on the pay as you go meter, and a short Python script wrote them to 63 files.
Does scraping to markdown keep code blocks?
In this run it did. The HTML held 838 pre elements and the markdown held 838 fenced blocks. The text inside each fenced block was the same as the text of its pre element on all 838, with indentation, blank lines and underscores as the page has them.
How much smaller is markdown than HTML?
On this one site, 8,635,386 bytes of HTML became 1,160,154 bytes of markdown, which is 7.4 to 1 in bytes. About half of that HTML was code blocks, most of it syntax highlighting markup. The figure is bytes on one site and not a token count.
When should I ask for raw HTML and not markdown?
When you need something markdown has no place for, such as an attribute or an element you select by CSS. In this run the raw crawl of the same 63 pages took 3.3 seconds and cost $0.0132, so on a site this size you can keep both.
Sources
- Measured run, 2026-11-17, 10:30 UTC: one crawl of sqlmodel.tiangolo.com with markdown output and one with raw HTML output, through the Spider crawl endpoint
- Spider OpenAPI file, the return_format values and the limit and request fields
- SQLModel documentation, the site that was crawled
- Spider pricing page, the pay as you go meter and the Unlimited plan
- What a web page costs in tokens: ten pages in four formats, counted with a tokenizer, measured 2026-11-05
Keep reading
Built-in proxies for web scraping: median $0.0025 a complete page
Proxies for web scraping come built into every Spider request. On November 17, 2026, a complete category or listing page cost a median of $0.0025, proxy included.
How web scraping works: what a plain fetch gets and misses
How web scraping works, measured on November 16, 2026. A plain HTTP fetch returned 90% or more of a browser's words on 26 of 29 docs, news and product pages.
Is web scraping legal? What 67 top sites ask for in robots.txt
We asked 100 top-ranked domains for robots.txt on 2026-11-16. Of the 67 that served one, 13 hold Disallow: / for unnamed crawlers and 29 name an AI crawler.
Run this on a page you care about
The playground sends the request this page describes and shows you the response. Keyless runs work without an account and are rate limited.