Build a fine-tuning dataset from the web in two API calls

To build a fine-tuning dataset from the web you need two calls. A search call finds the pages for your topic and can return their content in the same response. A crawl call fetches the site you pick as markdown. On 2026-11-06 I ran both for one topic. They took 11 seconds together and cost $0.052, and after two short scripts I had 396 training records and 84 held-out sections.

# 1. find pages for a topic, with their content
curl -s https://api.spider.cloud/search \
  -H "Authorization: Bearer $SPIDER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"search": "kubernetes pod disruption budget explained", "search_limit": 10,
       "fetch_page_content": true, "return_format": "markdown"}' -o search.json

# 2. fetch the source you chose, as markdown
curl -s https://api.spider.cloud/crawl \
  -H "Authorization: Bearer $SPIDER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://kubernetes.io/docs/concepts/", "limit": 60,
       "return_format": "markdown", "whitelist": ["/docs/concepts/"],
       "metadata": true, "request": "http"}' -o crawl.json

Search: a topic in, ten pages out

I started from a topic, not a list of URLs. The search call returned ten results in 4.5 seconds, each with its page content as markdown. All ten had status 200. Together they were 207,916 bytes and 47,931 tokens in the o200k_base encoding, and the cost fields in the response added up to $0.0041.

One of the ten, with the content trimmed:

{
  "url": "https://kubernetes.io/docs/tasks/run-application/configure-pdb/",
  "status": 200,
  "error": null,
  "content": "Specifying a Disruption Budget for your Application | Kubernetes\n[** Edit this page](https://github.com/kubernetes/website/edit/main/content/en/docs/tasks/run-application/configure-pdb.md)\n ...",
  "duration_elasped_ms": 1963,
  "costs": { "total_cost": 0.0008062928166666666 }
}

The mix of results surprised me more than the speed did. Two of the ten results were on kubernetes.io, which publishes its documentation under CC BY 4.0. The other eight were vendor blogs and tutorial sites. They read well, and I had not read their terms, so I had no business putting them in a training file. Search made finding the text cheap, 4.5 seconds for ten pages with their content. Checking the right to train on each one is yours to do, site by site.

Search told me where the topic lives, and the licence decided what I crawled. The search guide has the other parameters.

Crawl: one site to markdown

I pointed the crawl at the concepts section of the Kubernetes documentation. It is public, its robots.txt does not disallow it, and the licence permits reuse with attribution. Spider reads robots.txt by default (respect_robots defaults to true), and I left it on.

The second command above is the whole request. It came back in 6.5 seconds with 60 pages, all status 200. This is one of them, trimmed:

{
  "url": "https://kubernetes.io/docs/concepts/workloads/controllers/ttlafterfinished/",
  "status": 200,
  "error": null,
  "content": "Automatic Cleanup for Finished Jobs | Kubernetes\n[** Edit this page](https://github.com/kubernetes/website/edit/main/content/en/docs/concepts/workloads/controllers/ttlafterfinished.md)\n ... # Automatic Cleanup for Finished Jobs\nA time-to-live mechanism to clean up old Jobs that have finished execution.\n ...",
  "duration_elasped_ms": 3117,
  "costs": { "total_cost": 0.0007767593166666668 },
  "metadata": { "description": "A time-to-live mechanism to clean up old Jobs that have finished execution.", "domain": "kubernetes.io" }
}

The 60 pages were 860,186 bytes of markdown, 202,123 tokens. The costs.total_cost fields add up to $0.048.

Rights and terms

You are responsible for the rights to the pages you collect. For this run that meant three checks: the pages are public, robots.txt allows them, and the licence permits reuse. Do the same for every site you add. And if you put a model anywhere in your pipeline, to filter, label or rewrite, read that provider’s terms on training with its output before you do. This is not legal advice.

What markdown saved, measured

A claim that this is an efficient way to collect training data needs a number, so I fetched the same 60 URLs again as raw HTML and counted.

I did not expect the first row. The median page is 486,606 bytes of HTML and 11,673 bytes of markdown. A documentation page that reads as two screens of prose ships half a megabyte of navigation, scripts and inline SVG around it. The markdown is about 37 times smaller in tokens than the HTML, and the cleaned text is about 57 times smaller.

The table measures one thing: what you move and tokenize when you store raw HTML and convert later, 7.4 million tokens to keep 130,707. One crawl call, 6.5 seconds, returned the markdown that skipped all of that. The token cost post has the same measurement across ten different kinds of page.

I counted tokens with tiktoken 0.14.0 and o200k_base because it runs offline and sizes a dataset well enough. Your base model’s tokenizer may count differently.

Cleaning, with before and after

Markdown from a documentation page is still not all documentation. Look at the trimmed response above. Before the first heading there is an edit link, a link that opens a GitHub issue, and a print link. After the last section there is a feedback form and a last-modified line. On that page the markdown is 5,989 bytes and the prose between the h1 and the feedback form is 3,011.

The script keeps the text from the h1 to the “What’s next” or “Feedback” heading, drops link targets and keeps the link text, and cuts each page at h2 and h3 headings. Then it drops sections under 40 words, removes exact duplicates by hash, and splits any section over 800 tokens at line breaks.

import hashlib, json, re
import tiktoken

enc = tiktoken.get_encoding("o200k_base")
tok = lambda s: len(enc.encode(s))
pages = json.load(open("crawl.json"))
MIN_WORDS, MAX_TOKENS = 40, 800

def strip_page(md):
    start = re.search(r"^# ", md, re.M)                     # drop everything above the h1
    md = md[start.start():] if start else md
    md = re.split(r"^## (?:Feedback|What's next)\b", md, flags=re.M)[0]
    md = re.sub(r"\[\]\(#[^)]*\)", "", md)                  # heading anchors
    md = re.sub(r"\[([^\]]*)\]\(<?[^)]*>?\)", r"\1", md)    # keep link text, drop target
    return md.strip()

def sections(md):
    title = md.splitlines()[0].lstrip("# ").strip()
    parts = re.split(r"^(#{2,3} .+)$", md, flags=re.M)
    yield title, title, parts[0].split("\n", 1)[-1]
    for head, body in zip(parts[1::2], parts[2::2]):
        yield title, head.lstrip("# ").strip(), body

seen, out = set(), []
for p in pages:
    for title, head, body in sections(strip_page(p["content"])):
        body = body.strip()
        if len(body.split()) < MIN_WORDS:
            continue
        key = hashlib.sha256(re.sub(r"\W+", " ", body.lower()).encode()).hexdigest()
        if key in seen:
            continue
        seen.add(key)
        pieces, buf = [], ""
        for line in body.split("\n"):
            if buf and tok(buf + "\n" + line) > MAX_TOKENS:
                pieces.append(buf)
                buf = line
            else:
                buf = (buf + "\n" + line) if buf else line
        pieces.append(buf)
        for i, text in enumerate(pieces):
            out.append(dict(id=f"{key[:12]}-{i}", url=p["url"], title=title,
                            heading=head, text=text.strip(), tokens=tok(text)))

with open("chunks.jsonl", "w") as f:
    for c in out:
        f.write(json.dumps(c) + "\n")

The two heading names in strip_page are specific to this site. Every site needs its own pair, and finding them took me ten minutes of reading three pages.

Between the second and third rows, the 40-word floor took the 538 sections down to 452, exact dedupe removed none, and splitting the long ones produced 480 chunks. Chunk size runs from 31 to 800 tokens with a median of 208.

A third of the crawled tokens were the site’s own chrome and link targets. The strip step takes them out before chunking, and it is the one step you write per site.

Exact dedupe removed nothing. One section of one well-edited site does not repeat itself. I kept the step because a crawl across versions or mirrors will.

The 40-word floor is the part I am least happy with. It dropped 86 sections. Ten were empty parent headings and 76 had real text. One of the 76 is the “Limitations” section of the static Pods page, 25 words that say a static Pod cannot refer to a ConfigMap or a Secret. That is a fact worth learning and my filter threw it away. A lower floor keeps it and also keeps a lot of one-line stubs. I left the floor at 40, and that section is what it cost.

Build the fine-tuning dataset file

Split by page. If two sections of one page land on opposite sides, the eval score tells you the model remembers that page. Hashing the URL sends every chunk of a page the same way and gives the same split on every rerun.

import hashlib, json

EVAL_SHARE = 0.15
chunks = [json.loads(l) for l in open("chunks.jsonl")]

def bucket(url):
    h = int(hashlib.sha256(url.encode()).hexdigest()[:8], 16)
    return "eval" if h / 0xFFFFFFFF < EVAL_SHARE else "train"

train, held = open("train.jsonl", "w"), open("eval_chunks.jsonl", "w")
for c in chunks:
    if bucket(c["url"]) == "eval":
        held.write(json.dumps(c) + "\n")
        continue
    prompt = c["title"] if c["heading"] == c["title"] else f'{c["title"]}: {c["heading"]}'
    train.write(json.dumps({
        "messages": [
            {"role": "user", "content": f"Kubernetes concepts. {prompt}"},
            {"role": "assistant", "content": c["text"]},
        ],
        "meta": {"id": c["id"], "source": c["url"], "license": "CC BY 4.0"},
    }) + "\n")

I asked for 15 percent and got 11 pages of 60, which is 18. A hash does that on small sets.

The training record, one per line of train.jsonl:

{
  "messages": [
    {"role": "user", "content": "Kubernetes concepts. Operator pattern: Operators in Kubernetes"},
    {"role": "assistant", "content": "Kubernetes is designed for automation. Out of the box, you get lots of\nbuilt-in automation from the core of Kubernetes. ..."}
  ],
  "meta": {
    "id": "b9a2fd48f0a9-0",
    "source": "https://kubernetes.io/docs/concepts/extend-kubernetes/operator/",
    "license": "CC BY 4.0"
  }
}

The Kubernetes authors wrote both sides of that record. The user turn is the page title and the section heading. The assistant turn is the section. meta.source is there because CC BY asks for attribution and because you will want to trace a bad answer back to a page.

Each line of eval_chunks.jsonl is a held-out section with its id, url, title, heading, text and tokens. The model does not see those 11 pages in training, so they are what you test it on.

What the run cost

Collection came to $0.052 for 396 training records, which is $0.13 per thousand records. That figure belongs to this site and this cleaning script. A site with shorter pages or a stricter filter will land somewhere else. The raw HTML fetch I ran for the comparison cost another $0.047 and is not part of the pipeline. The pricing guide explains what goes into a page’s cost.

What the dataset is ready for

train.jsonl uses the messages layout that most chat fine-tuning tools read, so it is ready to load into the trainer you already run, on your own stack. Remove meta if yours rejects unknown keys. eval_chunks.jsonl is the held-out text to measure the result against.

The training prompts are a template. A model tuned on “title: heading” prompts learns to recite sections. It does not learn to answer a question phrased the way a person asks it. For that you need questions people wrote, paired with these sections.

49 pages proves the pipeline. Raise limit on the crawl call to widen the set before you judge the model; the call and the scripts do not change.

Splitting by page stops a page from leaking across the split. It does not stop a concept from leaking. Pods come up on many of the 60 pages, on both sides of the split.

Exact-hash dedupe misses near duplicates. A crawl of two versions of one site needs a shingle or embedding pass.

The documentation changes. The dataset is a copy from 2026-11-06; the crawl call is the refresh, so rerun it before each retrain and the set is as current as the site.

Run it on your topic

Put your topic in the search call, read what comes back, and pick the sources you have the right to use. Point the crawl at each one, find its two boilerplate headings, and rerun the two scripts. Both calls need a key, and you can get one here.