Skip to main content

Web scraping with Python: one script, 25 pages, three runs

A web scraping script in Python's standard library, printed whole. On November 16, 2026 it returned 25 of 25 pages as markdown in each of three runs.

Jeff Mendez Updated 6 min read

This is web scraping with Python in one script that imports nothing outside the standard library. On November 16, 2026 we ran it three times against the same 25 pages on five documentation sites. Each run returned all 25 pages as markdown, 37,415 words in total, in 34.0 to 57.5 seconds, for less than a cent. The whole script is below, and you can paste it into a file and run it as is.

The script has no HTML parser and no CSS selectors. It sends each URL to the Spider scrape endpoint and gets the page back as markdown. It is the short form of the request our guide to web scraping walks through step by step.

The script

Save it as scrape_docs.py, set your key and run it with python3 scrape_docs.py.

#!/usr/bin/env python3
"""Fetch documentation pages as markdown through the Spider API, one request per page.

    export SPIDER_API_KEY=...        # from https://spider.cloud/api-keys
    python3 scrape_docs.py

Python 3.8 or later, standard library only. Prints one line per page and a total.
"""
import hashlib
import json
import os
import time
import urllib.error
import urllib.request

URLS = [
    "https://fastapi.tiangolo.com/",
    "https://fastapi.tiangolo.com/features/",
    "https://fastapi.tiangolo.com/learn/",
    "https://fastapi.tiangolo.com/python-types/",
    "https://fastapi.tiangolo.com/async/",
    "https://docs.astro.build/ar/astro-courses/",
    "https://docs.astro.build/ar/basics/astro-components/",
    "https://docs.astro.build/ar/basics/astro-pages/",
    "https://docs.astro.build/ar/basics/layouts/",
    "https://docs.astro.build/ar/basics/project-structure/",
    "https://docs.docker.com/security/security-announcements/",
    "https://docs.docker.com/get-started/tutorials/run-an-app/",
    "https://docs.docker.com/guides/dhi-vex-walkthrough/",
    "https://docs.docker.com/reference/api/ai-governance-audit-logs/latest/",
    "https://docs.docker.com/reference/api/ai-governance/latest/",
    "https://pydantic.dev/docs/",
    "https://pydantic.dev/docs/ai/api/models/anthropic/",
    "https://pydantic.dev/docs/ai/api/models/base/",
    "https://pydantic.dev/docs/ai/api/models/batch/",
    "https://pydantic.dev/docs/ai/api/models/bedrock_mantle/",
    "https://vite.dev/acknowledgements",
    "https://vite.dev/blog",
    "https://vite.dev/blog/announcing-vite2",
    "https://vite.dev/blog/announcing-vite3",
    "https://vite.dev/blog/announcing-vite4-3",
]
BODY = {"request": "http", "return_format": "markdown", "cache": False}
KEY = os.environ["SPIDER_API_KEY"]


def scrape(url):
    req = urllib.request.Request(
        "https://api.spider.cloud/scrape",
        data=json.dumps(dict(BODY, url=url)).encode(),
        headers={"Authorization": "Bearer " + KEY, "Content-Type": "application/json"},
    )
    try:
        with urllib.request.urlopen(req, timeout=120) as resp:
            pages = json.loads(resp.read())
    except (urllib.error.URLError, TimeoutError, ValueError):
        return None
    return pages[0] if isinstance(pages, list) and pages else None


print("status  words  cost_usd  seconds  sha256_12  url")
start = time.time()
returned = words = 0
cost = 0.0
for url in URLS:
    t0 = time.time()
    page = scrape(url)
    secs = time.time() - t0
    if page is None:
        print(f"ERR  0  0  {secs:.2f}  -  {url}")
        continue
    text = page.get("content") or ""
    price = (page.get("costs") or {}).get("total_cost") or 0
    n = len(text.split())
    returned += page.get("status") == 200 and bool(text)
    words += n
    cost += price
    digest = hashlib.sha256(text.encode()).hexdigest()[:12]
    print(f"{page.get('status')}  {n}  {price:.8f}  {secs:.2f}  {digest}  {url}")
print(f"pages returned {returned} of {len(URLS)}, wall {time.time() - start:.2f} s, words {words}, cost ${cost:.6f}")

The request body has three settings, all from the API reference. request: "http" fetches the page without running JavaScript, which is enough for static documentation. return_format: "markdown" returns the page as markdown in the content field. cache: false makes every run fetch the page again, so the three runs below are three real fetches and not one fetch read back twice.

The scrape endpoint answers with a one element array. The script reads status, content and costs.total_cost from that element and prints the first 12 characters of a SHA-256 of the markdown, so two runs can be compared line by line.

Which pages, and why these

The 25 URLs are the first five entries in the sitemap of each of five documentation sites: FastAPI, Astro, Docker, Pydantic and Vite. We wrote the list down before the first request and did not change it after. That is why five of the Astro URLs are under /ar/, the Arabic edition, which comes first in that sitemap.

Before the first run we read each host’s robots.txt through the API. A URL would have come off the list if its host’s robots.txt disallowed the path or asked for a crawl delay over five seconds, or if the request for the file failed. None did. FastAPI and Vite serve no robots.txt, so those two hosts state no rules.

Three runs on the same 25 pages

We ran the script unchanged with python3 -I scrape_docs.py on Python 3.14.7, three times, about 16 and 30 minutes apart.

RunStarted (UTC)Pages returnedWall timeMedian per pageSlowest pageWordsCost
116:12:2225 of 2557.5 s1.69 s8.61 s37,415$0.006896
216:28:2025 of 2554.0 s1.44 s6.71 s37,415$0.006889
316:58:5525 of 2534.0 s1.24 s2.56 s37,415$0.006863

Run of 2026-11-16. Wall time runs from the first request sent to the last answer read. Cost is the sum of costs.total_cost over the 25 answers on the pay as you go meter.

Every page answered with status 200 and markdown in all three runs. The word total came out at 37,415 every time.

The wall time is the number that moved, from 57.5 seconds down to 34.0. The median page fell from 1.69 seconds to 1.24 and the slowest from 8.61 to 2.56. A script that waits for each answer before it sends the next pays for every slow one in full. We would read the three wall times as a range and not as the speed of anything. If 25 pages in under a minute is too slow for your job, send the requests in parallel with concurrent.futures. We did not time that here.

Cost moved by less than four thousandths of a cent between runs, about $0.00028 a page. These are plain fetches of static pages. A page that only renders in a browser needs request: "browser" and costs more.

We ran the script once more the next morning, at 10:37 UTC on November 17. It returned 25 of 25 pages in 33.8 seconds for $0.006884. The word total was 37,465, which is 50 more than the day before, so expect your own hashes to differ from ours.

The same run with the official client

We ran the script once more with the Python client, spider-client 0.1.93 from PyPI, in a fresh virtual environment. The loop, the counting and the hashing stayed the same. This is the request for one page:

from spider import Spider

app = Spider()  # reads SPIDER_API_KEY from the environment
BODY = {"request": "http", "return_format": "markdown", "cache": False}

pages = app.scrape_url("https://fastapi.tiangolo.com/", params=BODY)
page = pages[0]
print(page["status"], len(page["content"].split()), page["costs"]["total_cost"])

The client run started at 16:13:21 UTC on November 16, between the first and second script runs, and returned 25 of 25 pages. Its markdown matched the first urllib run on all 25 by hash, 37,415 words, and it cost $0.006874.

One difference is worth knowing before you pick. In version 0.1.93, as its source reads, the client retries a failed call up to five times without being asked. The urllib script makes one attempt per URL. It prints ERR when the request or its JSON fails and the page’s own status when the target fails, so a retry is yours to add.

What this run does not show

The 25 pages are static documentation on five sites, fetched one at a time from one machine on one afternoon and the morning after. Nothing here measures concurrency, a site that needs a browser, or a page behind a login. We ran the client once, so its timing is a single sample and we do not compare it with the script’s.

To run the script yourself, create an API key and export it as SPIDER_API_KEY.

Frequently asked questions

Can I scrape web pages in Python without installing anything?

Yes. The script on this page imports nothing outside Python's standard library and runs on Python 3.8 or later. It sends each URL to an API that fetches the page and returns markdown, so there is no HTML parser to install. On November 16, 2026 it returned 25 of 25 pages from five documentation sites in each of three runs.

How long does a Python scraper take for 25 pages?

Our script sends one request at a time. Its three runs on the same 25 pages took 57.5, 54.0 and 34.0 seconds on November 16, 2026. The median page took 1.69 seconds in the first run and 1.24 in the third, and the slowest page took 8.61 and 2.56. Send requests in parallel if the total matters.

Should I use the Spider Python client or plain urllib?

In our run both returned the same pages. We ran the script once more with spider-client 0.1.93 from PyPI, and it returned the same markdown as the first urllib run on 25 of 25 pages. The client retries a failed call up to five times without being asked. The urllib version has no dependency and shows every field it sends.

What does it cost to scrape 25 pages this way?

Each run of 25 pages cost between $0.006863 and $0.006896 on the pay as you go meter, read from the costs.total_cost field of each answer. These were plain HTTP fetches of static pages. A page that needs a browser costs more, as our run on how web scraping works measured.

Sources

  1. Measured run, 2026-11-16, 16:12 to 16:59 UTC: the script below run three times and the client version once, through the Spider API scrape endpoint. One more run of the script on 2026-11-17 at 10:37 UTC
  2. Spider OpenAPI file, the request, return_format and cache parameters and the costs field of a scrape answer
  3. spider-client on PyPI, the official Python client, version 0.1.93 in this run
  4. Python documentation for urllib.request, the only HTTP module the script uses
  5. How web scraping works: the cost of a plain fetch and of a rendered fetch on the same pages, measured 2026-11-16

Keep reading

Built-in proxies for web scraping: median $0.0025 a complete page

Proxies for web scraping come built into every Spider request. On November 17, 2026, a complete category or listing page cost a median of $0.0025, proxy included.

Web scraping for AI agents: what each call adds to the context

Web scraping for AI agents, measured November 17, 2026. Tokens, seconds and cost for a search with page content at 3, 5 and 10 results, one scrape and one crawl.

Web scraping to markdown: a 63-page docs site in one call

Web scraping to markdown, measured on November 17, 2026. One crawl call returned a 63-page docs site as markdown in 3.5 seconds for $0.0138.

Run this on a page you care about

The playground sends the request this page describes and shows you the response. Keyless runs work without an account and are rate limited.