Skip to main content

Web scraping with a headless browser, without running one

How to scrape pages that build their text with JavaScript, using a browser on Spider's side, and how to tell which pages need one and what a render costs.

10 min read

You can scrape a page that builds itself with JavaScript without running a headless browser yourself. In a run on November 17, 2026, I fetched 10 such public pages through the Spider API, once plain and once rendered. The plain HTTP fetch got the HTML each server sent, and it held 1 to 6 words of text. The rendered fetch ran each page in a browser on Spider’s side and returned 217 to 4,533 words a page, for $0.000062 to $0.009056 a page and $0.0306 for all 10.

I picked these 10 from a longer list of such pages, keeping the ones whose rendered answer held the page’s text.

What does a rendered call look like?

It is one POST to /scrape, the web scraping API endpoint, with request set to browser. This call asks for the docsify Quick start as markdown:

curl -X POST https://api.spider.cloud/scrape \
  -H "Authorization: Bearer $SPIDER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://docsify.js.org/#/quickstart", "request": "browser", "return_format": "markdown", "cache": false}'

I sent it at 16:34 UTC. It took 4.0 seconds and cost $0.000301, and the content field held 437 words of markdown. It begins:

Quick start - docsify
# [Quick start](#/quickstart?id=quick-start)
It is recommended to install `docsify-cli` globally, which helps initializing and previewing the website locally.

In the measured run below, the plain request for this page got one word, docsify. The cache option is on by default, and on /scrape a cached answer skips the browser, so every call on this page sets cache to false. This call ran after the table’s run, so its time and cost differ from the docsify row.

What did rendering return on each page?

Each page came back with its text: a README, a CVE record, 40 companies, an API’s operations. Between them, the 10 pages gave the plain fetch 30 words and the rendered fetch 9,741. Each page got one plain and one rendered request on November 17, 2026, with the cache off. Seconds and cost are for the rendered request.

PageWords, plainWords, renderedWhat the rendered answer heldSecondsCost
Open VSX, Language Support for Java34,533The extension’s README20.9$0.000122
cve.org, CVE-2021-4422852,191The CVE record for the Log4j flaw22.7$0.007138
Y Combinator company directory6779The filters and the first 40 companies17.9$0.009056
docsify, Quick start1437The Quick start guide14.3$0.000084
regulations.gov, docket EPA-HQ-OAR-2021-03172391The docket’s title and summary12.5$0.000062
EMBL-EBI OLS, Experimental Factor Ontology4372The ontology’s version, description and related ontologies25.9$0.000122
Terraform Registry, hashicorp/aws2302The provider overview and its usage code34.8$0.008808
Ember Observer, ember-concurrency2264The addon’s score, review and versions13.1$0.003099
Swagger Petstore2255The API’s operations5.2$0.000825
Redoc demo3217The demo API’s operations14.7$0.001243

The median rendered page took 16.3 seconds and cost $0.001034.

Why does a plain fetch get almost no text from these pages?

Their servers send a shell. On these 10 pages, the HTML held 1 to 6 words of text and 2 to 12 script tags. With request set to http, Spider runs none of those scripts, so the answer is the shell. With browser, Spider runs them in a browser on its side before it reads the page.

The docsify Quick start prints the shell you write to start a docsify site. This is part of the body of its example index.html:

  <body class="loading">
    <div id="app"></div>

    <!-- Configuration -->
    <script>
      window.$docsify = {
        //...
      };
    </script>

    <!-- Docsify.js -->
    <script src="//cdn.jsdelivr.net/npm/docsify@5"></script>

The docsify site uses the same kind of shell. Its HTML holds that empty div, and its server sent the plain fetch one word, docsify. An HTTP request leaves out the part of an address after #, so the server saw a request for https://docsify.js.org/ and could not tell which page was wanted.

A browser keeps the #/quickstart part for the page’s own scripts. In a headless browser, a browser with no window, the docsify script reads it and fills the div with the Quick start. The rendered fetch at the top of this page got the filled page, 437 words, in one call. The call carried no wait option, and the answer held the text the script writes into that div, so the browser on Spider’s side read the page after the script had filled it. The web scraping guide lists the headless browser with the other ways to get data off a page.

What word count says a page needs rendering?

On my pages, single digits. Counted the way the script below counts, the HTML their servers sent held 1 to 6 words on the 10 pages in the table and 158 to 3,041 on six pages whose servers send their text. I set the cut-off at 120.

This script fetches each URL with request set to http and return_format set to raw, so the answer is the HTML as the server sent it. For a 200 answer, it counts the words outside script, style, noscript, template and svg tags and prints which request to use. For any other status, it prints the status.

import json, os, re, sys, urllib.request
from html.parser import HTMLParser

class VisibleText(HTMLParser):
    skip = {"script", "style", "noscript", "template", "svg"}

    def __init__(self):
        super().__init__()
        self.depth, self.words = 0, 0

    def handle_starttag(self, tag, attrs):
        if tag in self.skip:
            self.depth += 1

    def handle_endtag(self, tag):
        if tag in self.skip and self.depth:
            self.depth -= 1

    def handle_data(self, data):
        if not self.depth:
            self.words += len(re.findall(r"\w+", data))

def check(url):
    body = json.dumps({"url": url, "request": "http", "return_format": "raw", "cache": False})
    req = urllib.request.Request("https://api.spider.cloud/scrape", data=body.encode(), headers={
        "Authorization": "Bearer " + os.environ["SPIDER_API_KEY"],
        "Content-Type": "application/json"})
    page = json.load(urllib.request.urlopen(req))[0]
    text = VisibleText()
    text.feed(page.get("content") or "")
    return page.get("status"), text.words

for url in sys.argv[1:]:
    status, words = check(url)
    if status != 200:
        print("status", status, url)
    else:
        print(words, "browser" if words < 120 else "http", url)

Save it as needs_browser.py and pass the URLs as arguments. At 17:27 UTC it printed:

5 browser https://www.cve.org/CVERecord?id=CVE-2021-44228
1 browser https://docsify.js.org/#/quickstart
3041 http https://www.recreation.gov/camping/campgrounds/232447
2243 http https://hub.docker.com/_/nginx

I ran the count three times between 17:11 and 17:23 UTC, and every run gave each of these 16 pages the same count. Any cut-off from 7 to 158 would have given the same split, so 120 has room on both sides. A server that sends a shell on one fetch and its text on the next can land on either side, so if a plain answer comes back nearly empty, send that page again with browser. A run over the 16 pages cost about $0.0023, a median under $0.00002 a page. Plain fetches of two pages it marks http, recreation.gov and Docker Hub, returned 3,035 and 2,131 words of markdown.

What does a rendered fetch cost?

On my 10 pages, $0.000062 to $0.009056 a page at pay-as-you-go rates, with a median of $0.001034 and $0.0306 in all.

The pricing guide prices content transferred at $1 per GB and compute at $0.0001 a minute. A browser also downloads the scripts and data the page calls for, and those bytes count toward the bill.

In this run, plain and rendered answers carried the same cost lines, and bytes_transferred_cost was 96% of the rendered $0.0306. compute_cost came to $0.0003 for all 10 pages, and file_cost and transform_cost made up the rest. The YC directory, the Terraform Registry and the CVE record cost the most, $0.007138 to $0.009056 each. Nearly all of that was bytes transferred, roughly 7 to 9 MB a page at $1 per GB. So use the check to pick each page’s mode, then confirm that the answer holds the text you need. A page that a plain fetch can read adds no browser bytes to the bill.

The Unlimited plan bills by concurrency and not by request, from $40 a month for 2 requests in flight, so these per-request figures do not apply to it.

When do I need a browser session instead of one call?

When more text shows up only after your code acts on the page, as on the YC directory. A rendered call returned its first 40 companies, and more load as you scroll.

Spider Browser opens the session on Spider’s side, and you drive it from your code. Install the client:

npm install spider-browser

This script opens the directory and counts the distinct company links. Then it scrolls to the bottom five times. After each scroll it looks for new links up to 20 times, pausing 500 ms before each look.

import { SpiderBrowser } from "spider-browser"

const browser = new SpiderBrowser({ apiKey: process.env.SPIDER_API_KEY, logLevel: "warn" })
const sleep = (ms) => new Promise((done) => setTimeout(done, ms))
const companies = () => browser.page.evaluate(
  `new Set([...document.querySelectorAll('a[href^="/companies/"]')].map((a) => a.pathname)).size`
)

// Scroll to the bottom, then look for new rows up to 20 times, pausing 500 ms before each look.
async function scrollOnce(before) {
  await browser.page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
  for (let tries = 0; tries < 20; tries++) {
    await sleep(500)
    if ((await companies()) > before) {
      await sleep(500) // let the rest of the batch render
      return companies()
    }
  }
  return before
}

try {
  await browser.init()
  await browser.goto("https://www.ycombinator.com/companies")
  await browser.page.content() // waits longer if the page still looks unfinished
  let count = await companies()
  console.log("on load:", count)
  for (let i = 0; i < 5; i++) count = await scrollOnce(count)
  console.log("after 5 scrolls:", count)
} finally {
  await browser.close()
}

Save it as scroll.mjs and run node scroll.mjs with SPIDER_API_KEY set. Three runs, at 16:55, 16:56 and 17:26 UTC on Node.js 24.2.0 with spider-browser 0.3.0, printed the same two lines in 39 to 45 seconds:

on load: 40
after 5 scrolls: 120

A session bills $0.0001 per minute while it is open plus $1 per GB, with no monthly minimum. An account can hold up to 1,000 sessions open at once and start up to 100 a minute, and a session can stay open for 30 minutes. The Spider Browser docs cover the client in TypeScript, Python and Rust.

How I ran it

I picked these 10 from a longer list of public pages whose servers send almost no text. The table keeps the ones whose rendered answer held the page’s text, so it shows what a render returns on such a page and is not a success rate. Each site’s robots.txt allowed the page, or the site had none.

Every request in the table went to the Spider API’s /scrape endpoint between 16:12 and 16:23 UTC, with cache set to false, one at a time from my machine. The plain request used request: http and the rendered one request: chrome, an older name the API still accepts for browser. Both asked for markdown. The docsify call at the top used browser.

Words are runs of letters and digits in each answer’s text, leaving out markup, scripts, styles, images and link addresses. Seconds are wall time at my machine, from sending a request to reading the whole answer. Cost is each answer’s costs.total_cost.

What this run does not show

The table holds one fetch of each page each way, from one afternoon, so a fetch at another time can return a different answer. I picked the 10 pages and the 16 in the check, so neither set is a sample of the web, and the check’s split is not an accuracy rate. The 16 are pages whose word counts held steady across the three runs. Times come from my machine and will differ at yours. Costs are pay-as-you-go prices for these pages on that day. The three session runs show what the script prints, not a success rate for sessions. Pages behind a login are out of scope.

To run the calls on this page, create an API key and export it as SPIDER_API_KEY. Before that, you can paste one of your own URLs into the web scraper and read what a single request returns.

Frequently asked questions

Can I scrape a JavaScript page without running a headless browser?

Yes. Send the URL to the Spider API's /scrape endpoint with request set to browser. Spider runs the page in a browser on its side and returns the text after the page's scripts have run. In a run on November 17, 2026, 10 public pages that build their text with JavaScript returned 217 to 4,533 words each this way. The HTML their servers sent held 1 to 6 words.

How much does a rendered fetch cost?

In a run on November 17, 2026, 10 rendered pages cost $0.000062 to $0.009056 each at pay-as-you-go rates, $0.0306 in all. Bytes transferred were 96% of that, since a browser also loads the page's scripts and data. Plain and rendered answers carried the same cost lines. The Unlimited plan bills by concurrency instead.

How long does a rendered fetch take?

In the November 17, 2026 run, 5.2 to 34.8 seconds a page, with a median of 16.3, timed at the client. A word-count check on each page's HTML picks the pages that need rendering, so you render only those.

What word count tells me a page needs rendering?

Fetch the page with request set to http and return_format set to raw, then count the words outside script, style, noscript, template and svg tags. On 16 pages whose counts held steady over three runs on November 17, 2026, the pages that needed rendering had 1 to 6 words and the rest had 158 to 3,041. A cut-off of 120 split them the same way each time, at a median cost under $0.00002 a page.

When do I need a browser session instead of one request?

When more text appears only after your code acts on the page, such as a list that loads more as you scroll. In a Spider Browser session, the Y Combinator company directory showed 40 companies on load and 120 after five scrolls. A session bills $0.0001 per minute while it is open plus $1 per GB, and an account can hold up to 1,000 sessions open at once.

Sources

  1. Measured run, 2026-11-17, 16:12 to 16:23 UTC: 10 public pages that build their text with JavaScript, kept from a longer list because their rendered answer held the page's text. Each was sent once to the Spider API's /scrape endpoint with request http and once with request chrome, the older name for browser, asking for markdown with cache false. Words leave out markup, scripts, styles, images and link addresses, seconds are wall time at the client and cost is each answer's costs.total_cost. Fetched at 16:03 to 16:07 UTC with request http and return_format raw, the HTML of each of the 10 held 1 to 6 words outside script, style, noscript, template and svg tags and 2 to 12 script tags, and the docsify page's HTML held an empty div with the id app
  2. The docsify call, 2026-11-17, 16:34:45 UTC, request browser: 437 words in 4.0 seconds for $0.000301. The plain request in the measured run, at 16:21:55 UTC, returned the one word docsify. The docsify Quick start prints the index.html quoted on this page
  3. The check script, 2026-11-17: three runs between 17:11 and 17:23 UTC gave the same split of the 10 pages in the table and six pages whose servers send their text, at about $0.0023 for those 16 pages a run. The printed output is the run at 17:27 UTC. Plain fetches of recreation.gov and Docker Hub at 16:37 and 16:38 UTC returned 3,035 and 2,131 words
  4. Three session runs, 2026-11-17, at 16:55, 16:56 and 17:26 UTC: the scroll script on Node.js 24.2.0 with spider-browser 0.3.0 against the Y Combinator company directory. Each printed 40 on load and 120 after five scrolls, in 39 to 45 seconds
  5. Spider OpenAPI file, read 2026-11-17: the request values http and browser, chrome as an older name for browser, the cache default and return_format raw
  6. RFC 9110, HTTP Semantics, section 7.1: the target URI excludes the fragment, since fragment identifiers are reserved for client-side processing
  7. Spider Browser: session pricing, the open-session limits and the 30-minute session cap
  8. Spider pricing guide: the bandwidth and compute rates and the bytes a browser adds to the meter

Keep reading

How to scrape a website, step by step

How to scrape a website, from reading robots.txt to saving a crawl one page at a time, with the curl command for each step and what it returned.

Proxies for web scraping, built into every request

Proxies for web scraping come built into every Spider request and rotate on their own. What they cost, and how to pick the exit country or use your own.

Web scraping for AI agents: search, scrape or crawl

Web scraping for AI agents: when an agent should search, scrape or crawl, and how many tokens of its context window each call fills, counted on real pages.

Run this on a page you care about

The playground sends the request this page describes and shows you the response. Keyless runs work without an account and are rate limited.