Web scraping with JavaScript: a Node.js script that streams each page to disk
Web scraping with JavaScript in Node.js and the Spider client. One script scrapes a page, reads a field by CSS selector and streams a 25 page crawl to a file. Run against the API on November 17, 2026.
The Node.js script on this page scrapes one page as markdown, reads a field by CSS selector and streams a 25 page crawl into a file, one line per page. On November 17, 2026 we ran it through the Spider API against scrapethissite.com, a public site built for scraping practice. The two scrapes took 1.5 and 1.4 seconds. The crawl streamed 25 pages in 5.4 seconds and cost $0.0019.
The day before, we ran it against a local server that answers the way the API documents, with https://example.com as every URL. A local server let us send a 429 and a 402 on purpose. After a 429 the script still saved 25 of 25 pages with no retry code of its own. After a 402 the crawl call returned with zero pages and no error, and that second result is why the script counts what it saved.
A Node scraper written by hand needs fetch and an HTML parser for static pages, and a browser library in its own process for pages that build themselves with scripts. This script hands both jobs to the API. One field, request, picks the fetch, and the web scraping guide shows which kinds of page need a browser at all.
The script is 41 lines and needs one package
npm install @spider-cloud/spider-client@0.2.5
export SPIDER_API_KEY=your_keySave this as scrape.mjs and run node scrape.mjs.
import { Spider } from "@spider-cloud/spider-client";
import { appendFileSync } from "node:fs";
const spider = new Spider();
const [page] = await spider.scrapeUrl("https://www.scrapethissite.com/pages/simple/", {
return_format: "markdown",
request: "http",
});
console.log(page.status, page.content.slice(0, 200));
const fields = await spider.scrapeUrl("https://www.scrapethissite.com/pages/simple/", {
return_format: "markdown",
css_extraction_map: {
"/": [{ name: "title", selectors: ["h1"] }],
},
});
console.log(fields[0].css_extracted);
let saved = 0;
await spider.crawlUrl(
"https://www.scrapethissite.com/pages/",
{ limit: 25, return_format: "markdown" },
true,
(page) => {
try {
appendFileSync("pages.jsonl", JSON.stringify({ url: page.url, status: page.status, content: page.content }) + "\n");
saved += 1;
} catch (err) {
console.error("could not save", page.url, err);
}
},
);
if (saved === 0) {
throw new Error("The crawl streamed no pages. Check the key and your credits.");
}
console.log(`saved ${saved} pages`);The client calls the fetch built into Node, which is on without a flag from Node.js 18. We ran the script on Node.js 24.2.0 with version 0.2.5 of the client, the version on npm on November 17, 2026. The .mjs extension makes the file an ES module, where top level await works.
The script appends to pages.jsonl. Delete the file before a second run, or the new lines land after the old ones.
The copy we ran against the API had a few more lines. It printed the time of each call and saved page.costs.total_cost with each line, which is where the crawl cost above comes from. This post reports totals and selected output. The per URL records of the run are not published.
scrapeUrl sends a crawl with a limit of one, so it returns an array
In version 0.2.5 scrapeUrl(url, params) posts to the crawl route with limit: 1 and your parameters on top. The answer is a JSON array with one object per page, which is why the script destructures [page]. Each object carries url, status, content and a costs object whose total_cost field is what the page cost.
Check page.status before you trust content. In the run against the API, 24 of the 25 crawled pages had status 200 and one had status 400, a URL the practice site answers that way. The script saves the status with each line for that reason. As the API reference puts it, an attempt bills for the bandwidth and compute it used, whatever status the page returns. A failed page is not free.
The API reference describes the request values. "http" runs no JavaScript and is the fastest. "browser" renders the page first. Leave the field out and you get "smart", the default, which tries HTTP and moves to a browser only when the page needs it. The crawl in the script leaves it out.
The script asks for markdown because a person or a model usually reads the text next. On ten public pages measured on November 5, 2026, markdown used 15 times fewer tokens than raw HTML at the median, counted in the token cost post.
Fields come back under css_extracted, under the name you gave each one
css_extraction_map is keyed by URL path. Each value is a list of { name, selectors } pairs. The API reference says the key "/" matches every page, any other key matches only the page whose path is exactly that key, and a prefix such as "/products" currently matches nothing. Start with "/" unless you mean one exact page.
The values come back as arrays, because a selector can match more than once. Against the practice site the script printed { title: [ 'Countries of the World: A Simple Example 250 items' ] }. Take [0] when you want one string.
A streamed crawl calls your function once per page and resolves with nothing
Pass true as the third argument and a function as the fourth. The client sends Content-Type: application/jsonl, reads the answer as it arrives, splits it on newlines and calls your function with each parsed page. In the run against the API and in the local success test, the function ran 25 times for limit: 25 and a fresh pages.jsonl held 25 lines.
Two things follow from that design. Each page is written when it lands, so a crawl that stops partway leaves the pages that already arrived in the file. And the await resolves with no value. The pages exist only where your function put them, so the function is the place to save, count or queue them.
Without the third and fourth arguments, crawlUrl waits for the whole crawl and resolves with the full array. That suits 25 pages. For a crawl of thousands it means holding all of them in memory before the first one is written.
On a 429 the client waits for Retry-After and sends the request again
The client wraps every POST in a backoff with five attempts in all. When the status is 429 it reads Retry-After, uses 1 second if the header is missing, and waits that long before the next attempt. It also records the RateLimit-Limit, RateLimit-Remaining and RateLimit-Reset headers on each answer.
We had the local server answer the first request with a 429 and Retry-After: 1. The script ran to the end and saved 25 of 25 pages, with no retry code in it. Five attempts is the whole budget, so a run that hits the limit again and again will still fail.
After a streamed crawl, check how many pages you saved
Outside stream mode, a 402 throws Failed to post to crawl. Status code: 402. and you see it. In stream mode, version 0.2.5 reads the body only when the status is a success. On a 402 it returns, your function never runs, and nothing is thrown. The same line of the source covers a 401, which we did not test. So after a streamed crawl the count of saved pages is your status check.
The JSON lines reader also catches any error thrown inside your function and drops it. We checked that by passing a function that throws on every page. It ran twice for two lines and no error reached the caller. So the try and catch inside the function are the only place a failed write gets logged.
The last check in the script covers a crawl that saved nothing. When saved is still zero after the crawl, it throws, and in our 402 test that is how the failure showed up. Keep both habits in your own code. Catch inside the function, and count after the call.
What these runs do not show
The run against the API was one run, on one morning, on one small site that exists to be scraped. Its times are wall time on our machine, and another site will give other times and costs.
Every request in the local tests went to a server on the same machine, and no site was fetched. Those tests show what the client does with answers the API documents, a 429 with Retry-After and a 402. We did not see the API itself send either one. We ran one Node version and one client version.
To run the script against the API, create an API key and export it as SPIDER_API_KEY.
Frequently asked questions
Can I do web scraping with JavaScript without running a browser in Node?
Yes. The script on this page needs one package, @spider-cloud/spider-client 0.2.5, and starts no browser on your machine. The request field tells the API to use a plain HTTP fetch or a browser on its side. On November 17, 2026 we ran the script on Node.js 24.2.0 against the API. It streamed 25 pages from a public practice site into a file in 5.4 seconds, 24 with status 200 and one with status 400.
How do I stream a crawl in Node.js instead of waiting for the whole result?
Pass true and a function as the third and fourth arguments of crawlUrl. The client then asks for JSON lines and calls your function once for each page as its line arrives. In that mode crawlUrl resolves with no value, so save or count pages inside the function. In our run against the API on November 17, 2026 the function ran 25 times for a limit of 25.
What does the Node client do when it gets a 429?
Version 0.2.5 makes up to five attempts. On a 429 it waits the number of seconds in the Retry-After header, or 1 second when the header is missing, before the next attempt. In our test of November 16, 2026 a local test server answered the first request with a 429 and Retry-After: 1, and the script still saved 25 of 25 pages with no retry code of its own.
Why did my streamed crawl finish with no pages and no error?
Count the pages you saved and fail when the count is zero, as the script on this page does. In stream mode, version 0.2.5 reads the response body only when the status is a success, and it throws nothing otherwise. A 402 or a 401 ends the call with your function never run. An error thrown inside that function does not reach the caller either, so catch and log it there. On November 16, 2026 we tested the 402 case and a function that throws against a local test server. The 401 case is from reading the same line of the source.
Can I use the Node client without an API key?
No. new Spider() reads SPIDER_API_KEY from the environment and throws No API key provided when it is missing, in version 0.2.5. Keyless access is /scrape only. It is rate limited. On a 429, honor Retry-After. Get a key for anything sustained.
Sources
- Measured run, 2026-11-17, about 10:35 UTC: the script below on Node.js 24.2.0 with @spider-cloud/spider-client 0.2.5, sent once through the Spider API to www.scrapethissite.com. The copy we ran also printed the time of each call and saved each page's cost
- Local tests, 2026-11-16: the same script with example.com as every URL, on Node.js 24.2.0 with @spider-cloud/spider-client 0.2.5, against a local server that answers as the API documents. Four tests: success, one 429, a 402, and a function that throws on every page
- Spider JavaScript client source at commit 34a69f8: scrapeUrl, crawlUrl, the key check and the retry on 429
- Spider JavaScript client source at commit 34a69f8: the stream reader and the JSON lines reader that calls your function once per page
- Spider OpenAPI file, read 2026-11-16: the request values, css_extraction_map key matching, the css_extracted field, the keyless access note and the note on billing for failed pages
- Node.js documentation, the global fetch: no longer behind a flag from v18.0.0
- Node.js documentation, ECMAScript modules: the .mjs extension and top level await
- What a web page costs in tokens, measured on ten public pages, 2026-11-05
Keep reading
Built-in proxies for web scraping: median $0.0025 a complete page
Proxies for web scraping come built into every Spider request. On November 17, 2026, a complete category or listing page cost a median of $0.0025, proxy included.
Web scraping for AI agents: what each call adds to the context
Web scraping for AI agents, measured November 17, 2026. Tokens, seconds and cost for a search with page content at 3, 5 and 10 results, one scrape and one crawl.
Web scraping to markdown: a 63-page docs site in one call
Web scraping to markdown, measured on November 17, 2026. One crawl call returned a 63-page docs site as markdown in 3.5 seconds for $0.0138.
Run this on a page you care about
The playground sends the request this page describes and shows you the response. Keyless runs work without an account and are rate limited.