Free web scraper: three ways to scrape a page without paying
Scrape a page for free three ways: in the browser with no key, from curl with no key, or with the MIT-licensed spider CLI on your own machine.
You can scrape a web page for free in three ways, and each one makes you pay with something other than money. The tool on our web scraper page costs you nothing but a click and gives you one page at a time. A keyless request from curl costs you a terminal and a rate limit. The open source CLI costs you an install, and after that your own machine and your own IP address do all the work.
This guide sets the three side by side, with the exact command for each. We sent the keyless request on November 17, 2026 and printed its answer below. The CLI flags were read from the CLI’s source the day before. We did not run the CLI commands for this page.
The three free paths at a glance
| In the browser | curl, no key | spider CLI | |
|---|---|---|---|
| Install | nothing | curl | Rust and cargo install spider_cli |
| Account or key | no | no | no |
| Pages per request | one | one | as many as --limit allows |
| Crawls a site | no | no, /crawl returns 402 | yes |
| Who fetches the page | the Spider API | the Spider API | your machine |
| robots.txt by default | followed | followed | ignored until you pass -r |
| Limit | rate limited | rate limited | your machine and your IP |
Read the last two rows first. The first two paths both run a keyless scrape through the Spider API, so the same rules apply to both. The CLI runs the open source crawler on your own computer, and it has its own defaults.
In the browser
Open the web scraper, paste a URL and run it. The page runs one keyless scrape and prints the markdown and the JSON that came back. You install nothing and sign up for nothing.
That is the right tool for checking whether a page can be scraped at all, or for reading what a scraper sees before you write code. It is the wrong tool for a list of URLs, because you paste them one at a time.
From curl, with no key
A keyless scrape also works from any terminal. Leave out the Authorization header:
curl -s https://api.spider.cloud/scrape \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com", "return_format": "markdown"}'We sent that request once, on November 17, 2026 at 10:29 UTC. It answered 200 in 0.96 seconds at the client. This is the answer, with the costs object left out:
[
{
"content": "Example Domain\nThis domain is for use in documentation examples without needing permission. This is not a service; avoid relying on it for testing and monitoring purposes.",
"duration_elasped_ms": 537,
"error": null,
"status": 200,
"url": "https://example.com/"
}
]The answer is a JSON array with one object per page. The page text is in content, 27 words here. The page’s own HTTP status is in status, and url is the address that was fetched. duration_elasped_ms is spelled that way in the API. Pipe the answer through jq -r '.[0].content' to get the markdown alone.
That is one request for one small page, so read the 0.96 seconds as an example and not as a speed claim. To check it, run the command and compare.
These are the rules for this path. Keyless access is /scrape only. It is rate limited. On a 429, honor Retry-After. Get a key for anything sustained.
Without a key, every route other than /scrape returns 402, except /unlimited/*, which returns 401. So search and crawl are not part of the free path. A loop of keyless calls over a long list will meet the rate limit, and the polite answer to a 429 is to wait the seconds in Retry-After before the next call.
With the open source spider CLI
The spider crawler is open source under the MIT license, and its command line tool installs with Cargo:
cargo install spider_cliScrape one page as markdown:
spider -r -u https://example.com --limit 1 scrapeCrawl up to 50 pages of a site and save the output:
spider -r -u https://example.com --limit 50 scrape > pages.jsonBoth commands pass -r, and yours should too. In the CLI, --respect-robots-txt is off unless you pass it, so a bare spider -u command reads no robots.txt at all. The hosted API is the reverse: its respect_robots parameter defaults to true. If you move a job from one to the other, check this flag first. Our survey of 67 robots.txt files shows how often top sites ask unnamed crawlers to stay out.
The CLI prints one indented JSON object per page, one after another, with the page’s url, content and links. Read the file with jq, which takes a stream of objects. content is markdown by default. --return-format switches it to raw, commonmark, text or xml, and scrape --output-html puts the raw HTML there. That is how the 2.54.0 source reads. -D sets a delay between requests in milliseconds, which is the polite choice when a site sets a crawl delay.
With no Spider key set, nothing here is metered, because no request reaches our servers. The CLI reaches the Spider API only when it finds a Spider key, from --spider-cloud-key, the SPIDER_CLOUD_API_KEY variable or a stored login. Leave all three unset and every fetch comes from your own network.
You carry the cost instead. Pages that build their text with JavaScript need the CLI’s browser mode, --headless. The default build includes it, and it needs a Chrome on the machine to drive. Every request leaves from your IP address, so a site that limits by address limits you.
When to stop using the free path
The free path runs out when you need many pages or search and crawl from the API, or a browser for every page that you do not want to run yourself. Whether paying then costs less than your own machine depends on your pages. Our guide to web scraping walks through what a plain fetch gets and what a render adds, which is usually the deciding question.
The paid path is metered. Pay as you go at $1 per GB of bandwidth plus $0.0001 per CPU minute. No subscription. Top up from $5. An attempt bills for the bandwidth and compute it used, whatever status the page returns. A failed page is not free.
Paste one URL into the web scraper and read what comes back before you choose.
Frequently asked questions
Is there a free web scraper that needs no sign-up?
Yes. The tool on the Spider web scraper page runs a scrape from your browser with no key, and a keyless scrape works from curl too. Keyless access is /scrape only. It is rate limited. On a 429, honor Retry-After. Get a key for anything sustained. The open source spider CLI needs no account at all, because it runs on your machine.
Can I crawl a whole site for free?
Not through the keyless API. Without a key, every route other than /scrape returns 402, except /unlimited/*, which returns 401, so /crawl needs a key. The open source spider CLI crawls a whole site from your own machine, and its --limit flag caps the number of pages.
Does the free CLI follow robots.txt?
Only when you ask. In the spider CLI the -r flag, --respect-robots-txt, is off unless you pass it. The hosted API works the other way, and its respect_robots parameter is true by default. Both scrape commands on this page pass -r.
What formats does a free scrape return?
The keyless curl request on this page sets return_format to markdown and gets a JSON array back, with the page text in content. On November 17, 2026 it returned the 27 words of example.com in 0.96 seconds. The CLI's scrape command prints one JSON object per page with markdown in content, and --return-format switches that to raw, commonmark, text or xml.
When does the free path run out?
When you need more than one page at a time from the API, or when the pages need a browser and you do not want to run one. Pay as you go at $1 per GB of bandwidth plus $0.0001 per CPU minute. No subscription. Top up from $5.
Sources
- Measured run, 2026-11-17, 10:29 UTC: the keyless curl request printed on this page, sent once through the Spider API
- Spider OpenAPI file, the keyless note on /scrape, the return_format and respect_robots parameters, and the 402 and 401 answers without a key, read 2026-11-16
- spider_cli on crates.io, MIT license; the --respect-robots-txt, --limit, --return-format and --headless flags, the default chrome feature, and the scrape command and its JSON output, read from the 2.54.0 source on 2026-11-16
- spider-rs/spider on GitHub, the crawler the CLI is built on
- Spider pricing, the pay as you go meter
Keep reading
Command line web scraping: 100 pages in about 3 seconds
Three Spider CLI runs each crawled 100 FastAPI docs pages to markdown in about 3 seconds. Install it, add a key, then pipe the JSON to jq or write a WARC file.
Open source web scraping: why MIT license matters
How MIT, Apache 2.0, AGPL and BSL affect your right to build commercial products on scraping tools, and why Spider picked MIT.
How to scrape a website, step by step
How to scrape a website, from reading robots.txt to saving a crawl one page at a time, with the curl command for each step and what it returned.
Run this on a page you care about
The playground sends the request this page describes and shows you the response. Keyless runs work without an account and are rate limited.