How to scrape a website: six steps, each one a request we sent
How to scrape a website, step by step for beginners. We sent every step to a public practice site on November 17, 2026 and print what came back, with the time and cost of each request sent with a key.
To scrape a website you send a request for a page, turn what comes back into text or named fields, and repeat that for the pages you need. Six steps cover it. On November 17, 2026 I sent each step below to scrapethissite.com, a public site built for scraping practice, and wrote down what came back. It took eight requests through the Spider API. The seven sent with a key cost $0.0045 together, and step three needs no key.
The web scraping guide explains why each step works. This page is the order to do them in, with a command for each, written for someone who has never scraped a page. Every step ends with the decision it settles. In this run, step two is where the default setting returned an empty table.
Step 1. robots.txt tells you which paths the site asks crawlers to skip
curl -s https://www.scrapethissite.com/robots.txtThe practice site’s file is three lines:
User-agent: *
Disallow: /lessons/
Disallow: /faq/The User-agent: * group is for every crawler the file does not name in a group of its own. Each Disallow line asks those crawlers to skip paths that start with its value, so Disallow: / would cover the whole site. The format is RFC 9309. Some files also carry a Crawl-delay line that asks for a pause between requests. It is not part of the RFC.
You can fetch the file straight from the site, as above, with no key. I fetched it through the API like every other request here, with a /scrape request for that URL and "return_format": "raw". It took 0.86 seconds and cost $0.000002.
The decision is whether to go on. Here the pages under /pages/ are open, so I went on. If the path you want is disallowed for all agents, pick another source. Our robots.txt survey of top sites shows what large sites ask for.
Step 2. The page source tells you whether to try a plain fetch or a browser
Open the page in your browser and press Ctrl+U, or Cmd+Option+U on a Mac, to see the HTML the server sent. Search it for a sentence you can see on the page.
If the sentence is there, a plain HTTP fetch will usually get your text. If it is missing, a script most likely adds it after the page loads, and a browser that runs the script is the thing to try. The decision is the value of one field, "request": "http" or "request": "browser".
The practice site has a page of Oscar winning films that loads its table with a script. I asked for it three ways. This is the third, and it needs a key in SPIDER_API_KEY, as steps four to six do.
curl -X POST https://api.spider.cloud/scrape \
-H "Authorization: Bearer $SPIDER_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://www.scrapethissite.com/pages/ajax-javascript/#2015", "return_format": "markdown", "request": "browser"}'request | Films in the answer | Seconds | Cost |
|---|---|---|---|
"http" | 0 of 16 | 1.11 | $0.000033 |
left out, so "smart" | 0 of 16 | 1.09 | $0.000033 |
"browser" | 16 of 16 | 4.38 | $0.000359 |
Run of 2026-11-17, one request each. Seconds are wall time at the client.
The plain fetch returned the page heading and the table header with no rows under it. The browser returned all 16 films, took about four times as long and cost about eleven times as much. All three answers had status 200, so the status tells you nothing here.
The middle row is why I do this check by hand. "smart" is the default. The API reference says it tries HTTP and moves to a browser only when the page needs it. On this page it returned the same empty table as the plain fetch. That is one page and one request, and the plain fetch against browser measurement has numbers for more pages by kind. When your own check says the text is missing from the source, set "browser" yourself.
Step 3. One request with no key returns the page as markdown
curl -X POST https://api.spider.cloud/scrape \
-H "Content-Type: application/json" \
-d '{"url": "https://www.scrapethissite.com/pages/simple/", "return_format": "markdown", "request": "http"}'The answer is a JSON array with one object per page. content holds the markdown, status the HTTP status the site returned, and costs.total_cost what the request cost. This page lists 250 countries, and the request came back with status 200 and 22,492 characters of markdown. Read the markdown and check that a sentence you can see on the page is in it.
Keyless access is /scrape only. It is rate limited. On a 429, honor Retry-After. Get a key for anything sustained.
Markdown is a good default when a person or a model reads the text next. On ten public pages measured on November 5, 2026, markdown used 15 times fewer tokens than raw HTML at the median, as the token cost post shows. Ask for "raw" when you need the HTML itself.
Step 4. Name the fields you want, and they come back under css_extracted
When you want a name or a number instead of the whole page, give each field a name and a CSS selector. To find a selector, right click the element in your browser, choose Inspect, and read its tag and class.
curl -X POST https://api.spider.cloud/scrape \
-H "Authorization: Bearer $SPIDER_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://www.scrapethissite.com/pages/simple/",
"return_format": "markdown",
"request": "http",
"css_extraction_map": {
"/": [
{ "name": "country", "selectors": [".country-name"] },
{ "name": "capital", "selectors": [".country-capital"] },
{ "name": "population", "selectors": [".country-population"] }
]
}
}'Each name comes back as a list under css_extracted, because a selector can match more than once. This request took 1.55 seconds and cost $0.000318. Each of the three lists held 250 values, and the first of each was Andorra, Andorra la Vella and 84000.
I keyed the map by "/", which the API reference says matches every page. It also says any other key matches only the page whose path is exactly that key, and that a prefix such as "/products" currently matches nothing. Start with "/".
The decision here is whether you need fields at all. If the next step is a person reading or a model answering questions, the markdown from step three is enough.
Step 5. A crawl follows the links from one page, up to the limit you set
curl -X POST https://api.spider.cloud/crawl \
-H "Authorization: Bearer $SPIDER_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://www.scrapethissite.com/pages/", "limit": 25, "return_format": "markdown"}'limit is the most pages the crawl returns. Set it on every crawl. Start at 25, read what came back, then raise it. depth caps how many links deep the crawl goes from the first page, 25 by default.
This crawl returned 25 pages in 5.82 seconds for $0.0019. Read the statuses before the content. 24 pages had status 200. One had status 400, a URL the practice site answers that way. As the API reference puts it, an attempt bills for the bandwidth and compute it used, whatever status the page returns. A failed page is not free.
The site’s menu links to /lessons/ and /faq/, the two paths its robots.txt disallows. None of the 25 pages was under either one. That is what respect_robots, true by default, should do. With a limit of 25 I cannot show the crawl would have reached them otherwise.
Before you raise the limit, know the bill. This crawl works out to about $0.08 per 1,000 pages. On day 1 of a seven-day benchmark, November 6, 2026, documentation pages cost $0.27 per 1,000 and articles $0.33 on the pay as you go meter. Expect your own site to land somewhere else, and read costs.total_cost on a small crawl first.
Step 6. Ask for JSON lines and each page is written as it arrives
curl -X POST https://api.spider.cloud/crawl \
-H "Authorization: Bearer $SPIDER_API_KEY" \
-H "Content-Type: application/jsonl" \
-d '{"url": "https://www.scrapethissite.com/pages/", "limit": 25, "return_format": "markdown"}' \
-o pages.jsonlThe body is the same JSON as in step five. The Content-Type: application/jsonl header is how the API reference has you ask for JSON lines. The answer is one JSON object per line, one line per page, and the reference says it streams as pages finish. My file ended with 25 lines after 8.12 seconds, and the crawl cost $0.0019, the same as step five to four decimal places.
If the lines arrive as pages finish, a crawl that stops partway leaves you the lines already written. I did not interrupt this one to test that. Each line has the same url, status and content fields as the answer in step three, so any language can read the file one line at a time.
One crawl is one request to the API. If a script of yours sends many requests and gets a 429, wait the number of seconds in the Retry-After header, then send the request again.
What this run does and does not show
I sent each request once, on one morning, to one small site that exists to be scraped. Times are wall time at my machine and will differ at yours. I report no time or cost for the keyless request in step three. This page prints the commands and a summary of each answer. The stored responses are not published. One request to one script-built page says little about how often the default gets such pages right.
Pages behind a login, forms, infinite scroll and clicks are out of scope. So is application code. Every step is a curl command, so you can see each request whole.
To run steps two and four to six, create an API key and export it as SPIDER_API_KEY.
Frequently asked questions
How do I scrape a website step by step?
Six steps. Read the site's robots.txt. Open the page source and look for the text you want, which tells you whether to try a plain fetch or a browser. Send one scrape request and read the markdown. Name the fields you want by CSS selector. Crawl the site with a page limit. Save each page as it arrives. Step three needs no API key.
Can I scrape a website without an API key?
Yes, one page at a time. Keyless access is /scrape only. It is rate limited. On a 429, honor Retry-After. Get a key for anything sustained. The curl in step three on this page sends that keyless request, and on November 17, 2026 it returned status 200 and 22,492 characters of markdown.
How much does it cost to scrape a website?
It depends on the page. On November 17, 2026, the 25 page crawl on this page cost $0.0019 and the seven requests sent with a key cost $0.0045 together. On day 1 of a seven-day benchmark, November 6, 2026, documentation pages cost $0.27 per 1,000 and articles $0.33 on the pay as you go meter.
How do I know if a page needs a browser to scrape?
Open the page source and search for a sentence you can see on the page. If it is missing, a script most likely adds it after the page loads. On the practice page in step two, a plain fetch on November 17, 2026 returned the table header and none of the 16 films. The same request with request set to browser returned all 16.
Does the crawler follow robots.txt?
The respect_robots option of the Spider API defaults to true in the OpenAPI file as of November 16, 2026. In the crawl on this page, the practice site's menu links to /lessons/ and /faq/, its robots.txt disallows both, and none of the 25 pages returned was under either path.
Sources
- Measured run, 2026-11-17, 10:31:50 to 10:32:47 UTC: eight requests through the Spider API to www.scrapethissite.com, one each. The robots.txt fetch, three fetches of the films page, the keyless scrape, the fields scrape and the two crawls
- Spider OpenAPI file, read 2026-11-16: the request values, respect_robots default, css_extraction_map key matching, limit, depth, the application/jsonl content type, the keyless access note and the note on billing for failed pages
- Web scraping cost per 1,000 pages, by kind of page, day 1 measured 2026-11-06
- What a web page costs in tokens, measured on ten public pages, 2026-11-05
- How web scraping works: what a plain fetch gets and misses, measured 2026-11-16
- RFC 9309, the Robots Exclusion Protocol
Keep reading
Built-in proxies for web scraping: median $0.0025 a complete page
Proxies for web scraping come built into every Spider request. On November 17, 2026, a complete category or listing page cost a median of $0.0025, proxy included.
Web scraping for AI agents: what each call adds to the context
Web scraping for AI agents, measured November 17, 2026. Tokens, seconds and cost for a search with page content at 3, 5 and 10 results, one scrape and one crawl.
Web scraping to markdown: a 63-page docs site in one call
Web scraping to markdown, measured on November 17, 2026. One crawl call returned a 63-page docs site as markdown in 3.5 seconds for $0.0138.
Run this on a page you care about
The playground sends the request this page describes and shows you the response. Keyless runs work without an account and are rate limited.