What is a web scraping API? One request, read field by field
What a web scraping API does between your request and its answer, why a 200 call can carry a 404 page, and five things to check in any scraping API.
A web scraping API is a service that fetches web pages for you. You send it a URL and a few options over HTTP. It downloads the page and can run a browser when the page builds its text with JavaScript. It converts the result into a format such as markdown or plain text and sends it back. In Spider’s API the answer is JSON by default, and it carries the page’s status and what the request cost. You never run the crawler or the browser yourself.
A scraping library is the other way to do it. Requests and Beautiful Soup in Python, or the open source spider crate in Rust, run inside your own program, on your machine, from your IP address. You write the code that fetches each page and picks out what you want.
The examples below call our scrape endpoint, and one of them is a request we sent and read back field by field.
What happens between your request and the answer
A scraping API that picks the fetch method for you, as Spider’s default smart mode does, works through five steps for each page.
- Fetch the page over plain HTTP. This is the fastest and cheapest request. In our run of November 16, 2026 it returned at least 90% of a browser’s words on 26 of 29 documentation, news and store product pages.
- Decide whether that was enough. A page that ships an empty shell and fills it with JavaScript needs a second pass.
- Render the page in a browser when it needs one, wait for its scripts, and take the finished HTML.
- Convert the HTML into the format you asked for. Markdown keeps headings and links and drops the tags and scripts.
- Answer with the content, the page’s own status, any error, and the cost of the attempt.
In Spider’s API, steps 1 to 3 are one parameter. request set to http fetches without JavaScript, browser renders the page, and smart, the default, tries HTTP and moves to a browser only when the page needs it. Step 4 is return_format. Step 5 is the response.
curl -s https://api.spider.cloud/scrape \
-H "Authorization: Bearer $SPIDER_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com", "request": "smart", "return_format": "markdown"}'The answer is a JSON array with one object per page:
| Field | What it holds |
|---|---|
url | The address that was fetched |
content | The page in the return_format you asked for |
status | The page’s own HTTP status |
error | Why the page failed, or null |
costs | What the attempt billed, or null when nothing was billed |
duration_elasped_ms | Time spent on the page, in milliseconds. The key is misspelled in the API, and this is the spelling to read |
Read the status row twice. A scraping API has two statuses. The HTTP status of the call says whether the API did its job. The status inside each object is what the site answered. A 200 from the API can carry a page with status 404, so check both.
One request, two statuses
We tried it on November 17, 2026 at 10:29 UTC. The request was the one above with no Authorization header and with an address on our own site that does not exist, https://spider.cloud/this-page-does-not-exist.
| What we read | Value |
|---|---|
| HTTP status of the call | 200 |
| Time at the client | 1.5 seconds |
status in the page object | 404 |
error | null |
content | 100 words, our “This page does not exist” page as markdown |
costs.total_cost | $0.00007 |
error was null and content was not empty, because the site did answer, with its error page. A script that tests only the call, or only for empty content, would have stored our 404 page as if it were the article. And the attempt had a cost. The figure is for this one small page, and cost varies by site. Test status on every object before you keep its content.
This is one request on one site. A site that answers a missing address with a 200 and an error page would pass this test, so the status check is the least you do.
The same page with a library and with an API
Here is a library version in Python. It fetches the HTML and pulls out the title and the paragraphs:
import requests
from bs4 import BeautifulSoup
resp = requests.get("https://example.com", timeout=30)
resp.raise_for_status()
soup = BeautifulSoup(resp.text, "html.parser")
print(soup.title.string)
for p in soup.find_all("p"):
print(p.get_text(strip=True))It is short, it pays no fee per page, and it is right for this page. It also holds three assumptions. The text is in the first response, the selectors match this site’s markup, and the site answers your IP address. Each one breaks on some sites, and each fix is code you maintain: a headless browser for the first, new selectors per site for the second, retries and pacing for the third.
Here is the API version. It is the curl request from the first section, sent from Python:
import os
import requests
resp = requests.post(
"https://api.spider.cloud/scrape",
headers={"Authorization": f"Bearer {os.environ['SPIDER_API_KEY']}"},
json={"url": "https://example.com", "request": "smart", "return_format": "markdown"},
timeout=120,
)
resp.raise_for_status()
page = resp.json()[0]
if page["status"] == 200:
print(page["content"])
else:
print("page status", page["status"], page["error"])There are no selectors, because the answer is the whole page as markdown with its headings intact. The same code works on a site with different markup. The trade is a per-request cost and a network call to someone else’s servers.
When a library is enough
A library is enough when you scrape a few sites whose markup you know and whose text is in the first HTTP response. When the pages must not leave your network, the fetching has to run inside it, with a library or a crawler you host yourself. A plain fetch is often enough, as the 26 of 29 above shows, and our guide to web scraping covers what a render adds. Check whether the text is in the first HTTP response before you assume you need a browser.
An API is the better fit when some pages need a browser you do not want to run, when the sites are many and their markup differs, or when a model reads the output. In our run on ten public pages on 2026-11-05, markdown cut a page’s tokens 15x at the median compared with the raw HTML.
That is the short version. The developer’s guide to choosing a scraping stack works through the choice in full, with a decision matrix.
What to check in any scraping API
Before you wire one in, read its reference for these five things. Here is how Spider’s reads:
- Whether it follows robots.txt by default. Spider’s
respect_robotsis true unless you change it. - What a failed page costs. An attempt bills for the bandwidth and compute it used, whatever status the page returns. A failed page is not free.
- Whether the page status is separate from the call status. In Spider’s API it is, as the 404 run above shows.
- Which formats it returns. Spider’s
return_formattakesmarkdown,commonmark,text,html2text,xml,bytesandraw, among others, and the response can be JSON, CSV, JSONL or XML. - What a 429 tells you. In Spider’s API, wait the number of seconds in Retry-After, then send the request again.
Send the curl request above with your own URL and read both statuses in the answer. Create an API key to start, or paste a URL into the web scraper to read an answer before you have a key.
Frequently asked questions
What is a web scraping API?
A service that fetches web pages for you. You send it a URL and a few options over HTTP. It downloads the page, runs a browser when the page needs one, turns the result into the format you asked for, such as markdown or plain text, and sends it back. You run no crawler or browser yourself.
How is a web scraping API different from a scraping library?
A library runs inside your program, on your machine and from your IP address, and you write the code that fetches and parses each site. An API does the fetching and rendering on its own servers and hands back the content. The library charges no fee per page, though you still pay for the machines and bandwidth it runs on. The API has no browsers or parsers for you to keep running.
Is a web scraping API the same as a proxy?
No. A proxy forwards your own request and returns the raw response, so your code still parses the HTML and runs any browser. A scraping API makes the request itself and returns the content in the format you chose.
Why did the API return 200 when the page was not found?
Because a scraping API has two statuses. The HTTP status of the API call says whether the API did its job. Each page in the answer carries its own status field with the site's answer. In Spider's API a 200 from the API can carry a page with status 404. On November 17, 2026 one request for a missing page did that, and its content field held the 100 words of the site's error page.
Do I need a web scraping API for Python?
No. requests and Beautiful Soup scrape static HTML well. An API helps when pages need a browser, when you scrape many sites with different markup, or when you want markdown instead of selectors.
Is using a web scraping API legal?
The same questions apply as for any scraper: the site's terms, the kind of data and the law where you are. Our robots.txt survey counts what 67 top sites ask of crawlers. None of this is legal advice.
Sources
- Spider OpenAPI file, the /scrape request, the request and return_format parameters, and the PageResult fields url, content, status, error, costs and duration_elasped_ms, read 2026-11-16
- Measured run, 2026-11-17, 10:29 UTC: one scrape request through the Spider API for an address on spider.cloud that does not exist
- How web scraping works, the run of 2026-11-16 where a plain fetch got the text on 26 of 29 pages
- Requests documentation
- Beautiful Soup documentation
- What a web page costs in tokens, measured on ten public pages, 2026-11-05
- Spider pricing, how an attempt is billed
- Is web scraping legal? What 67 top sites ask for in robots.txt, the survey of 2026-11-16
Keep reading
Proxies for web scraping, built into every request
Proxies for web scraping come built into every Spider request and rotate on their own. What they cost, and how to pick the exit country or use your own.
Web scraping for AI agents: search, scrape or crawl
Web scraping for AI agents: when an agent should search, scrape or crawl, and how many tokens of its context window each call fills, counted on real pages.
Web scraping to markdown: a whole docs site into files
Web scraping to markdown with one crawl call and a short Python script that writes a file per page, then a check of the code blocks and tables it kept.
Run this on a page you care about
The playground sends the request this page describes and shows you the response. Keyless runs work without an account and are rate limited.