GET dnswl.org HTTP 200110 ms3.5 KB$0.000147 captured Aug 7, 2026
dnswl.org, as data
One API turns any dnswl.org page into markdown, structured JSON, its link graph or a screenshot, and crawls the whole site the same way. The panel shows the real response we captured.
- Render mode
- Plain HTTP, no browser needed
- Response
- HTTP 200 in 110 ms
- Page size
- 3.5 KB of markdown, 21 lines
- Fields
- Business Name, Address, Phone, Category and 2 more
- Captured
- Aug 7, 2026
Free balance on signup, no card. This dnswl.org page cost $0.000147 to fetch.
At the M3AAWG meeting (see previous post) there was considerable interest in the proposal on domain-based whitelisting information. However there is no operational experience which would allow anyone to see how this would perform.Now you can test 🙂 Continue reading “Whitelisting DKIM-signed domains”At the 41st M3AAWG meeting in Toronto, we got the opportunity to present the dnswl.org project to a wider community.The slides linked below were shown – please let us know if you have any questions or further input.I hope we will be back for the M3AAWG meeting in Munich next year.dnswl.org will remove a workaround-file for Postfix < 2.8 from download on August 31st 2017. Continue reading “Removal of workaround-file for Postfix < 2.8 from download (updated)"TL;DR: There are over 500’000 legitimate mailservers on the Internet. If you are not managing the IPs of your own mailservers at dnswl.org yet, you should do it!Continue reading “How many legitimate mailservers are there?”TL;DR: Sign up to the dnswl.org Self Service and start managing the reputation of your IPs.Continue reading “Do you manage the reputation of your IPs?”dnswl.org limits the use of the public nameserver infrastructure to those doing fewer than 100’000 lookups per 24 hours and to non-commercial use (see here for details). Continue reading “Enforcement of dnswl.org access limits”For historical reasons, the website to manage subscriptions to dnswl.org data was using a separate website (and “separate-everything”). Over the past months we worked to integrate this into the more generic Self Service. Continue reading “Subscription website moved into Self Service”Since the dnswl.org Self Service went live in mid December 2015, we got over 1’000 Self Service user signups, and these users created over 3’000 Change Requests. Of course we are happy about the great up-take of the Self Service in the community.What Spider does on dnswl.org
Same key, five endpoints. The numbers under a cell were measured on this page.
Page to markdown
Clean text for RAG and LLM context, boilerplate removed. The lane that runs without a key.
110 ms · $0.000147Page to JSON
Spider reads the page and names the fields. Pass your own schema when you need exact keys.
Rendered capture
Real Chromium, full-page PNG. The same call also returns the rendered HTML.
The call behind the panel
This request produced the response above. Paste it with your key and you get the same bytes.
curl -X POST https://api.spider.cloud/scrape \
-H "Authorization: Bearer $SPIDER_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://dnswl.org/", "return_format": "markdown"}'from spider import Spider
app = Spider()
params = {"return_format": "markdown"}
page = app.scrape_url("https://dnswl.org/", params=params)
print(page[0]["content"])import { Spider } from "@spider-cloud/spider-client";
const app = new Spider();
const [page] = await app.scrapeUrl("https://dnswl.org/", {
return_format: "markdown",
});
console.log(page.content);What dnswl.org costs
Multiplied from the measured 3.5 KB page. Estimates round to the cent.
1 GB of transfer costs $1, plus $0.001 per CPU minute, and failed requests cost $0. Crawling daily? The Unlimited plan is a flat monthly rate.
Point this at the rest of dnswl.org.
The capture above took 110 ms and cost $0.000147. The same call takes any URL on dnswl.org.