Skip to main content

GET aeon.co HTTP 200512 ms23.8 KB$0.000401 captured Aug 7, 2026

aeon.co, as data

One API turns any aeon.co page into markdown, structured JSON, its link graph or a screenshot, and crawls the whole site the same way. The panel shows the real response we captured.

target profile from the capture
Render mode
Plain HTTP, no browser needed
Response
HTTP 200 in 512 ms
Page size
23.8 KB of markdown, 104 lines
Fields
Title, Content, Date, Source and 2 more
Captured
Aug 7, 2026

Free balance on signup, no card. This aeon.co page cost $0.000401 to fetch.

response 200 · 512 ms · 23.8 KB
The workshop to solve this problem began as a hallway conversation in 1994 at the second International World Wide Web Conference in Chicago. Weibel, who worked in the research group at the library consortium OCLC, was standing around drinking coffee in the hallway with five or six people, including his boss, Terry Noreault, and Eric Miller, his colleague on the OCLC research team. As Weibel remembers: ‘We were talking about how nice it would be if there were easier ways to find the 500,000 individually addressable objects [documents] on the web … I looked at my boss, and he just nodded and agreed to organise a workshop for it.’The workshop was quickly co-organised by Weibel and Miller, who wanted to be able to take the results to the next web conference in Germany the following spring. In order to develop a system that worked, they knew they needed input from three different groups of people: encoding and markup experts in specialised disciplines, who could help ensure that metadata was effectively associated with the online files; computer scientists; and librarians, or, as multiple people who attended the first workshop told me with deep affection, ‘the freaks, the geeks, and the ones with sensible shoes’.Some 52 people showed up to the workshop in Dublin, Ohio. The variety of attendees, and their perspectives on how documents on the web should be organised, was striking. As Priscilla Caplan, a librarian who attended the conference, wrote at the time: ‘There were the IETF [Internet Engineering Task Force] guys, astonishingly young and looking as if they were missing a fraternity party to be there. There were TEI [Text Encoding Initiative] people … geospatial metadata people … publishers and software developers and researchers.’ All had very different goals, but ‘nearly everyone agreed that there was a tremendous need for some standard’.In 1995, most librarians were using MARC (MAchine-Readable Cataloguing) to create metadata for their library catalogues. MARC records are complex, extremely long, and require deep expertise to create. These kinds of elaborate descriptions could never work at scale for the entire web. Automated approaches weren’t on the table back then, and it soon became clear to all attendees, even those who had showed up thinking that they might be tweaking an existing system, that the metadata standard for the web would have to be something entirely new: simple enough for anyone to label their own documents as they posted them online, but still meaningful and specific enough for other people and machines to find and index them. A brand-new, simple and succinct metadata system would mean that, for the half a million existing items online, and the millions and billions more that everyone knew were coming, there would need to be one agreed-upon way of adding the metadata tags, with the same kinds of information in the tags themselves.Creating these labels involved figuring out not just what would be needed to find files that were online now, but also what might be needed later as web content continued to snowball. There was no formalised voting or veto process to come up with the system; each piece of metadata was created through consensus, compromise and, occasionally, real fights. Much of the argument, in fact, concerned the nature of the future no one could truly predict in full.For example, many attendees didn’t anticipate that automated search engines were coming, though some of the more technical people saw them on the horizon and were pushing requirements for improved geolocated discovery. As Miller says: ‘I remember introducing the [geolocation] coverage element and getting a lot of blowback. I made the point that coverage is going to be local as well as global, like: *Find a restaurant near me*. We were trying to push the envelope so that we would be ready when other technologies advanced and other services became available.’ Other attendees saw geospatial data as something put in to assuage a person or community, and they weren’t sure it made sense given the need to keep the system lean.In the beginning, the disagreements seemed insurmountable, and Miller felt disheartened: ‘The first night we thought: *This is gonna fail miserably*. At first nobody saw eye to eye or trusted each other enough yet to let each other in and try to figure out the art of the possible.’ But as concessions and then agreements were made, people began to feel energised by the creation of a new system, even if imperfect; one piece at a time, their system could bring the content of the web within reach for everyone. As Caplan remembers: ‘By the second day, there was a lot of drinking and all-night working groups. We were running on adrenaline and energy. By the last day, we realised we were making history.’Dublin Core was revolutionary in its creation of a very new middle groundThe result of all the arguments was ‘Dublin Core’ (DC) metadata, the first metadata standard for describing content on the web. The final short group of DC tags, or metadata ‘elements’, was drawn from a longer list that had been developed, iterated, analysed, argued over, and eventually cut down to a list of 13. In his workshop report, Weibel provided an example of the elements, using the University of Virginia Library’s record of Maya Angelou’s poem ‘On the Pulse of Morning’, transcribed by the library from Angelou’s performance at Bill Clinton’s inauguration:* **Title:** On the Pulse of Morning* **Publisher:** University of Virginia Library Electronic Text Center
POST /scrape · aeon.co/essays/the-birth-of-our-system-for-describing-web-content.md · markdown · Showing 11 of 104 lines · captured Aug 7, 2026

What Spider does on aeon.co

Same key, five endpoints. The numbers under a cell were measured on this page.

/scrape

Page to markdown

Clean text for RAG and LLM context, boilerplate removed. The lane that runs without a key.

512 ms · $0.000401
/fetch

Page to JSON

Spider reads the page and names the fields. Pass your own schema when you need exact keys.

/crawl

Whole site

Follows aeon.co's own links, respects robots, streams pages as they land.

/links

Link graph

Every internal and external URL on a page, before you decide what to fetch.

/screenshot

Rendered capture

Real Chromium, full-page PNG. The same call also returns the rendered HTML.

The call behind the panel

This request produced the response above. Paste it with your key and you get the same bytes.

request scrape.sh
curl -X POST https://api.spider.cloud/scrape \
  -H "Authorization: Bearer $SPIDER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://aeon.co/essays/the-birth-of-our-system-for-describing-web-content", "return_format": "markdown"}'
plain request · no browser, no selectors

What aeon.co costs

Multiplied from the measured 23.8 KB page. Estimates round to the cent.

cost measured
This page, once 23.8 KB over HTTP, measured $0.000401
This page, daily for a month 30 fetches at the measured price ≈ $0.01
1,000 pages this size 23.3 MB at the measured price ≈ $0.40
10,000 pages this size 232.7 MB at the measured price ≈ $4.01

1 GB of transfer costs $1, plus $0.001 per CPU minute, and failed requests cost $0. Crawling daily? The Unlimited plan is a flat monthly rate.

Point this at the rest of aeon.co.

The capture above took 512 ms and cost $0.000401. The same call takes any URL on aeon.co.

Get an API key

More in International

All International scrapers