arxiv.org

arXiv Scraper

Extract preprint papers, abstracts, author lists, and citation metadata from arXiv open-access research repository.

No key required

curl -X POST https://api.spider.cloud/scrape \
  -H "Content-Type: application/json" \
  -d '{"url": "https://arxiv.org/html/2403.04665v2", "return_format": "markdown"}'

This exact call produced the response beside it. Get a key →

Content: 186 lines 10.7 KB markdown
Render: No browser plain request is enough
Fetch time: 1.1 s one measured fetch
This fetch cost: $0.000075 bandwidth plus compute

PAPER TITLE (use style: PAPER TITLE) https://arxiv.org/html/2403.04665v2

Tables should be formatted as shown in the example below with no column lines, unless these are essential for clarifying the content of the table. Row lines can be used to distinguish the column headings from the content of the table.
tables must be numbered and cited within the text in strict numerical order. Table captions must be above the table and in 10 pt font.
Equations should fit into a two-column print format and be single spaced.
When writing mathematics, avoid confusion between characters that could be mistaken for one another, e.g. the letter ‘l’ and the number one.
Vectors and matrices should be in bold italic and variables in italic.
If your paper contains superscripts or subscripts, take special care to ensure that the positioning of the characters is unambiguous.
Exponential expressions should be written using superscript notation, i.e. 5×1035superscript1035\times 10^{3}5 × 10 start_POSTSUPERSCRIPT 3 end_POSTSUPERSCRIPT not 5E03.
A multiplication symbol should be used, not a dot.
Refer to equations using round brackets e.g. (1)
Use SI (MKS) units only and avoid spelling the unit in full instead of using the shortened notation e.g. use kJ not kilo Joules.
If for any reason you must use mixed units, the units used for each quantity in an equation must be stated.
Place a zero before decimal points: “0.10” do not put “.10”.
Submissions should always include the following sections: an abstract; an introduction; a conclusion and a references section. If any of the above sections are not included the paper may be asked to add the relevant section or be rejected.
Acknowledgements should be placed after the conclusion and before the references section. Details of grants, financial aid and other special assistance should be noted.
You should number your references sequentially throughout the text, and each
reference should be individually numbered and enclosed in square brackets

HTTP 200

You just saw the output.

That was one page. A key runs the same call across every URL on arxiv.org, with browser rendering, proxies, and concurrency. Sign up and a free balance lands on your account. No card required to test.

Free balance on signup, no card
Failed requests cost $0
robots.txt respected by default

Get an API key Read the API docs

The same call, in code.

The keyless call above returns markdown. These examples add a key, so you get browser rendering, proxies, and concurrency on arxiv.org.

arxiv-scraper.ts

import { SpiderBrowser } from "spider-browser";

const spider = new SpiderBrowser({
  apiKey: process.env.SPIDER_API_KEY!,
});

await spider.connect();
const page = spider.page!;
await page.goto("https://arxiv.org/list/cs.AI/recent");

// No selectors, no schema. Spider reads the page and names the fields.
const data = await page.scrape();

console.log(data);
await spider.close();

import { SpiderBrowser } from "spider-browser";

const spider = new SpiderBrowser({
  apiKey: process.env.SPIDER_API_KEY!,
});

await spider.connect();
const page = spider.page!;
await page.goto("https://arxiv.org/list/cs.AI/recent");
await page.content();

const data = await page.evaluate(`(() => {
  const papers = [];
  document.querySelectorAll("#dlpage dt, #dlpage dd").forEach((el, i, all) => {
    if (el.tagName === "DT") {
      const dd = all[i + 1];
      const title = dd?.querySelector(".list-title")?.textContent?.replace("Title:", "").trim();
      const authors = dd?.querySelector(".list-authors")?.textContent?.replace("Authors:", "").trim();
      const abstract = dd?.querySelector(".mathjax")?.textContent?.trim();
      const id = el.querySelector("a[title='Abstract']")?.textContent?.trim();
      if (title) papers.push({ id, title, authors, abstract: abstract?.slice(0, 200) });
    }
  });
  return JSON.stringify({ total: papers.length, papers: papers.slice(0, 15) });
})()`);

console.log(JSON.parse(data));
await spider.close();

ready to run · spider-browser, no selectors

Ready for volume? Get an API key →

Fields you can pull.

Paper titleAuthorsAbstractCategoriesSubmission datePDF linkDOIComments

Spider names these from the page. The capture above came back as markdown; the same call with return_format: "json" returns them as keys.

What arxiv.org costs to scrape.

The capture above cost $0.000075 to fetch. Pricing is $1 per GB of pre-transformation bandwidth plus $0.001 per CPU minute, so a page like this one lands at a fraction of a cent. Failed requests are billed at $0.

Free balance on signup
No card required to test
Balance never expires

See the full pricing →

More Science & Research scrapers.

pubmed.ncbi.nlm.nih.gov

Start scraping arxiv.org.

You already have the call. A key raises the rate limit and turns on browser rendering, proxies, and concurrency. Balance never expires, and top-ups go through secure checkout.

Get started free View pricing