bioRxiv Scraper
Spider read biorxiv.org in 22.2 s without a browser and returned 226 lines of clean markdown.
# Keywords to success: a practical guide to maximise the visibility and impact of academic papersView ORCID ProfilePatrice Pottier, View ORCID ProfileMalgorzata Lagisz, View ORCID ProfileSamantha Burke, View ORCID ProfileSzymon M. Drobniak, View ORCID ProfilePhilip A. Downing, View ORCID ProfileErin L. Macartney, View ORCID ProfileApril Robin Martinig, View ORCID ProfileAyumi Mizuno, View ORCID ProfileKyle Morrison, View ORCID ProfilePietro Pollo, View ORCID ProfileLorenzo Ricolfi, View ORCID ProfileJesse Tam, View ORCID ProfileCoralie Williams, View ORCID ProfileYefeng Yang, View ORCID ProfileShinichi Nakagawadoi: https://doi.org/10.1101/2023.10.02.5598611Evolution & Ecology Research Centre, School of Biological, Earth and Environmental Sciences, The University of New South Wales, Sydney, New South Wales, Australia* Find this author on Google Scholar* Search for this author on this site* ORCID record for Patrice Pottier2Theoretical Sciences Visiting Program, Okinawa Institute of Science and Technology Graduate University, Onna, 904-0495, Japan* ORCID record for Malgorzata Lagisz* ORCID record for Samantha Burke3Institute of Environmental Sciences, Jagiellonian University, Kraków, Poland* ORCID record for Szymon M. Drobniak4Ecology and Genetics Research Unit, University of Oulu, Finland* ORCID record for Philip A. Downing5Department of Zoology, Stockholm University, Stockholm* ORCID record for Erin L. Macartney* ORCID record for April Robin Martinig6Department of Biology, Faculty of Science, Hokkaido University, Sapporo, 060-0810, Japan* ORCID record for Ayumi Mizuno* ORCID record for Kyle Morrison* ORCID record for Pietro Pollo* ORCID record for Lorenzo Ricolfi7Centre for Ecosystem Science and School of Biological, Earth and Environmental Sciences, UNSW, Sydney, NSW 2052, Australia* ORCID record for Coralie WilliamsThe same call, in code.
The capture above came back as markdown. These examples add a key, so you get browser rendering, proxies, and concurrency on biorxiv.org.
import { SpiderBrowser } from "spider-browser";
const spider = new SpiderBrowser({
apiKey: process.env.SPIDER_API_KEY!,
});
await spider.connect();
const page = spider.page!;
await page.goto("https://www.biorxiv.org/search/genomics");
// No selectors, no schema. Spider reads the page and names the fields.
const data = await page.scrape();
console.log(data);
await spider.close();import { SpiderBrowser } from "spider-browser";
const spider = new SpiderBrowser({
apiKey: process.env.SPIDER_API_KEY!,
});
await spider.connect();
const page = spider.page!;
await page.goto("https://www.biorxiv.org/search/genomics");
await page.content(10000);
const data = await page.evaluate(`(() => {
const papers = [];
document.querySelectorAll(".search-result, .highwire-cite").forEach(el => {
const title = el.querySelector(".highwire-cite-title a")?.textContent?.trim();
const authors = el.querySelector(".highwire-cite-authors")?.textContent?.trim();
const date = el.querySelector(".highwire-cite-metadata-date")?.textContent?.trim();
const doi = el.querySelector(".highwire-cite-metadata-doi")?.textContent?.trim();
if (title) papers.push({ title, authors, date, doi });
});
return JSON.stringify({ total: papers.length, papers: papers.slice(0, 15) });
})()`);
console.log(JSON.parse(data));
await spider.close();Ready for volume? Get an API key →
Fields you can pull.
Spider names these from the page. The capture above came back as markdown; the same
call with return_format: "json" returns them as keys.
What biorxiv.org costs to scrape.
The capture above cost $0.000238 to fetch. Pricing is $1 per GB of pre-transformation bandwidth plus $0.001 per CPU minute, so a page like this one lands at a fraction of a cent. Failed requests are billed at $0.
- Free balance on signup
- No card required to test
- Balance never expires
Run it keyless, no account
More Science & Research scrapers.
arXiv Scraper
Extract preprint papers, abstracts, author lists, and citation metadata from arXiv open-access research repository.
PubMed Scraper
Extract biomedical literature, abstracts, MeSH terms, and citation data from PubMed National Library of Medicine database.
ResearchGate Scraper
Extract researcher profiles, publication lists, citation metrics, and project data from ResearchGate academic network.
Start scraping biorxiv.org.
You already have the call. A key raises the rate limit and turns on browser rendering, proxies, and concurrency. Balance never expires, and top-ups go through secure checkout.