Hex Packages Scraper

A directory listing of all packages available on the Hex package manager for the Erlang ecosystem. Built on spider-browser .

Get started Docs

target
hex.pm: success rate
99.9%: latency
~4ms

Quick start

Extract data in minutes.

hex-pm-scraper.ts

import { SpiderBrowser } from "spider-browser";

const spider = new SpiderBrowser({
  apiKey: process.env.SPIDER_API_KEY!,
});

await spider.connect();
const page = spider.page!;
await page.goto("https://www.hex.pm");

const data = await page.extractFields({
  about_hex: "div.footer-nav ul li",
  about_hex_link: "div.footer-nav ul li a",
  about_hex_title: "div.footer-nav h4",
  help: "div.footer-nav ul li",
  help_link: "div.footer-nav ul li a",
  help_title: "div.footer-nav h4",
});

console.log(data);
await spider.close();

ready to run · spider-browser · TypeScript

Fetch API

One endpoint for hex.pm.

Structured JSON from hex.pm with a single POST. AI-resolved selectors, cached on the first call.

POST /fetch/hex.pm/

About HexAbout Hex LinkAbout Hex TitleHelpHelp LinkHelp Title

Try it Fetch docs

cURL

curl -X POST https://api.spider.cloud/fetch/hex.pm/ \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"return_format": "json"}'

Python

import requests

resp = requests.post(
    "https://api.spider.cloud/fetch/hex.pm/",
    headers={
        "Authorization": "Bearer YOUR_API_KEY",
        "Content-Type": "application/json",
    },
    json={"return_format": "json"},
)
print(resp.json())

Node.js

const resp = await fetch("https://api.spider.cloud/fetch/hex.pm/", {
  method: "POST",
  headers: {
    "Authorization": "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ return_format: "json" }),
});
const data = await resp.json();
console.log(data);

Extraction

Fields you can pull.

About HexAbout Hex LinkAbout Hex TitleHelpHelp LinkHelp TitleNew Package LinkNew Package TitlePackage LinkPackage TitlePolicies And TermsPolicies And Terms LinkPolicies And Terms TitleRecently Updated LinkRecently Updated TitleSearch ButtonSearch LabelTitleUsing With Box ElixirUsing With Box Erlang

Rendering