AI & Developer

Hex Packages Scraper

A directory listing of all packages available on the Hex package manager for the Erlang ecosystem. Powered by spider-browser .

Get Started Documentation

hex.pm target

99.9% success rate

~4ms latency

Quick Start

Extract data in minutes

hex-pm-scraper.ts

import { SpiderBrowser } from "spider-browser";

const spider = new SpiderBrowser({
  apiKey: process.env.SPIDER_API_KEY!,
});

await spider.connect();
const page = spider.page!;
await page.goto("https://www.hex.pm");

const data = await page.extractFields({
  about_hex: "div.footer-nav ul li",
  about_hex_link: "div.footer-nav ul li a",
  about_hex_title: "div.footer-nav h4",
  help: "div.footer-nav ul li",
  help_link: "div.footer-nav ul li a",
  help_title: "div.footer-nav h4",
});

console.log(data);
await spider.close();

✓ ready to run | spider-browser | TypeScript

Fetch API

Structured data endpoint

Extract structured JSON from hex.pm with a single POST request. AI-configured selectors, cached for fast repeat calls.

POST /fetch/hex.pm/

About HexAbout Hex LinkAbout Hex TitleHelpHelp LinkHelp Title

Try it Fetch API Docs

curl

curl -X POST https://api.spider.cloud/fetch/hex.pm/ \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"return_format": "json"}'

Python

import requests

resp = requests.post(
    "https://api.spider.cloud/fetch/hex.pm/",
    headers={
        "Authorization": "Bearer YOUR_API_KEY",
        "Content-Type": "application/json",
    },
    json={"return_format": "json"},
)
print(resp.json())

Node.js

const resp = await fetch("https://api.spider.cloud/fetch/hex.pm/", {
  method: "POST",
  headers: {
    "Authorization": "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ return_format: "json" }),
});
const data = await resp.json();
console.log(data);

Extraction

Data you can extract

About HexAbout Hex LinkAbout Hex TitleHelpHelp LinkHelp TitleNew Package LinkNew Package TitlePackage LinkPackage TitlePolicies And TermsPolicies And Terms LinkPolicies And Terms TitleRecently Updated LinkRecently Updated TitleSearch ButtonSearch LabelTitleUsing With Box ElixirUsing With Box Erlang

Rendering