Terraform Registry Scraper

Extract provider details, module documentation, resource schemas, and version histories from Terraform Registry. Built on spider-browser .

Get started Docs

target
registry.terraform.io: success rate
99.9%: latency
~4ms

Quick start

Extract data in minutes.

terraform-registry-scraper.ts

import { SpiderBrowser } from "spider-browser";

const spider = new SpiderBrowser({
  apiKey: process.env.SPIDER_API_KEY!,
});

await spider.connect();
const page = spider.page!;
await page.goto("https://registry.terraform.io/browse/providers");
await page.content(10000);

const data = await page.evaluate(`(() => {
  const providers = [];
  document.querySelectorAll("[class*='provider-card'], [class*='ProviderCard']").forEach(el => {
    const name = el.querySelector("h3, [class*='name']")?.textContent?.trim();
    const desc = el.querySelector("p, [class*='description']")?.textContent?.trim();
    const downloads = el.querySelector("[class*='downloads']")?.textContent?.trim();
    const version = el.querySelector("[class*='version']")?.textContent?.trim();
    if (name) providers.push({ name, desc, downloads, version });
  });
  return JSON.stringify({ total: providers.length, providers: providers.slice(0, 15) });
})()`);

console.log(JSON.parse(data));
await spider.close();

ready to run · spider-browser · TypeScript

Fetch API

One endpoint for registry.terraform.io.

Structured JSON from registry.terraform.io with a single POST. AI-resolved selectors, cached on the first call.

POST /fetch/registry.terraform.io/

Provider nameModule nameVersionDownloadsDescriptionResources

Try it Fetch docs

cURL

curl -X POST https://api.spider.cloud/fetch/registry.terraform.io/ \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"return_format": "json"}'

Python

import requests

resp = requests.post(
    "https://api.spider.cloud/fetch/registry.terraform.io/",
    headers={
        "Authorization": "Bearer YOUR_API_KEY",
        "Content-Type": "application/json",
    },
    json={"return_format": "json"},
)
print(resp.json())

Node.js

const resp = await fetch("https://api.spider.cloud/fetch/registry.terraform.io/", {
  method: "POST",
  headers: {
    "Authorization": "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ return_format: "json" }),
});
const data = await resp.json();
console.log(data);

Extraction

Fields you can pull.

Provider nameModule nameVersionDownloadsDescriptionResourcesDocumentationPublisher

Rendering

React SPA handling

Full browser rendering for streaming content and dynamic React UI.

Content

Structured parsing

Extract code blocks, documentation, repository data, and metadata.

Waiting

Load completion

Smart network idle detection waits for dynamically loaded content to finish.

More AI & Developer scrapers.

chatgpt.com

ChatGPT Scraper

Extract shared ChatGPT conversations, prompts, and AI-generated content from public links.

huggingface.co

Hugging Face Scraper

Extract ML model cards, dataset info, leaderboard data, and paper metadata from Hugging Face.

github.com

GitHub Scraper

Extract trending repositories, star counts, contributor data, and code snippets from GitHub.

Start

Start scraping registry.terraform.io.

Grab an API key and call the endpoint above. The first request resolves the config; every request after hits cache.

Get started free All scrapers