JSON Scraping

Many websites embed structured data directly in their HTML — product details in JSON-LD, page data in Next.js __NEXT_DATA__ scripts, or API payloads in SSR hydration. Spider can extract this structured data directly, bypassing the need to parse HTML. Set return_json_data: true to get the embedded JSON from any page.

Example JSON-LD Microformat

Extracting JSON-LD from a Page

Here is a practical example: extracting recipe data from a page that embeds JSON-LD structured data. The return_format: "empty" parameter tells Spider to skip the HTML content and only return the extracted JSON.

Scraping JSON from HTML Using API in Python

import requests
import os

headers = {
  'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
  'Content-Type': 'application/json',
}

params = {
  "url": "https://www.allrecipes.com/recipe/223312/nutella-hazelnut-cookies",
  "return_format": "empty",
  "return_json_data": True # Return the JSON data embedded in the HTML
}

response = requests.post(
  'https://api.spider.cloud/scrape',
  headers=headers,
  json=params
)

print(response.json())

Example Response

The JSON will be in the response under the other_scripts array:

Example Response JSON

{
  "costs": {
    "ai_cost": 0,
    "file_cost": 0.0005,
    "total_cost": 0.0005
  },
  "error": null,
  "json_data": {
    "other_scripts": [
      {
        "@context": "http://schema.org",
        "@type": ["Recipe"],
        "aggregateRating": {
          "@type": "AggregateRating",
          "ratingCount": "102",
          "ratingValue": "4.7"
        },
        "author": [
          {
            "@type": "Person",
            "name": "Carmella DiNardo"
          }
        ],
        "cookTime": "PT10M",
        "datePublished": "2020-06-18T23:50:15.000-04:00",
        "description": "Nutella cookies made with chocolate-hazelnut spread...",
        "headline": "Nutella Cookies",
        "name": "Nutella Cookies"
      }
    ]
  }
}

Scrape Next.js SSR data embedded in the HTML

Using the same return_json_data parameter, we can also scrape the SSR data on Next.js pages and other similar JS frontend frameworks. Example response for SSR, the JSON object is found in the NEXT_DATA property:

Scraping Next.js SSR Data

[
  {
    "content": null,
    "costs": {
      "ai_cost": 0,
      "bytes_transferred_cost": 0,
      "compute_cost": 0,
      "file_cost": 0.0005,
      "total_cost": 0.0005,
      "transform_cost": 0
    },
    "error": null,
    "json_data": {
      "NEXT_DATA": {
        "props": {
          "pageProps": {
            "geo": {
              "_id": "city:ca_san-jose",
              "area_type": "city",
              "city": "San Jose",
              "state_code": "CA",
              "country": "USA"
            },
            "pageType": "forSale",
            "page": 1,
            "properties": [
              {
                "property_id": "1143655170",
                "list_price": 6498000,
                "primary_photo": {
                  "href": "https://ap.rdcpix.com/..."
                }
              }
            ]
          }
        }
      }
    }
  }
]