Skip to main content
Docs API Ref

JSON scraping

Many sites ship structured data inside their HTML: product details in JSON-LD, page state in a Next.js __NEXT_DATA__ script, API payloads left behind by server rendering. Set return_json_data: true and Spider returns that JSON as is, so there is no HTML to parse.

JSON-LD in a page

<script type="application/ld+json">
{
  "@context": "http://schema.org",
  "@type": "Person",
  "name": "Emma Johnson",
  "image": "https://example.com/images/emma-johnson.jpg",
  "jobTitle": "Senior Software Engineer",
  "telephone": "+1-408-555-7890",
  "email": "emma.johnson@techcorp.com",
  "address": {
    "@type": "PostalAddress",
    "streetAddress": "456 Innovation Drive",
    "addressLocality": "San Jose",
    "addressRegion": "CA",
    "postalCode": "95131",
    "addressCountry": "US"
  }
}
</script>

Extract JSON-LD from a page

This request pulls the recipe data a page embeds as JSON-LD. return_format: "empty" tells Spider to skip the HTML content and return only the extracted JSON.

Python

import requests
import os

headers = {
  'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
  'Content-Type': 'application/json',
}

params = {
  "url": "https://www.allrecipes.com/recipe/223312/nutella-hazelnut-cookies",
  "return_format": "empty",
  "return_json_data": True # Return the JSON data embedded in the HTML
}

response = requests.post(
  'https://api.spider.cloud/scrape',
  headers=headers,
  json=params
)

print(response.json())

Example response

The JSON comes back under the other_scripts array:

Response

{
  "costs": {
    "ai_cost": 0,
    "file_cost": 0.0005,
    "total_cost": 0.0005
  },
  "error": null,
  "json_data": {
    "other_scripts": [
      {
        "@context": "http://schema.org",
        "@type": ["Recipe"],
        "aggregateRating": {
          "@type": "AggregateRating",
          "ratingCount": "102",
          "ratingValue": "4.7"
        },
        "author": [
          {
            "@type": "Person",
            "name": "Carmella DiNardo"
          }
        ],
        "cookTime": "PT10M",
        "datePublished": "2020-06-18T23:50:15.000-04:00",
        "description": "Nutella cookies made with chocolate-hazelnut spread...",
        "headline": "Nutella Cookies",
        "name": "Nutella Cookies"
      }
    ]
  }
}

Scrape Next.js SSR data

The same return_json_data parameter also picks up the server-rendered state on Next.js pages and similar JS frameworks. In that case the object sits under the NEXT_DATA property:

Next.js SSR response

[
  {
    "content": null,
    "costs": {
      "ai_cost": 0,
      "bytes_transferred_cost": 0,
      "compute_cost": 0,
      "file_cost": 0.0005,
      "total_cost": 0.0005,
      "transform_cost": 0
    },
    "error": null,
    "json_data": {
      "NEXT_DATA": {
        "props": {
          "pageProps": {
            "geo": {
              "_id": "city:ca_san-jose",
              "area_type": "city",
              "city": "San Jose",
              "state_code": "CA",
              "country": "USA"
            },
            "pageType": "forSale",
            "page": 1,
            "properties": [
              {
                "property_id": "1143655170",
                "list_price": 6498000,
                "primary_photo": {
                  "href": "https://ap.rdcpix.com/..."
                }
              }
            ]
          }
        }
      }
    }
  }
]