Efficient scraping

Ways to cut latency and credits when you scrape thousands of pages.

Several URLs in one request

Pass a comma-separated list in url. Spider fetches them concurrently in one request, which suits paginated lists or any known set of pages that share one configuration. Add streaming to read results as they finish.

Multiple URLs

params = {
	"url": "https://www.example.com, https://example2.com",
	"limit": 100,
	"request": "smart",
	"return_format": "markdown"
}

Batch mode

When each URL needs its own limit, request mode or return_format, send an array of parameter objects. Each entry runs with its own settings.

Batch mode

params = [{
        "url": "https://www.example.com",
        "limit": 5,
        "request": "browser",
        "return_format": "markdown"
    },
    {
        "url": "https://www.example2.com/",
        "limit": 10,
        "request": "smart",
        "return_format": "markdown"
    },
    {
        "url": "https://www.example3.com/",
        "limit": 1,
        "request": "browser",
        "return_format": "raw"
    }

Retries

Spider retries failed requests on its own and moves from datacenter to residential proxies as it goes. If you retry by hand, stop at 2 attempts and change something each time: switch http to browser, or set the proxy type to residential. Do not retry a hard failure such as 404 or 401.

Tip

Use one proxy setting per request.

Timeouts

request_timeout caps how long Spider waits on one page; the default is 120 seconds. Raise it for pages with interactions. crawl_timeout caps the whole crawl. Size it to your page limit.

Crawl timeout

params = [{
		"url": "https://www.example.com",
		"limit": 20,
		"request": "browser",
		"return_format": "markdown",
	    "crawl_timeout": {
            "secs": 120,
            "nanos": 0
		}
	}
}

Client timeouts

Your HTTP client can time out too. With Python's requests library, pass separate connection and read timeouts.

Client timeout

import requests, os

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

CONNECTION_TIMEOUT = 15
READ_TIMEOUT = 30

params = {
	"url": "https://www.example.com",
	"limit": 30,
	"depth": 3,
	"request": "smart",
	"return_format": "markdown"
}

response = requests.post(
    'https://api.spider.cloud/crawl',
    headers=headers,
    json=params,
	stream=True, # Applies to time between consecutive data chunks in streaming mode
	timeout=(CONNECTION_TIMEOUT, READ_TIMEOUT)
)

print(response.json())

Tip

For single-page requests, a client read timeout of 60 secs with one immediate retry keeps a slow page from holding up the queue. Tune both to your volume and page count.

Concurrency and streaming

Pages fetch concurrently, up to 10,000 requests per minute. Pair that with streaming to read results as they arrive.