POST /crawl

Crawl

Point Spider at any URL and it recursively discovers every page on the domain. Results stream back as they are found, so you can process pages before the full crawl completes. Set depth limits, page caps, and output format to control exactly what you get back.