Frequently Asked Questions (FAQ)
Need help with getting started? Check some of these common questions first in the FAQ.
What is Spider?
Spider is a leading web crawling tool designed for speed and cost-effectiveness, supporting various data formats including LLM-ready markdown.
How can I try Spider?
Purchase credits for our cloud system or test the Open Source Spider engine to explore its capabilities.
What are the rate limits?
Everyone has access to 50,000 requests per second for the core API.
Is Spider suitable for large scraping projects?
Absolutely, Spider is ideal for large-scale data collection and offers a cost-effective dashboard for data management.
Can you crawl all pages?
Yes, Spider accurately crawls all necessary content without needing a sitemap.
What formats can Spider convert web data into?
Spider outputs HTML, raw, text, and various markdown formats. It supports JSON, JSONL, CSV, and XML for API responses.
Does it respect robots.txt?
Yes, compliance with robots.txt is default, but you can disable this if necessary.
Am I billed for failed requests?
We do not charge for any failed request on our endpoints.