Common questions
Billing, rate limits, output formats, robots.txt, and what happens when a crawl fails.
What is Spider?
The data infrastructure for AI. Spider indexes any website and returns clean structured data as markdown, HTML, JSON, or plain text, for agents, RAG pipelines, and LLMs.
How can I try Spider?
Sign up for a free balance to test, or explore the open-source Spider engine on GitHub.
Does my balance expire?
No. There is no monthly subscription. Top up once and use the balance whenever you need it.
What if I need to top up more?
Top up any amount, any time. Single purchases of $500 or more include an automatic bonus, and the exact percentage for each amount is shown on the top up form. No negotiations.
What are the rate limits?
Up to 10,000 core API requests per minute by default. Contact us if you need higher throughput.
What output formats are supported?
HTML, raw, plain text, and several markdown formats. API responses also support JSON, JSONL, CSV, and XML.
Can Spider crawl all pages?
Yes. Spider crawls all necessary content without needing a sitemap. We rate-limit individual URLs per minute to avoid overloading the target server.
Does it respect robots.txt?
Yes. robots.txt compliance is on by default. You can disable it on a per-request basis when needed.
What if a crawl fails?
Failed requests are billed at $0. You only pay for responses that return data.
What if I get blocked?
Spider includes an Unblocker with stealth, rotating proxies, and automatic retries. Heavily protected sites route to the Browser Cloud, which runs full browser sessions with anti-detection built in.
How does billing work?
Each request is billed for bandwidth ($1/GB) plus compute ($0.001/min). Most pages cost a fraction of a cent. Estimate your spend with the calculator at /compare.