Common questions
Billing, rate limits, output formats, robots.txt, and what happens when a crawl fails.
What is Spider?
A fast web scraping and crawling API for AI agents, RAG pipelines, and LLMs. Spider returns clean structured data — markdown, HTML, JSON, or plain text.
How can I try Spider?
Sign up for a free balance to test, or explore the open-source Spider engine on GitHub.
Does my balance expire?
No. There is no monthly subscription. Top up once and use the balance whenever you need it.
What if I need to top up more?
Top up any amount, any time. Single purchases of $500 or more include an automatic bonus, up to 12% extra at $2K. No negotiations.
What are the rate limits?
Up to 10,000 core API requests per minute by default. Contact us if you need higher throughput.
What output formats are supported?
HTML, raw, plain text, and several markdown formats. API responses also support JSON, JSONL, CSV, and XML.
Can Spider crawl all pages?
Yes. Spider crawls all necessary content without needing a sitemap. We rate-limit individual URLs per minute to avoid overloading the target server.
Does it respect robots.txt?
Yes. robots.txt compliance is on by default. You can disable it on a per-request basis when needed.
What if a crawl fails?
Failed requests are billed at $0. You only pay for responses that return data.
What if I get blocked?
Spider includes an Unblocker with stealth, rotating proxies, and automatic retries. Heavily protected sites route to the Browser Cloud, which runs full browser sessions with anti-detection built in.
How does billing work?
Each request is billed for bandwidth ($1/GB) plus compute ($0.001/min). Most pages cost a fraction of a cent. Estimate your spend with the calculator at /compare.