Web Unblocker API
Some websites go beyond standard protections with advanced anti-bot systems, CAPTCHAs, and browser fingerprint checks. The Unblocker applies specialized techniques to access this content when standard crawling can't get through.
What Gets Bypassed
Spider's Unblocker navigates through multiple defense layers automatically.
When to Use Unblocker
Standard Crawl / Scrape
Works for most websites. Spider's Smart mode handles JS rendering, common bot detection, and standard protections. Start here — it's cheaper and faster.
- ✓Standard JS-rendered sites
- ✓Basic rate limiting / IP blocks
- ✓Common Cloudflare / CDN protections
Unblocker
For sites where standard requests return blocks, challenges, or empty content. Costs 10-40 additional credits per successful request.
- ✓Advanced CAPTCHA / challenge pages
- ✓Sophisticated fingerprint checks
- ✓Sites that actively detect bots
How It Works
Key Capabilities
If the first attempt is blocked, Spider automatically retries with different fingerprints, proxy paths, and timing patterns to maximize success rate.
Maintain browser sessions across requests with session. Once a site is unblocked, subsequent requests reuse the authenticated session state.
Combine with residential, mobile, or ISP proxies for maximum stealth. Geo-target requests so they originate from the right country.
Accepts the same parameters as crawl and scrape. Switch between endpoints by changing the path — no code restructuring needed.
The additional 10-40 credits are only charged on successful unblocking. If Spider can't get through, you don't pay the premium.
Run JavaScript on the page after unblocking with evaluate_on_new_document. Interact with elements, expand content, or trigger actions.
Credit Cost
Standard crawl credits
Per successful unblock
If unblocking fails
Code Examples
from spider import Spider
client = Spider()
# Access a site with heavy bot protection
result = client.unblocker(
"https://protected-site.com/data",
params={
"return_format": "markdown",
"proxy_enabled": True,
"country_code": "us",
}
)
print(result[0]["content"])curl -X POST https://api.spider.cloud/unblocker \
-H "Authorization: Bearer $SPIDER_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://protected-site.com",
"return_format": "markdown",
"session": true,
"fingerprint": true,
"proxy_enabled": true
}'import Spider from "@spider-cloud/spider-client";
const client = new Spider();
const result = await client.unblocker("https://protected-site.com/data", {
return_format: "markdown",
proxy_enabled: true,
country_code: "us",
});
console.log(result[0].content);