Spider vs. Crawlera (Zyte): Predictable Pricing, Full Browser Control

Crawlera was one of the original smart proxy managers. Zyte deprecated it in 2023 and folded it into the Zyte API, which combines proxy rotation, browser rendering, and AI extraction into a single service.

If you’re migrating from Crawlera or evaluating Zyte, here’s how it compares to Spider.

The Scrapy connection

Zyte maintains Scrapy, the most popular Python scraping framework. That gives them deep roots in the Python community and a natural upgrade path: start with Scrapy, hit a wall with anti-bot detection, upgrade to Zyte API.

Spider takes a different approach. Instead of a framework you host and maintain, Spider is infrastructure: a Rust-powered API that crawls pages and returns clean data, plus a browser automation SDK for interactive workflows. No framework to deploy, no Scrapy middleware to configure.

Zyte’s complexity-tier pricing

Zyte charges per request, but the cost depends on which of five “website complexity” tiers a site falls into:

You don’t choose the tier. Zyte classifies each website automatically. A site that was “Moderate” last month might become “Complex” after updating its bot protection, and your costs jump 2.6x with no change to your code.

Spider bills based on bandwidth and compute time — there are no complexity tiers and no credit multipliers. Your cost scales naturally with page size and processing time, not with what anti-bot system a site runs. A typical workload averages around $0.65 per 1,000 pages. See spider.cloud/credits for the full breakdown.

Browser control

Zyte is a request-response API. You send a URL and get HTML or extracted data back. You can’t interact with the page.

That means you can’t:

• Fill a login form and scrape authenticated content
• Click through multi-page wizards
• Scroll through infinite feeds or expand collapsed sections
• Take a screenshot mid-workflow
• Build an agent that navigates based on what it finds

Spider Browser gives you a live browser session. Here’s a real-world example, extracting job listings that require clicking through filters and pagination:

import { SpiderBrowser } from "spider-browser";

const spider = new SpiderBrowser({
  apiKey: process.env.SPIDER_API_KEY,
});

await spider.init();
await spider.page.goto("https://jobs.example.com");

// Use the AI act() method to interact naturally
await spider.page.act("Click the 'Remote' filter and select 'Engineering' department");
await spider.page.act("Sort by 'Most Recent'");

// Extract structured data from the filtered results
const jobs = await spider.page.extract(
  "Extract all visible job listings with title, location, salary range, and posted date as a JSON array"
);

console.log(jobs);
await spider.close();

Zyte would give you the initial, unfiltered page HTML.

Rate limits

Zyte caps standard plans at 500-1,400 requests per minute. Spider’s API handles 50,000+ requests per minute with no per-user rate limit on paid plans. If you’re crawling at scale (sitemaps, product catalogs, news feeds), the throughput difference matters.

AI extraction pricing

Both Spider and Zyte offer AI-powered data extraction, but the pricing models are different.

Zyte charges per token: $0.002 per 1,000 input tokens + $0.01 per 1,000 output tokens, on top of the base request cost. On a content-heavy page, the AI extraction can cost more than the request itself.

Spider Browser’s extractFields() pulls structured data from the DOM in a single call with no additional token charges. The extract() method handles natural-language queries the same way, with no per-token surcharge.

Side-by-side

Zyte

import requests

response = requests.post(
    "https://api.zyte.com/v1/extract",
    auth=("YOUR_API_KEY", ""),
    json={
        "url": "https://example.com",
        "browserHtml": True,
    },
)
html = response.json()["browserHtml"]
# Now parse the HTML yourself, or pay extra for AI extraction

Spider

import requests

response = requests.post(
    "https://api.spider.cloud/crawl",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "url": "https://example.com",
        "return_format": "markdown",
    },
)
# Clean markdown, ready for LLM consumption
content = response.json()[0]["content"]

Spider returns markdown by default. No parsing library, no extra extraction step. If you need structured JSON, the Browser SDK’s extractFields() and extract() methods handle that directly.

When Zyte still makes sense

If you’re already deep in the Scrapy ecosystem with production spiders deployed, Zyte is a natural fit since it’s built around that workflow. Zyte’s pre-built AI models for product pages are also solid if e-commerce extraction is your primary use case.

But if you need browser automation, predictable pricing, or throughput beyond 1,400 RPM, Spider is a better foundation.

• Spider API docs
• Spider Browser SDK
• 1,004 ready-to-use scrapers