Spider vs. Crawlera (Zyte): Predictable Pricing, Full Browser Control

Crawlera was one of the original smart proxy managers. Zyte deprecated it in 2023 and folded it into the Zyte API, which combines proxy rotation, browser rendering, and AI extraction into a single service.

If you’re migrating from Crawlera or evaluating Zyte API, here’s how it compares to Spider.

The Scrapy connection

Zyte maintains Scrapy, the most popular Python scraping framework. If you’ve written scraping code in Python, there’s a good chance you’ve used it. That gives Zyte a natural upgrade path: start with Scrapy locally, hit a wall with anti-bot detection, upgrade to Zyte API with their Scrapy middleware.

Spider skips the framework layer entirely. Instead of deploying and maintaining a Scrapy project, you call an API. Here’s the same task, scraping a product catalog, in both approaches:

With Scrapy + Zyte API

# settings.py
ADDONS = {"scrapy_zyte_api.Addon": 500}
ZYTE_API_KEY = "YOUR_KEY"
ZYTE_API_BROWSER_HTML = True

# spider.py
import scrapy

class ProductSpider(scrapy.Spider):
    name = "products"
    start_urls = ["https://shop.example.com/products"]

    def parse(self, response):
        for product in response.css(".product-card"):
            yield {
                "name": product.css(".title::text").get(),
                "price": product.css(".price::text").get(),
            }
        next_page = response.css("a.next::attr(href)").get()
        if next_page:
            yield scrapy.Request(next_page, callback=self.parse)

You deploy this spider, manage its dependencies, handle retries and middleware configuration, and pay per-request based on Zyte’s complexity tier (more on that below).

With Spider

import requests

response = requests.post(
    "https://api.spider.cloud/crawl",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "url": "https://shop.example.com/products",
        "return_format": "markdown",
        "limit": 100,
    },
)

for page in response.json():
    print(page["url"], len(page["content"]))

No framework, no deployment, no middleware. The API handles pagination, rendering, and anti-bot detection internally. You get back clean data.

Zyte’s complexity-tier pricing

Zyte charges per request, but the cost depends on which “website complexity” tier a site falls into. HTTP-only requests and browser-rendered requests are priced on separate scales:

HTTP API requests:

Browser-rendered requests:

Source: zyte.com/pricing

You don’t choose the tier. Zyte classifies each website automatically based on its anti-bot protection. Here’s why that matters: a site that was “Moderate” ($2.44/1K) can become “Complex” ($6.47/1K) after updating its bot protection. Your costs jump 2.6x overnight. Same code, same site, same data, bigger bill. You only find out when you check your usage dashboard.

Spider’s pricing doesn’t shift based on what a target site does. You pay for the bytes transferred and the compute time used. If a site adds Cloudflare tomorrow, your per-page cost stays the same. The production average is about $0.48 per 1,000 pages (details).

AI extraction: per-token vs. included

Both Spider and Zyte offer AI-powered data extraction, but the cost models are very different.

Zyte charges per token on top of the base request cost: $0.002 per 1,000 input tokens and $0.01 per 1,000 output tokens for their “Generate” method.

Source: Zyte API reference

Here’s what that looks like in practice. A typical product page might have 3,000 input tokens (the page content) and produce 500 output tokens (the extracted fields). That’s $0.006 input + $0.005 output = $0.011 per page in AI extraction fees alone. On top of the $2.44-$6.47/1K base request cost. At 50,000 pages, the AI extraction surcharge adds $550.

Spider’s extract() method runs on the live DOM in your browser session, with no per-token billing. Extract from 50,000 pages and the cost is the same as visiting those pages without extraction. The AI is part of the SDK, not an add-on.

Browser control

Zyte API is request-response. You send a URL, get HTML or extracted data back. You can’t interact with the page: no login flows, no multi-step wizards, no scrolling through feeds.

Spider Browser gives you a live session. Here’s an example that requires interaction, extracting filtered job listings:

import { SpiderBrowser } from "spider-browser";

const spider = new SpiderBrowser({
  apiKey: process.env.SPIDER_API_KEY,
});

await spider.init();
await spider.page.goto("https://jobs.example.com");

await spider.page.act("Click 'Remote' filter, select 'Engineering' department");
await spider.page.act("Sort by 'Most Recent'");

const jobs = await spider.page.extract(
  "Extract all visible job listings with title, location, salary range, and posted date"
);

console.log(jobs);
await spider.close();

Zyte would give you the initial, unfiltered page.

Throughput

Zyte’s default rate limit is 1,400 requests per minute. Spider handles 50,000+ requests per minute with no per-plan concurrency cap. At scale (sitemaps, product catalogs, news archives), that’s the difference between a job that finishes in minutes and one that takes hours.

Where each tool fits

Zyte’s strength is the Scrapy ecosystem. If you already have production spiders deployed and you want a managed anti-bot layer that plugs into your existing code, Zyte’s Scrapy middleware integration is seamless. Their pre-built AI models for e-commerce product pages are also strong; if that’s your primary vertical, they’ve invested heavily there.

Spider’s advantages are predictable pricing (no tier shifts), browser automation for interactive workflows, included AI extraction (no per-token add-on), and throughput for large-scale crawls. If you’re starting fresh or outgrowing Zyte’s rate limits and complexity tiers, Spider is a simpler foundation.

Spider pricing. Free credits to start, no card required.