Skip to main content

Route one request across the providers you already use.

Put a router object in a scrape or crawl body. Ask for a provider you already pay for to sit behind Spider's own stack, ask for it to run ahead, or set mode to off and keep the URL on Spider. The same body works against every provider, so changing one field is how you compare what each returns for a page. Send your own keys and each provider invoices you directly.

Get an API key Read the router docs

What the router object gives you.

Three things, each set by one field in the request body.

Compare providers on the same page

The request body is the same for every provider. Send it with router.provider set to zyte, then again with firecrawl, and read the two results side by side. provider is a preference rather than a pin, so costs.vendor names the provider that served each page.

Say where the provider sits

Set mode to fallback to ask for the provider behind Spider's own stack, or to first to ask for it ahead. first needs provider and token on the same request. One response comes back either way, and costs.vendor.attempts counts the provider routes tried.

Bring your own keys

Send your provider key on the request with token or credentials. Billing follows the route that ran: your key has to cover that route for the provider to invoice you directly, and a failover to a route your keys do not cover runs on Spider's pooled credentials. Set funding to own to rule that case out. The key authenticates that dispatch and goes into no response, log or webhook.

Five fields shape a routed request.

Every one is optional. Set mode to off to keep a URL on Spider's own stack; leaving the object out is not the same thing.

mode fallback

Asks for the provider to sit behind Spider's own stack. costs.vendor.attempts on the response counts the provider routes tried.

mode first

Asks for your named provider to run ahead of Spider. Needs provider and token on the same request. A key sent under credentials does not satisfy it.

mode off

Keeps the URL on Spider's own stack. No provider is asked, whatever keys the request carries.

provider a provider name

Which provider you would rather have. It is a preference, so another eligible provider may still serve the page. costs.vendor names the one that did.

token or credentials your provider keys

token is the single secret for the provider named in provider. credentials is a map keyed by the credential name, for sending more than one.

funding any or own

any lets any eligible route run, including one on Spider's credentials, and is the default. own keeps the request to routes your own keys can pay for. With no usable key of your own, no provider attempt runs and the request finishes on Spider.

One body, three ways to route it.

The endpoint and the request body you already send do not change. The response gains one costs.vendor object when a provider serves the page.

curl -X POST https://api.spider.cloud/scrape \
  -H "Authorization: Bearer $SPIDER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/pricing",
    "return_format": "markdown",
    "router": {
      "mode": "fallback",
      "provider": "zyte",
      "credentials": {
        "ZYTE_API_KEY": "YOUR_ZYTE_API_KEY",
        "FIRECRAWL_API_KEY": "YOUR_FIRECRAWL_API_KEY"
      }
    }
  }'
Response excerpt
[
  {
    "url": "https://example.com/pricing",
    "status": 200,
    "content": "# Pricing\n\n| Plan | Monthly | ...",
    "error": null,
    "costs": {
      "total_cost": 0.0002,
      "vendor": {
        "provider": "zyte",
        "route": "zyte.api",
        "vendor_cost": 0.0008,
        "billed_cost": 0.0002,
        "byok": true,
        "attempts": 2
      }
    }
  }
]

Values are illustrative. vendor_cost is what the provider charged; billed_cost is what Spider charged for the page; byok says whether your key paid for it; attempts counts the provider routes tried before one served the page, so 1 means the first choice worked. A page Spider served itself carries no vendor object.

Nine providers take a key on the request.

Name one in router.provider and send its key under the credential name shown. Each link opens the provider's own setup docs.

Before you route a request.

Does one request return results from more than one provider?

No. One route serves a page, so no response mixes providers. To compare providers, send the same body again with a different router.provider and read the two responses side by side.

Which providers can I route to?

Nine by key: Firecrawl, ZenRows, ScrapingBee, ScraperAPI, Bright Data, Zyte, Apify, Crawlbase, and Diffbot. Oxylabs and DataForSEO are also valid provider names, but each needs two secrets, so they run on Spider's credentials. Set funding to own and neither can run, so the request finishes on Spider.

Who bills me?

Billing follows the route that ran. When your own key covers that route, the provider invoices you at its own rate and Spider charges its markup on the page. A failover to a route your keys do not cover runs on Spider's pooled credentials and is billed at the pooled rate. Set funding to own to keep the request to routes your keys can pay for. costs.vendor shows both amounts and whether your key paid.

What happens to a key I send on a request?

Spider reads it to authenticate that one dispatch. It is never echoed in a response, sent to a webhook, or written to a log; the type that carries it cannot be serialized and prints only a count when debugged.

Can I keep one URL away from every provider?

Yes. Set mode to off on that request and Spider keeps the URL on its own stack, whatever keys the request carries. Leaving the router object out does not do this; off is the value that does.

Is provider routing on yet?

Provider routing is rolling out. Spider accepts the router object on every request today, and costs.vendor on the response names the provider that served a page. Write to support@spider.cloud to check access.

More from the API.

Scrape API

The endpoint the router object goes in. One page, any format.

Crawl API

One router body covers a whole crawl, not one page at a time.

Unblocker API

For pages that block Spider's own stack, before you reach for a provider.

Send the same request to every provider you pay for.

The endpoint and the body you already send do not change. The response gains one costs.vendor object when a provider serves the page. Set funding to own to keep a request to routes your keys can pay for, or mode to off to keep a URL on Spider. Provider routing is rolling out.

Get an API key Read the router docs