Provider router

Router options

The object and all five fields are optional. Omitting the object asks for no placement. mode set to off is the one value that keeps a URL on Spider's own stack.

Mode and provider names ignore surrounding whitespace and capitalization. mode accepts fallback, first, and off. Any other text, a blank string, or an omitted mode all mean no placement was asked for. None of them return an error. funding accepts any and own; anything else is read as any. An unknown provider name is ignored the same way. A field with the wrong JSON type, such as a number for mode or a string for credentials, is rejected.

Setup and examples

Send a provider key on the request

1. Get an API key using your provider's setup documentation below.
2. Put it in router.token with router.provider naming the provider, or in router.credentials under the credential name from the provider table.
3. Send it on every request that should be able to use it. Spider reads a key on the request to authenticate that one dispatch. It is never echoed in a response, sent to a webhook, or written to a crawl log.

A credentials map alone does not qualify a request for first; send provider and token as shown below.

Try Spider first

Set SPIDER_API_KEY in your shell before running this example. With no provider preference, the router chooses among eligible routes. Without a usable key of your own, a provider attempt can use Spider's credentials and pooled billing.

curl https://api.spider.cloud/scrape \
  -H "Authorization: Bearer $SPIDER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "return_format": "markdown",
    "router": { "mode": "fallback" }
  }'

Request a provider first

Set SPIDER_API_KEY and ZYTE_API_KEY in your environment. This Python example uses only the standard library. Spider authentication stays in the header. The provider key goes in router.token.

import json
import os
from urllib.request import Request, urlopen

body = {
    "url": "https://example.com",
    "return_format": "markdown",
    "router": {
        "mode": "first",
        "provider": "zyte",
        "token": os.environ["ZYTE_API_KEY"],
    },
}
request = Request(
    "https://api.spider.cloud/scrape",
    data=json.dumps(body).encode(),
    headers={
        "Authorization": f"Bearer {os.environ['SPIDER_API_KEY']}",
        "Content-Type": "application/json",
    },
)
with urlopen(request) as response:
    print(json.load(response))

The request still has to meet the route's capability and budget checks. first does not force a specific provider. Provider-first routing is rolling out.

Multiple keys and credential precedence

Replace these placeholders with your own keys in server-side code. Leaving out provider lets the router choose among eligible providers.

{
  "url": "https://example.com",
  "router": {
    "mode": "fallback",
    "credentials": {
      "ZYTE_API_KEY": "YOUR_ZYTE_API_KEY",
      "FIRECRAWL_API_KEY": "YOUR_FIRECRAWL_API_KEY"
    }
  }
}

For the same credential name, the highest-priority value wins:

1. router.token, mapped to the named provider.
2. router.credentials.
3. The top-level vendor_credentials map.

Keys for different providers merge.

The top-level provider and vendor_credentials fields are the older form of router.provider and router.credentials. They still work and fold into the same shape. When both forms name a provider, router.provider wins. Use the router object for new requests.

Billing follows the provider route that actually runs. Your own key must cover that route for BYOK billing, where you pay the provider directly and Spider charges its markup. Set funding to own and the request never runs on Spider's credentials; with no usable key of your own, no provider attempt runs. A provider preference alone does not change billing, and failover can reach a route using Spider's pooled credentials.

What the response reports

When an outside provider served the page, costs on that page carries a vendor object. When Spider served it, including every request with mode set to off, there is no vendor object.

{
  "costs": {
    "total_cost": 0.0010292944,
    "vendor": {
      "provider": "zyte",
      "route": "zyte.api",
      "vendor_cost": 0.0008,
      "billed_cost": 0.0002,
      "byok": true,
      "attempts": 2
    }
  }
}

provider

The provider that served the page, by its provider value.

route

The route that served the page. Treat it as an opaque label; compare it across responses, do not parse it.

vendor_cost

What the provider charged, in dollars.

billed_cost

What Spider charged for the page, in dollars.

byok

true when your own key paid for the attempt.

attempts

How many provider routes were tried before one served the page. 1 means the first choice worked.

Values above are illustrative.

Provider setup links

provider accepts eleven names: the nine below plus oxylabs and dataforseo. The nine below take a single key through router.token. The credential name column is the key to use in router.credentials. Each link opens the provider's own documentation for account and API setup.

oxylabs and dataforseo are valid provider values, but each needs a username and password pair, and a request can only carry a single secret per provider. Neither token nor credentials can authenticate them, so they do not qualify for the token-based first request. Their routes run on Spider's pooled credentials.

Provider-specific options

Put vendor-native settings in the top-level provider_options object, outside router, keyed by provider name. Consult the provider's documentation for supported settings.

{
  "url": "https://example.com",
  "router": {
    "mode": "fallback",
    "provider": "zyte",
    "token": "YOUR_ZYTE_API_KEY"
  },
  "provider_options": {
    "zyte": {
      "geolocation": "US"
    }
  }
}

Spider applies these settings only when the route uses your own provider key. It drops them on pooled routes. Options for one provider never carry over to another provider during failover.

Supply the target in url and authentication in router.token or router.credentials. Target and authentication keys inside provider_options, including url, api_key, and token, are dropped.

Troubleshooting

• If routing appears unchanged, provider routing is still rolling out. The API accepts the object before a provider is dispatched on it.
• If first does not try your provider first, check that you sent a supported provider and nonblank token. A credentials map does not substitute for that token. If those fields are correct, provider-first routing has not reached the fleet yet.
• If a token has no effect, check the provider spelling. The token contributes no key if it is blank, if the provider is unknown or missing, or if the provider requires paired credentials.
• If another provider handles the request, the preferred route may be unavailable, unsuitable for the request, or may have failed. A provider preference is not a restriction to that provider.
• If native settings have no effect, check that provider_options is outside router and that the dispatched route uses your own key.
• If a request with mode set to off still shows costs.vendor, write to support with the request id; an opt-out never dispatches.
• If funding is own and the response has no costs.vendor, no route could run on your keys and Spider served the page itself. Check the credential name against the provider table.

Continue with the common parameter reference, Scrape API , or Crawl API.