Crawling authenticated pages with cookies or scripts

Dashboards, member areas and internal tools sit behind a login wall, so a plain crawl of those URLs returns the sign-in page instead of the content you wanted. Spider Cloud supports two methods for crawling authenticated pages:

1. Cookies: Pass a session cookie directly in the request.
2. Execution scripts: Run custom JavaScript to log in through a form.

Both run in browser mode, so the page renders exactly as it would in a logged-in tab.

Setting a cookie

If you already have a valid session cookie (from your browser dev tools or a login API), pass it with the cookies parameter:

import requests, os

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

response = requests.post('https://api.spider.cloud/crawl',
  headers=headers,
  json={
    "url": "https://example.com/dashboard",
    "return_format": "markdown",
    "cookies": "session_id=abc123; auth_token=xyz789",
    "request": "browser"
  }
)

for page in response.json():
    print(page['url'], len(page.get('content', '')), 'chars')

To get the cookie value, log into the site in your browser, open dev tools (F12), go to the Application tab, and copy the relevant cookies.

Using execution scripts

The execution_scripts parameter runs JavaScript on the page before Spider extracts content. This is useful for filling in login forms programmatically.

import requests, os

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

login_script = """
document.querySelector('#username').value = 'your_username';
document.querySelector('#password').value = 'your_password';
document.querySelector('form').submit();
"""

response = requests.post('https://api.spider.cloud/crawl',
  headers=headers,
  json={
    "url": "https://example.com/login",
    "return_format": "markdown",
    "request": "browser",
    "return_cookies": True,
    "execution_scripts": {
      "https://example.com/login": login_script
    }
  }
)

print(response.json())

The script runs after the page loads in headless Chrome. After the form submits, Spider follows the redirect and crawls the authenticated pages. Set return_cookies: true to capture the session cookies for subsequent requests.

Which method should you use?

Reach for cookies when you already hold a valid session, from your browser or from a login API you control. It is one parameter and there is no form to drive.

Reach for execution scripts when the session has to be created as part of the crawl, or when the cookie expires faster than you can refresh it by hand. Setting return_cookies: true gives you the session back, so a script-based login can seed the cookie approach for later requests.

Tips for crawling behind a login

• Use browser mode: Authentication flows require JavaScript rendering. Always set request: "browser" (the legacy chrome value still works).
• Cookie format: Pass cookies as a semicolon-separated string, matching the format from browser dev tools.
• wait_for: If the login redirects to a page that loads dynamically, use wait_for to wait for a specific element before extraction.
• Scope with whitelist/blacklist: After login, control which paths Spider crawls using the whitelist and blacklist parameters.

When a login is worth doing another way

Some sign-in flows are not worth driving from a crawl. Two-factor prompts, one-time email codes and captchas need a person or a second system to answer them, and a script that fills a form cannot. If the site issues API tokens, ask for one and skip the browser login entirely.

Sessions also expire in the middle of long jobs. The symptom is easy to miss: pages come back with a 200 status and the content of the sign-in page. Check for that marker in the returned markdown, refresh the cookie, and resume from the URL that failed.

For all available parameters, see the API reference.