Crawling Authenticated Pages

Dashboards, member areas and internal tools sit behind a login wall, so a plain crawl of those URLs returns the sign-in page instead of the content you wanted. Spider supports two methods for crawling authenticated pages:

1. Cookies: Pass a session cookie directly in the request.
2. Execution scripts: Run custom JavaScript to log in through a form.

Both run in browser mode, so the page renders exactly as it would in a logged-in tab.

Setting a Cookie

If you already have a valid session cookie (from your browser dev tools or a login API), pass it with the cookies parameter:

import requests, os

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

response = requests.post('https://api.spider.cloud/crawl',
  headers=headers,
  json={
    "url": "https://example.com/dashboard",
    "return_format": "markdown",
    "cookies": "session_id=abc123; auth_token=xyz789",
    "request": "browser"
  }
)

for page in response.json():
    print(page['url'], len(page.get('content', '')), 'chars')

To get the cookie value, log into the site in your browser, open dev tools (F12), go to the Application tab, and copy the relevant cookies.

Using Execution Scripts

The execution_scripts parameter runs JavaScript on the page before Spider extracts content. This is useful for filling in login forms programmatically.

import requests, os

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

login_script = """
document.querySelector('#username').value = 'your_username';
document.querySelector('#password').value = 'your_password';
document.querySelector('form').submit();
"""

response = requests.post('https://api.spider.cloud/crawl',
  headers=headers,
  json={
    "url": "https://example.com/login",
    "return_format": "markdown",
    "request": "browser",
    "return_cookies": True,
    "execution_scripts": {
      "https://example.com/login": login_script
    }
  }
)

print(response.json())

The script runs after the page loads in headless Chrome. After the form submits, Spider follows the redirect and crawls the authenticated pages. Set return_cookies: true to capture the session cookies for subsequent requests.

Which method should you use?

Reach for cookies when you already hold a valid session, from your browser or from a login API you control. It is one parameter and there is no form to drive.

Reach for execution scripts when the session has to be created as part of the crawl, or when the cookie expires faster than you can refresh it by hand. Setting return_cookies: true gives you the session back, so a script-based login can seed the cookie approach for later requests.

Tips for crawling behind a login

• Use browser mode: Authentication flows require JavaScript rendering. Always set request: "browser" (the legacy chrome value still works).
• Cookie format: Pass cookies as a semicolon-separated string, matching the format from browser dev tools.
• wait_for: If the login redirects to a page that loads dynamically, use wait_for to wait for a specific element before extraction.
• Scope with whitelist/blacklist: After login, control which paths Spider crawls using the whitelist and blacklist parameters.

For all available parameters, see the API reference.