Why patched headless browsers keep breaking on modern sites

If you collect public web data at any real scale, you have met pages that load fine in your own browser and come back empty, or as a challenge page, from your scraper. Those sites are not broken. They are checking whether the thing asking for the page is a browser, and most scraping stacks fail that check in ways their authors never see.

This post covers what those checks look like at the connection, in the page, and over time. It then goes into why the standard fix, a headless browser with patches layered on top, has a shelf life measured in weeks, and why a browser engine built for automation from the start does not have that problem.

What a demanding site checks

Protection services differ in where they sit and what they weigh, but the signals they read fall into three groups.

At the connection

Every request carries evidence about the client before a byte of HTML moves.

• IP reputation. Datacenter ranges, known proxy providers, and addresses with a history of abusive traffic get flagged at once.
• TLS handshake. The ClientHello carries a particular combination of cipher suites, extensions, and supported curves. Services hash it and compare it against known browser profiles. If your HTTP client’s handshake looks like python-requests/2.31 instead of Chrome 124, the request is scored accordingly.
• HTTP/2 behavior. SETTINGS frames, WINDOW_UPDATE parameters, header order, and pseudo-header order. Real browsers produce consistent, predictable patterns. Most HTTP libraries do not.
• Header consistency. Missing headers, unusual order, or headers that contradict the claimed User-Agent all count against the request.

Many services turn these into a per-request score and let the site operator set thresholds: block the lowest scores, challenge the middle, let the rest through. The defaults are strict enough to catch naive automation.

In the page

When the connection-level score is ambiguous, the site serves a JavaScript challenge. Most of these are invisible. There is no checkbox or image grid. The script runs, collects what it needs, and either passes the session or fails it.

What it collects is the browser environment itself. Canvas and WebGL output, where the pixel data read back from a rendered image differs across GPU drivers, font engines, and anti-aliasing implementations. The WebGL renderer string, supported extensions, and shader precision formats. The output of a short signal processed through the Web Audio API. navigator.hardwareConcurrency and navigator.deviceMemory, so that a headless instance claiming to be a phone while reporting 32 cores and 64GB of RAM is an obvious contradiction. Installed fonts, screen dimensions, timezone, language, and dozens of other API outputs. It also measures execution timing, and a browser on real hardware produces consistent timing where a headless browser inside a container on a server does not.

Then come the consistency checks. Is navigator.webdriver present? Does window.chrome exist with the expected properties? Does Notification.permission return a plausible value? Have native prototypes been overridden, getters replaced, or toString() on built-in functions changed? Are there elements in the DOM that an automation framework left behind? Puppeteer and Playwright change the JavaScript environment in detectable ways, even with patch plugins applied, and every patch is a place where the browser’s story and its engine disagree.

Some operators go further and force a JavaScript challenge on every single request. It adds latency for real visitors, so it is usually reserved for login pages and checkout flows.

Over time

Once a session is in, behavioral models watch it. Mouse velocity, acceleration, jitter, and curvature. Scroll speed, direction changes, and momentum. Inter-key timing and hold duration. On mobile, touch pressure, contact area, and gesture shape. Humans follow biomechanical patterns, and synthetic input (linear interpolation, bezier curves at uniform speed) is statistically distinguishable from them.

Two session-level checks matter most. If two sessions produce identical behavioral patterns, one of them is synthetic. And the time between page load and first interaction, between clicks, and between navigations tends to run at machine speed in automation, faster than any person.

Some services add a proof-of-work step when they are unsure. The browser has to finish a hash computation before the request goes through, and the difficulty scales with suspicion: around 100ms for a mildly suspicious session, several seconds for a highly suspicious one. That adds cost to large-scale automation, and it tests whether the client has the compute you would expect from a real browser on real hardware.

All of this telemetry streams to the service’s backend in real time, and the models retrain continuously on fresh data. A pattern that passes today has a short shelf life.

Why patches on a headless browser stop working

The common fixes each address one of the groups above, and each stops working for the same reason.

Header rotation and TLS matching

The first thing most developers try is rotating User-Agent strings and matching request headers to a real browser. The next step is a library such as curl-impersonate or tls-client that reproduces the exact TLS handshake of a specific browser version.

Why it breaks. Services read well beyond TLS. Matching the handshake of Chrome 124 is necessary but not sufficient. You also need Chrome 124’s HTTP/2 SETTINGS frame, its HPACK behavior, its pseudo-header order, and a dozen other protocol details that change with every browser release.

Patch plugins on headless Chrome

Puppeteer and Playwright both have plugin ecosystems that patch the known checks: remove navigator.webdriver, fill in window.chrome, fix the Permissions API, normalize WebGL output.

Why it breaks. The patches become signatures. When a plugin removes navigator.webdriver, it does so in a way that is subtly different from a browser where the flag was never present, and the sensor scripts test for exactly that difference. A new patch works for a few weeks before the service learns its pattern, and then you need another one.

The deeper problem is structural. A headless build of Chrome was never meant to pass for a person’s browser, and every patch adds another place where what the browser claims and what the engine does can disagree. You are not closing a gap. You are creating new ones faster than you can find them.

Residential proxies

Residential proxies route traffic through consumer IP addresses, which get higher trust than datacenter ranges.

Why it breaks. A clean IP addresses one dimension. If the TLS handshake, the browser environment, or the behavior is flagged, the IP does not save you. Providers also recycle addresses across customers, so another customer’s aggressive crawl can burn an IP before your request reaches it.

Cost is the other issue. Residential bandwidth runs $5 to $15 per gigabyte depending on provider and geography. At scale, proxy costs often exceed the cost of the scraping infrastructure itself.

Outsourced challenge solving

Services exist that take a visual challenge, have a human or a model answer it, and return a token.

Why it breaks. Most modern challenges are invisible. There is no image to classify, because the challenge is the browser environment itself, and no third party can supply that on your behalf. Where a visual challenge does appear, the round trip of forwarding it, solving it, and returning the token (typically 10 to 30 seconds) is often enough on its own to flag the session. A person clears a simple challenge in 2 to 5 seconds. A session that takes 25 seconds looks automated.

The maintenance bill

Suppose you assemble a patched stack that works today. Here is what keeping it working looks like.

Update frequency. Some services ship detection updates weekly, some biweekly, and some retrain continuously. Your patches have a half-life measured in days.

Combinatorial complexity. Different sites configure different thresholds, different strictness, and different fallback behavior. A setup that works for one site’s configuration of a given service fails on another site using the same service. You end up maintaining per-site configurations.

Infrastructure cost. Headless browsers at scale take a lot of compute. Add residential proxies, outsourced solving, and the engineering time to keep the patches current, and the total cost per page often exceeds what a managed service charges.

Risk concentration. One mistake, such as a leaked profile pattern, a proxy provider that gets flagged, or a plugin update that introduces a new tell, can burn your entire IP pool and browser profile set. Recovery means rebuilding from scratch.

If you collect from one or two sites you know well, this is manageable and gives you full control. Across hundreds of domains with varying protection, per-site maintenance becomes untenable.

A browser that does not need patching

We took the other route. Spider Browser is a browser runtime written in Rust that drives Chromium, Gecko, and WebKit from one process. Chrome, Firefox, Safari, Brave, and Edge are profiles it presents, and each profile is built from the engine that renders the page. There is no story to keep straight, because the TLS handshake, the HTTP/2 frames, the JavaScript environment, and the rendered output all come from the same place. A site sees a browser because it is one.

That is why none of the checks above need a workaround. The handshake and HTTP/2 behavior match a consumer browser down to the SETTINGS frame and header order because they are produced by a browser. The environment checks pass because nothing was patched. Each session keeps one consistent profile from its first request to its last, and no two sessions share one, so behavioral models see what they expect from a real visitor.

How smart mode works

Spider’s default smart mode inspects each target URL and picks the cheapest, fastest path that will load the page completely:

1. Static fetch. If the page serves its content without JavaScript, Spider uses a lightweight HTTP fetch. This is the fastest and cheapest path.
2. Browser render. If the page needs JavaScript to produce its content, Spider renders it in the native browser.
3. Full profile. If the page still does not load completely, Spider retries with the full native browser profile and the premium network, and tries a different configuration on each failure.

You do not need to know what a site runs or configure anything per site. Spider detects and adapts.

What that looks like in code

import requests, os

response = requests.post(
    "https://api.spider.cloud/scrape",
    headers={
        "Authorization": f"Bearer {os.getenv('SPIDER_API_KEY')}",
        "Content-Type": "application/json",
    },
    json={
        "url": "https://protected-site.example.com",
        "return_format": "markdown",
    },
)

for page in response.json():
    print(page["content"])

Twelve lines, with no proxy configuration, patch plugins, retry logic, or profile management in them. Spider loads the page, retries on failure, and returns clean content.

Where each side wins

A managed browser is not better on every axis.

When you should run your own browsers

There are real reasons to keep the stack in house:

• Single-site, deep expertise. If you collect from one or two sites you deeply understand, your own stack gives you full control and full visibility. You can tune per page, debug by inspecting every request, and react to changes on your own timeline rather than waiting for a vendor.
• Security research and auditing. Understanding the detection mechanism is the goal. A managed service hides the details you need to study.
• Data residency and compliance. Your traffic routes through your own infrastructure. No third-party network touches the data.
• Cost at single-site scale. For one well-understood site at high volume, a tuned setup with curl-impersonate or Playwright can be cheaper than any managed API, because you pay only for compute and proxies with no per-page fee.

Loading a page completely is infrastructure, not application logic. Whether you build it or buy it, your application code should not contain if site_a: do_x(); elif site_b: do_y(). That belongs in a layer below your application, maintained by people whose job is keeping up with how sites change.

The question is whether that maintenance is a good use of your engineering team’s time. For most teams collecting across many domains, it is not. For teams focused on one or two high-value sites, it might be.