Detecting and surviving anti-bot systems

Anti-bot vendors do not have one detector; they have a scoring pipeline. Understanding roughly what feeds it tells you which problems a proxy solves and which ones it cannot touch, which in turn saves you from paying for residential bandwidth to fix a TLS fingerprint.

Layer one: network reputation

Is this address a datacenter? Has it been seen behaving badly recently? Is it a known VPN or Tor exit? This layer is fast, cheap and runs before anything else. It is also the only layer where the choice of proxy is the whole answer — a clean residential address passes it and a bulk datacenter address frequently does not.

Layer two: transport fingerprinting

The order of TLS extensions, the cipher list, the HTTP/2 SETTINGS frame and header ordering all vary by client. Python's requests library produces a fingerprint that no browser produces, so a request claiming to be Chrome while fingerprinting as OpenSSL is trivially detectable.

# curl_cffi impersonates real browser TLS and HTTP/2 fingerprints
from curl_cffi import requests

r = requests.get(
    "https://example.com",
    impersonate="chrome124",
    proxies={"https": "http://wp-acc4821-country-us:pass@res.wproxy.io:8000"},
)

Layer three: browser environment

Canvas and WebGL rendering, font enumeration, screen dimensions, navigator.webdriver, plugin lists, the presence of automation-specific globals. Headless Chrome differs from headed Chrome in dozens of measurable ways, most of which patched drivers address and some of which they do not.

Layer four: behaviour

Request timing, mouse movement, scroll patterns, navigation order. This is where distributed crawls give themselves away: a thousand different addresses all fetching the same URL sequence at the same interval is a pattern, and the address diversity actively highlights it.

• Randomise inter-request delays with real variance, not a fixed jitter window.
• Vary the navigation order across workers.
• Do not fetch every product page in catalogue order at three in the morning local time.
• Let some sessions do something other than the money URL — a search, a category page, a bounce.

A triage procedure

1. 1Fetch with a real browser through the same proxy, manually. If that works and your script does not, the problem is your client, not the address.
2. 2Compare your TLS fingerprint against a reference. If it says Python or Go, fix that first.
3. 3Check whether the block is immediate or after N requests. Immediate means fingerprint or reputation; after N means rate or behaviour.
4. 4Only after all three: consider a different pool.

We say this against our own commercial interest, and we say it constantly in support tickets: most block problems are not proxy problems. Upgrading the pool is the most expensive way to not fix them.