Best practices
What separates a pipeline that works from one that constantly breaks.
Match the pool to the target
Start on datacenter and escalate only where it fails. Teams that default to residential for everything routinely overspend by an order of magnitude on targets that never inspected the network in the first place.
Rotate at the right granularity
- Stateless fetching: full rotation, no session key.
- Anything with a login or a cart: one session key per logical user, held for the length of the journey.
- Paginated results: one session for the whole pagination run, or you will get inconsistent ordering.
- Retries after a block: always a new session key. Retrying the same exit that just got blocked accomplishes nothing.
Be honest about headers
A German exit sending Accept-Language: en-US with a New York timezone is more suspicious than a datacenter address would have been. If you are going to the trouble of geo-targeting, make the rest of the request agree with it.
Cut bandwidth you do not need
- Send Accept-Encoding: gzip, br — compressed responses cost proportionally less.
- Block images, fonts, media and stylesheets in headless browsers.
- Prefer a site's JSON endpoints over its rendered HTML where they exist; the payload is often a twentieth of the size.
- Do not re-fetch pages whose ETag or Last-Modified has not changed.
Instrument before you scale
Log the x-wproxy-exit-ip and x-wproxy-exit-city headers alongside every response. When success rates fall, that log is the difference between a five-minute diagnosis and a day of guessing.
Crawl politely
Concurrency limits exist on the target side too. A crawl that flattens a small site's origin is both rude and self-defeating — you will be blocked at the network level, and no proxy pool fixes that.