How to rotate proxies in Python without making a mess
Session handling, retry policy and connection pooling patterns that survive contact with a real crawl.
Jonas Okonkwo
Developer relations
04 Aug 2026 · 11 min read
Most rotation tutorials show you a list of proxy strings and random.choice. That works for a demo and falls apart in production, where you need per-request sessions, sane retries, and connection reuse that does not accidentally pin you to one exit for an hour.
Rotation is server-side
The first thing to internalise: with a gateway-based provider you do not maintain a proxy list at all. There is one endpoint. Rotation happens because the gateway picks a different exit per request, unless you tell it not to.
import requests
PROXY = "http://wp-acc4821-country-us:pass@res.wproxy.io:8000"
PROXIES = {"http": PROXY, "https": PROXY}
for url in urls:
r = requests.get(url, proxies=PROXIES, timeout=30) # new exit each timeWhen you need the exit to stay put
Anything stateful — logins, carts, multi-step forms, paginated results — needs a sticky session. Generate a key, hold it for the length of the logical journey, discard it afterwards.
import uuid, requests
from contextlib import contextmanager
@contextmanager
def sticky(country="us", minutes=30):
key = uuid.uuid4().hex[:12]
user = f"wp-acc4821-country-{country}-session-{key}-sesstime-{minutes}"
proxy = f"http://{user}:pass@res.wproxy.io:8000"
s = requests.Session()
s.proxies = {"http": proxy, "https": proxy}
try:
yield s
finally:
s.close()
with sticky(country="de") as s:
s.post("https://example.com/login", data=creds)
for page in range(1, 11):
s.get(f"https://example.com/results?page={page}")Retries that do not fight the gateway
urllib3's Retry is fine for transient network faults but it retries on the same connection, which means the same exit. For block-driven failures that is precisely wrong. Handle those at a level where you can change the session key.
import time, uuid, requests
BLOCKED = {403, 429, 503}
def fetch(url, country="us", attempts=4):
for i in range(attempts):
key = uuid.uuid4().hex[:12] # fresh exit every attempt
user = f"wp-acc4821-country-{country}-session-{key}"
proxy = f"http://{user}:pass@res.wproxy.io:8000"
try:
r = requests.get(url, proxies={"https": proxy}, timeout=30)
if r.status_code not in BLOCKED:
return r
except requests.RequestException:
pass
time.sleep(min(2 ** i, 8))
raise RuntimeError(f"exhausted retries for {url}")Do not retry authentication failures
A 407 means your credentials are wrong. Retrying it four times per URL across a million-URL crawl will trip our brute-force protection and lock the sub-user, which is a genuinely annoying way to spend an afternoon.
Concurrency without thrashing
Residential pools have no concurrency limit on our side, but your own machine does. Bound the worker count, and give each worker its own Session so connection pooling actually helps.
import asyncio, uuid, httpx
async def worker(queue, results, country="us"):
key = uuid.uuid4().hex[:12]
user = f"wp-acc4821-country-{country}-session-{key}"
proxy = f"http://{user}:pass@res.wproxy.io:8000"
async with httpx.AsyncClient(proxy=proxy, timeout=30) as client:
while True:
url = await queue.get()
try:
r = await client.get(url)
results.append((url, r.status_code, len(r.content)))
finally:
queue.task_done()
async def crawl(urls, workers=32):
queue, results = asyncio.Queue(), []
for u in urls:
queue.put_nowait(u)
tasks = [asyncio.create_task(worker(queue, results)) for _ in range(workers)]
await queue.join()
for t in tasks:
t.cancel()
return resultsNote what this does: each worker holds one sticky session for its lifetime, so connections are reused, but thirty-two workers means thirty-two concurrent exits. That is usually the right shape — full per-request rotation defeats connection pooling and adds a TLS handshake to every fetch.
Log the exit
r = requests.get(url, proxies=PROXIES)
log.info(
"fetch",
extra={
"url": url,
"status": r.status_code,
"exit_ip": r.headers.get("x-wproxy-exit-ip"),
"exit_city": r.headers.get("x-wproxy-exit-city"),
"rotated": r.headers.get("x-wproxy-session-rotated"),
},
)When success rates drop three weeks from now, this log is the difference between a five-minute diagnosis and a day of speculation.