Scraping frameworks

Scrapy and other high-throughput crawler configuration.

Scrapy

Scrapy honours the standard proxy meta key. The cleanest integration is a downloader middleware that attaches a fresh session per request, so retries land on different exits automatically.

middlewares.py

import uuid

class WproxyMiddleware:
    GATEWAY = "http://res.wproxy.io:8000"
    USER = "wp-acc4821"
    PASSWORD = "s3cr3t-pass"

    def process_request(self, request, spider):
        country = request.meta.get("wproxy_country", "us")
        session = uuid.uuid4().hex[:10]
        user = f"{self.USER}-country-{country}-session-{session}"

        request.meta["proxy"] = self.GATEWAY
        request.headers["Proxy-Authorization"] = basic_auth_header(user, self.PASSWORD)

settings.py

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.WproxyMiddleware": 350,
}

CONCURRENT_REQUESTS = 64
CONCURRENT_REQUESTS_PER_DOMAIN = 16
DOWNLOAD_TIMEOUT = 30
RETRY_TIMES = 3
RETRY_HTTP_CODES = [403, 408, 429, 500, 502, 503, 504]

Do not set AUTOTHROTTLE against a proxy pool

AutoThrottle infers server load from latency, and proxy latency varies for reasons that have nothing to do with the target. It will throttle you to a crawl. Set explicit concurrency instead.

httpx and aiohttp

import httpx

proxy = "http://wp-acc4821-country-us:s3cr3t-pass@res.wproxy.io:8000"

async with httpx.AsyncClient(proxy=proxy, timeout=30) as client:
    r = await client.get("https://example.com")

Code examples

Browser automation

Something inaccurate?Tell support