Browser automation

Playwright, Puppeteer and Selenium with authenticated proxies.

Headless browsers are the fiddliest proxy clients, because Chromium's --proxy-server flag does not accept credentials. Each framework works around this differently.

Playwright

import { chromium } from "playwright";

const browser = await chromium.launch({
  proxy: {
    server: "http://res.wproxy.io:8000",
    username: "wp-acc4821-country-us-session-a91f",
    password: "s3cr3t-pass",
  },
});

const page = await browser.newPage();
await page.goto("https://example.com");
await browser.close();

Per-context proxies

Launching a browser per exit is expensive. Playwright lets you set a proxy per browser context, so one browser process can hold dozens of independent sessions — pass a different -session- key to each context.

javascript

const browser = await chromium.launch();

const contexts = await Promise.all(
  ["us", "gb", "de"].map((country) =>
    browser.newContext({
      proxy: {
        server: "http://res.wproxy.io:8000",
        username: `wp-acc4821-country-${country}-session-${country}1`,
        password: "s3cr3t-pass",
      },
    }),
  ),
);

Puppeteer

Puppeteer takes the server as a launch flag and the credentials through page.authenticate. Both are required; supplying only one silently produces 407s.

javascript

import puppeteer from "puppeteer";

const browser = await puppeteer.launch({
  args: ["--proxy-server=http://res.wproxy.io:8000"],
});

const page = await browser.newPage();
await page.authenticate({
  username: "wp-acc4821-country-us-session-a91f",
  password: "s3cr3t-pass",
});

await page.goto("https://example.com", { waitUntil: "networkidle2" });
await browser.close();

Selenium

Selenium has no credential hook at all. The usual approach is a tiny local forwarder — mitmproxy, or a generated Chrome extension — that injects the Proxy-Authorization header on the browser's behalf.

python

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import seleniumwire.undetected_chromedriver as uc  # pip install selenium-wire

options = {
    "proxy": {
        "http": "http://wp-acc4821-country-us:s3cr3t-pass@res.wproxy.io:8000",
        "https": "http://wp-acc4821-country-us:s3cr3t-pass@res.wproxy.io:8000",
        "no_proxy": "localhost,127.0.0.1",
    }
}

driver = uc.Chrome(seleniumwire_options=options)
driver.get("https://example.com")

Browsers are expensive per gigabyte

A headless page load pulls images, fonts and analytics you almost certainly do not need, and you pay for all of it. Blocking non-essential resource types typically cuts bandwidth by 60–80% with no effect on the data you are after.

javascript

await page.route("**/*", (route) => {
  const type = route.request().resourceType();
  return ["image", "media", "font", "stylesheet"].includes(type)
    ? route.abort()
    : route.continue();
});

Scraping frameworks

API overview

Something inaccurate?Tell support