Python Proxy Configuration Examples

This page explains Python proxy configuration for routing HTTP client traffic through ProxyMesh. Find runnable scripts in proxy-examples (Python).

Jump to a topic:

Configure a proxy URL and pick a library. When you need ProxyMesh-specific headers on HTTPS, use the python-proxy-headers package to send and read headers on the proxy CONNECT request. Scrapy uses the separate scrapy-proxy-headers package.

For basic proxy routing only: set the client's proxy URL with credentials or IP authentication.

Custom headers (for example X-ProxyMesh-Country or X-ProxyMesh-IP) need python-proxy-headers (Python 3.8+) or scrapy-proxy-headers. See custom proxy headers.

See basic-proxy scripts per library in proxy-examples (Python). Those scripts do not use python-proxy-headers.

Background: HTTP proxy overview, proxy headers, and HTTPS and CONNECT.

Proxy URL and authentication

Use an http://  proxy URL for both HTTP and HTTPS targets. With username/password auth:

proxy_url = "http://USERNAME:PASSWORD@PROXYHOST:31280"

With IP authentication, omit USERNAME:PASSWORD@. Many tools honor HTTP_PROXY / HTTPS_PROXY:

export HTTPS_PROXY="http://USERNAME:PASSWORD@PROXYHOST:31280"

Python HTTP clients at a glance

Use this table to choose a client and see if python-proxy-headers or scrapy-proxy-headers can send or read ProxyMesh headers on HTTPS CONNECT. Examples cover Python Requests proxy setup, httpx proxy configuration, aiohttp proxy routing, urllib3 proxy and PycURL proxy usage, Scrapy proxy spiders, Selenium proxy WebDriver options, and custom CONNECT proxy headers. See per-library guides on Python proxy devdocs.

Basic proxy (no custom headers)

These examples route only traffic through the proxy. The examples do not set X-ProxyMesh-* on CONNECT. For country or sticky IP headers on HTTPS, use the custom proxy headers section.

Requests

Set both http and https in the proxies dictionary. See requests-proxy.py in proxy-examples.

import requests

proxy_url = "http://USERNAME:PASSWORD@PROXYHOST:31280"
proxies = {"http": proxy_url, "https": proxy_url}

response = requests.get(
    "https://api.ipify.org?format=json",
    proxies=proxies,
    timeout=30,
)
print(response.text)

httpx

Pass the proxy URL to httpx.Client. See httpx-proxy.py in proxy-examples.

import httpx

proxy_url = "http://USERNAME:PASSWORD@PROXYHOST:31280"

with httpx.Client(proxy=proxy_url, timeout=30.0) as client:
    response = client.get("https://api.ipify.org?format=json")

print(response.text)

Scrapy

Set request.meta["proxy"] on each request (see scrapy-proxy.py in proxy-examples). You may also set the http_proxy environment variable—see Scrapy environment variable below. Basic routing does not send X-ProxyMesh-* on CONNECT; use Scrapy with scrapy-proxy-headers for that.

import scrapy
from scrapy.crawler import CrawlerProcess

proxy_url = "http://USERNAME:PASSWORD@PROXYHOST:31280"

class ProxiedSpider(scrapy.Spider):
    name = "proxied_ipify"

    def start_requests(self):
        yield scrapy.Request(
            "https://api.ipify.org?format=json",
            meta={"proxy": proxy_url},
            dont_filter=True,
        )

    def parse(self, response):
        print(response.text)

if __name__ == "__main__":
    process = CrawlerProcess()
    process.crawl(ProxiedSpider)
    process.start()

Custom proxy headers (python-proxy-headers)

On plain HTTP, set ProxyMesh headers as with any request header.

On HTTPS, the client sends a proxy CONNECT first. Put custom headers on that CONNECT request. The CONNECT response returns values such as X-ProxyMesh-IP (see proxy headers).

pip install python-proxy-headers

Install the HTTP client package you use (for example requests, httpx, aiohttp, or pycurl). The python-proxy-headers package requires Python 3.8+ and is published on PyPI. Scrapy uses the separate scrapy-proxy-headers package (not python-proxy-headers).

Requests — send and read

Set the proxy with proxies . Send headers through proxy_headers  on requests_adapter . Read the assigned IP from response.headers.get("X-ProxyMesh-IP") .

from python_proxy_headers import requests_adapter

proxy_url = "http://USERNAME:PASSWORD@PROXYHOST:31280"
response = requests_adapter.get(
    "https://api.ipify.org?format=json",
    proxies={"http": proxy_url, "https": proxy_url},
    proxy_headers={"X-ProxyMesh-Country": "US"},
    timeout=30,
)

print(response.headers.get("X-ProxyMesh-IP"))

httpx — send and read

Configure httpx.Proxy  with your proxy URL. Attach send headers on the Proxy  object. Use HTTPProxyTransport  to get CONNECT response headers on the final response.

import httpx
from python_proxy_headers.httpx_proxy import HTTPProxyTransport

proxy = httpx.Proxy(
    "http://USERNAME:PASSWORD@PROXYHOST:31280",
    headers={"X-ProxyMesh-Country": "US"},
)
transport = HTTPProxyTransport(proxy=proxy)

with httpx.Client(mounts={"http://": transport, "https://": transport}) as client:
    response = client.get("https://api.ipify.org?format=json")

print(response.headers.get("X-ProxyMesh-IP"))

aiohttp — send and read

Pass the proxy URL on each request. Send headers with proxy_headers . Use ProxyClientSession  to read response.headers  after CONNECT.

import asyncio
from python_proxy_headers import aiohttp_proxy

async def main():
    async with aiohttp_proxy.ProxyClientSession() as session:
        async with session.get(
            "https://api.ipify.org?format=json",
            proxy="http://USERNAME:PASSWORD@PROXYHOST:31280",
            proxy_headers={"X-ProxyMesh-Country": "US"},
        ) as response:
            print(response.headers.get("X-ProxyMesh-IP"))

asyncio.run(main())

urllib3 — send and read

Use ProxyHeaderManager  for the proxy URL. Pass send headers in proxy_headers . Read values from response.headers .

from python_proxy_headers import urllib3_proxy_manager

proxy = urllib3_proxy_manager.ProxyHeaderManager(
    "http://USERNAME:PASSWORD@PROXYHOST:31280",
    proxy_headers={"X-ProxyMesh-Country": "US"},
)
response = proxy.request("GET", "https://api.ipify.org?format=json")

print(response.headers.get("X-ProxyMesh-IP"))

PycURL — send and read

Set pycurl.PROXY  to your proxy URL. Call set_proxy_headers  before calling perform() . Read CONNECT headers from HeaderCapture.proxy_headers .

import pycurl
from io import BytesIO
from python_proxy_headers.pycurl_proxy import HeaderCapture, set_proxy_headers

curl = pycurl.Curl()
body = BytesIO()
curl.setopt(pycurl.URL, "https://api.ipify.org?format=json")
curl.setopt(pycurl.PROXY, "http://USERNAME:PASSWORD@PROXYHOST:31280")
curl.setopt(pycurl.WRITEDATA, body)

set_proxy_headers(curl, {"X-ProxyMesh-Country": "US"})
capture = HeaderCapture(curl)

curl.perform()
print(capture.proxy_headers.get("X-ProxyMesh-IP"))
curl.close()

Scrapy — send and read (scrapy-proxy-headers)

Scrapy cannot put ProxyMesh headers in request.headers for HTTPS—the tunnel encrypts them. Use the scrapy-proxy-headers package to put headers on the proxy CONNECT request and expose CONNECT response headers on response.headers. Install from PyPI; full API notes are in the package documentation.

pip install scrapy-proxy-headers

Register the HTTPS download handler in settings.py or the spider's custom_settings (see quick start in the docs):

# settings.py
DOWNLOAD_HANDLERS = {
    "https": "scrapy_proxy_headers.HTTP11ProxyDownloadHandler",
}

Set the proxy URL in request.meta["proxy"] . Send ProxyMesh headers in request.meta["proxy_headers"] . Read assigned IPs from response.headers  after the handler merges the CONNECT response.

import scrapy

class ProxyMeshSpider(scrapy.Spider):
    name = "proxymesh_scrapy"

    custom_settings = {
        "DOWNLOAD_HANDLERS": {
            "https": "scrapy_proxy_headers.HTTP11ProxyDownloadHandler",
        },
    }

    def start_requests(self):
        yield scrapy.Request(
            "https://api.ipify.org?format=json",
            meta={
                "proxy": "http://USERNAME:PASSWORD@PROXYHOST:31280",
                "proxy_headers": {"X-ProxyMesh-Country": "US"},
            },
            callback=self.parse_ip,
        )

    def parse_ip(self, response):
        data = response.json()
        proxy_ip = response.headers.get(b"X-ProxyMesh-IP")
        self.logger.info("Public IP: %s", data.get("ip"))
        if proxy_ip:
            self.logger.info("Proxy IP: %s", proxy_ip.decode())
        yield {
            "public_ip": data.get("ip"),
            "proxy_ip": proxy_ip.decode() if proxy_ip else None,
        }

To reuse the same proxy IP on later requests, put the prior X-ProxyMesh-IP value in proxy_headers on the next Request (session consistency is described in the scrapy-proxy-headers docs).

ProxyMesh returns X-ProxyMesh-IP on the CONNECT response when you use scrapy-proxy-headers. Read it from response.headers (see proxy headers).

Scrapy middleware and environment options

Besides meta["proxy"]  on each request, you may point Scrapy at a proxy with the http_proxy  environment variable or with downloader middleware. This section covers scrapy-rotating-proxies , scrapy_proxies , and Splash.

Scrapy environment variable

For the Scrapy framework, you may set the http_proxy environment variable before starting the crawler. Scrapy's built-in proxy support picks it up for downloads.

export http_proxy="http://USERNAME:PASSWORD@PROXYHOST:31280"

For HTTPS targets, use IP authentication and omit USERNAME:PASSWORD@ from the URL. After setting the variable, you may enable other downloader middleware that expects a proxy to be configured.

Rotating proxies middleware

The scrapy-rotating-proxies package rotates through a proxy list, checks that proxies are alive, and may adjust crawl speed. Install it, then configure your proxy hosts in settings.py:

pip install scrapy-rotating-proxies
# settings.py — example ProxyMesh hosts (use your assigned hosts/ports)
ROTATING_PROXY_LIST = [
    "PROXYHOST1:31280",
    "PROXYHOST2:31280",
]

You do not set http_proxy when this middleware manages proxies. The package tracks working and dead proxies and periodically re-checks failed ones. See the PyPI project page for downloader middleware class names and full setup.

Random proxy middleware

As another multi-proxy option, RandomProxyMiddleware (package scrapy_proxies) picks a random proxy from a list for each request. Install and configure per the upstream README:

pip install scrapy_proxies

The rotating-proxies middleware above is more actively maintained for health-checking; use random-proxy middleware when only its feature set fits your project.

Scrapy with Splash

For Scrapy Splash, pass the proxy on the SplashRequest (typically via meta). Without a proxy on that request, you may see 503 Service Unavailable from Splash. See Splash proxy examples on Stack Overflow.

Selenium

Selenium WebDriver drives a real browser (Chrome or Firefox) through ProxyMesh. Use IP authentication when possible—it is the most reliable option for HTTPS in browsers. Standard Selenium does not send ProxyMesh custom headers on HTTPS CONNECT tunnels—for that, see Selenium Wire for HTTPS Requests or Using Selenium in the Proxy.

Chrome

Install selenium  (Selenium 4.6+ includes Selenium Manager for ChromeDriver). Point Chrome at the proxy with --proxy-server . Use an http://  proxy URL even for HTTPS sites.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

proxy_url = "http://PROXYHOST:31280"  # IP authentication (no USERNAME:PASSWORD@)

options = Options()
options.add_argument(f"--proxy-server={proxy_url}")

driver = webdriver.Chrome(options=options)
driver.get("https://api.ipify.org?format=json")
print(driver.page_source)
driver.quit()

With username/password authentication, Chrome often ignores credentials embedded in --proxy-server. Configure a Chrome extension that supplies proxy auth, or use headless browser proxy authentication for a working pattern.

Firefox

Use FirefoxOptions and Mozilla proxy preferences (Selenium 4; do not use the removed FirefoxProfile constructor). Set the proxy host and port; use IP authentication and omit USERNAME:PASSWORD@ in the preference values when possible.

from selenium import webdriver
from selenium.webdriver.firefox.options import Options

options = Options()
options.set_preference("network.proxy.type", 1)
options.set_preference("network.proxy.http", "PROXYHOST")
options.set_preference("network.proxy.http_port", 31280)
options.set_preference("network.proxy.ssl", "PROXYHOST")
options.set_preference("network.proxy.ssl_port", 31280)
options.set_preference("network.proxy.share_proxy_settings", True)

driver = webdriver.Firefox(options=options)
driver.get("https://api.ipify.org?format=json")
print(driver.page_source)
driver.quit()

To keep the same outgoing IP across Firefox sessions, use X-ProxyMesh-IP or X-ProxyMesh-Prefer-IP with a client that can set proxy headers. Selenium alone cannot set those CONNECT headers; see Selenium Testing on a Single IP in Firefox for header options with ProxyMesh.

CloudScraper and AutoScraper

For CloudScraper, use python_proxy_headers.cloudscraper_proxy.create_scraper. For AutoScraper, use python_proxy_headers.autoscraper_proxy.ProxyAutoScraper. See the package docs for CloudScraper, AutoScraper, and PycURL.

Common pitfalls

• Proxy URL scheme: use http:// on the proxy host, even when the target URL is https://.
• HTTPS custom headers: use python-proxy-headers, scrapy-proxy-headers, or a client proxy-header API. A typical request header will not send X-ProxyMesh-Country on CONNECT.
• Reading X-ProxyMesh-IP: follow the adapter patterns under custom proxy headers. Some clients may send headers but hide CONNECT response headers.
• Connection failures: see connection troubleshooting.

Back to HTTP proxy overview