Using urllib3 with Proxy Headers

When you’re preparing to scrape the web using a specific IP address, it’s common to obtain that address from a response header through a proxy tunnel. We've created an extension module for urllib3 that makes this easy. The python-proxy-headers projects supports multiple different python libraries, including urllib3.

Here's an example of how to get the X-ProxyMesh-IP header from a proxied request.

from python_proxy_headers import urllib3_proxy_manager
proxy = urllib3_proxy_manager.ProxyHeaderManager('http://PROXYHOST:PORT')
r = proxy.request('GET', 'https://api.ipify.org?format=json')
r.headers['X-ProxyMesh-IP']

Now you can pass that X-ProxyMesh-IP value back in using custom proxy headers:

from python_proxy_headers import urllib3_proxy_manager
proxy = urllib3_proxy_manager.ProxyHeaderManager('http://PROXYHOST:PORT', proxy_headers={'X-ProxyMesh-IP': IP)
r = proxy.request('GET', 'https://api.ipify.org?format=json')
r.headers['X-ProxyMesh-IP']

And if you like the requests library, the requests_adapter module builds on the urllib3_proxy_manager:

from python_proxy_headers import requests_adapter
r = requests_adapter.get('https://api.ipify.org?format=json', proxies={'http': 'http://PROXYHOST:PORT', 'https': 'http://PROXYHOST:PORT'}, proxy_headers={'X-ProxyMesh-Country': 'US'})
r.headers['X-ProxyMesh-IP']

Hopefully this makes it easy for you to handle our custom proxy headers in Python.