Python Proxy Configuration Examples

Requests

Requests is a great Python library for doing HTTP requests, specifically version 2.7.0 and higher. The example below shows the most reliable way to use proxy authentication, but if you're using IP authentication, then you can remove USERNAME:PASSWORD@ in the proxies dictionary.

>>> import requests
>>> proxies = {'http': 'http://USERNAME:PASSWORD@HOST:PORT',
               'https': 'http://USERNAME:PASSWORD@HOST:PORT'}
>>> response = requests.get('http://example.com', proxies=proxies)

To use multiple proxy servers, you can randomly choose one for each request. Your code might look like this:

>>> import random
>>> import requests
>>> proxy_choices = ['HOST1:PORT', 'HOST2:PORT']
...
>>> proxy = random.choice(proxy_choices)
>>> proxies = {'http': 'http://%s' % proxy, 'https': 'http://%s' % proxy}
>>> response = requests.get('http://example.com', proxies=proxies)

Scrapy

For the  scrapy crawling framework, you must set the http_proxy environment variable, then activate the HttpProxyMiddleware.

$ export http_proxy=http://USERNAME:PASSWORD@HOST:PORT

For https requests, you should use  IP authentication, and remove USERNAME:PASSWORD@ from the http_proxy variable.

To make use of multiple proxies, use  RandomProxyMiddleware.

Scrapy with Splash Request

For a splash request via the proxy, add a 'proxy' argument to the SplashRequest object. Without this argument, you may receive a 503 service unavailable response.

Click here to view sample code for a splash request.

Selenium + Chrome

To configure the Python webdriver for Selenium to use Chrome, see  how do i set proxy for chrome in python webdriver. Be sure to use IP authentication before configuring Selenium.

Selenium + Firefox

To set the network proxy settings for Selenium to use Firefox, you can do something like this. Be sure to use  IP authentication before configuring Selenium.

profile = webdriver.FirefoxProfile()
profile.set_preference("network.proxy.type", 1)
profile.set_preference("network.proxy.http", 'HOST')
profile.set_preference("network.proxy.http_port", 31280)
profile.set_preference("network.proxy.ssl", 'HOST')
profile.set_preference("network.proxy.ssl_port", 31280)
driver = webdriver.Firefox(firefox_profile=profile)

Still need help? Contact Us Contact Us