Proxy Server Request & Response Headers

This article covers the following topics:

• Custom request headers that can be used to modify the default behavior of the ProxyMesh servers
• Response headers from the proxy server
• Response codes indicating possible errors or server issue
• The HTTPS CONNECT method used with unencrypted custom headers.

The ProxyMesh proxy servers accept a few custom headers that can be used to modify the default behavior. The proxy server will also return certain response codes when a request is not successful. All requests to the proxy server will return with an additional header, X-ProxyMesh-IP, which contains the IP used to complete the request. Note that HTTPS requests require extra work to make use of the custom headers.

All HTTP clients should allow for adding custom headers and inspecting response headers. For the Chrome web browser, you can use plugins such as Header-Editor or ModHeader to manipulate headers.

If you have any questions about these headers, or problems using them, please email ProxyMesh Support.

Custom Request Headers

If no custom headers are given, the proxy server will choose an IP at random, and use a timeout of 20 seconds when waiting for a response from the remote site. You can use the headers described below to alter this behavior.

X-ProxyMesh-IP

Use this header when you want to use a specific IP for a request. This IP should come from the X-ProxyMesh-IP response header of a previous request. If the IP is no longer available, you will get a 502 error code with the X-ProxyMesh-IP-Not-Found response header.

X-ProxyMesh-Prefer-IP

Use this header when you want to use a specific IP for a request, but you also want the request to succeed even if the IP is no longer available. This is like the X-ProxyMesh-IP header, but when an IP is not available, a random IP is chosen instead. You can tell when this happens by comparing the X-ProxyMesh-IP response header to the IP you sent with the X-ProxyMesh-Prefer-IP header. If they are different, then the IP you tried to use is no longer available.

X-ProxyMesh-Not-IP

Use this header to exclude a specific IP from the available choices. The actual IP chosen will be random, as long as it is not the given IP. You can also send a comma list of IPs to exclude multiple IPs. You will get a 502 error code if no more IPs are available, after excluding the IPs sent in the header.

Excluded IPs in Responses

Via the X-ProxyMesh-Not-IP custom header, you can set an array of IP addresses to exclude from your requests. Your code might look like the example below. This example comes from PHP, but counterparts can be found in most other languages:

array (
'proxy' => 'http://USERNAME:PASSWORD@HOST:PORT',
'X-ProxyMesh-Not-IP' => '12.345.678.90,098.76.54.321',
)

If these IPs still show up in the X-ProxyMesh-IP response headers, here are some possible causes and solutions:

• 403 Error Message: This means the remote site has been blocked and can only be accessed through the Open Proxy or the World Proxy. Or, if you've gotten this message from a request to a Google site or a search engine, please see HTTP Requests to HTTPS Sites below.

• HTTPS Requests: You may have sent an HTTPS request but not passed the header in the initial CONNECT request. The header can’t be read if it is part of the request payload, which is fully encrypted. Instead,  include the header in the initial CONNECT request. Please see Proxy Server Requests over HTTPS for details. 
  • Do not cache the IPs for more than 1 day, as they will be out-of-date or offline after 12 hours.
  • Also, because a rotating proxy has only 10 IPs, you should not list all of them for exclusion in the X-ProxyMesh-Not-IP header.

• HTTP Requests to HTTPS Sites: You may have sent an HTTP request to an HTTPS site, such as Google, LinkedIn, Facebook, Twitter, or other social media. ProxyMesh by itself does not work well with these sites, but you can get good results by combining ProxyMesh with a scraping tool such as Saleshub. If you prefer unassisted proxies for requests of this type, we recommend other proxy services, such as Trusted Proxies.or Luminati. Also see How the Service Works.

X-ProxyMesh-Country

Our proxy service is great for scanning sites from specific regions while preventing blocks.

The open and world proxies have IPs in multiple countries. Use the X-ProxyMesh-Country header to restrict the IP choices to a specific country. The value of the header should be a 2 character ISO country code, such as US or RU. Check the status page for each proxy to see what country codes and how many IPs are currently available. You will get a 503 response code if you use a country code with no IPs.

Note: To connect to sites – especially in large numbers – in a specific geographical area, we recommend specifying proxies located near that area and, if possible, in the same domain. This can help speed responses to requests.

X-ProxyMesh-Timeout

The proxy server has a default response timeout of 20 seconds. If the remote server does not respond in that time, you will get a 408 response code. If you need to wait longer for a complete response, use the X-ProxyMesh-Timeout header to specify the number of seconds you want to wait.

Compression Headers

Compression can be useful for high-traffic research with sizeable amounts of data per request. A compression header you send to the remote server will be passed through the proxy, so that the proxy will send back the requested data in compressed format.

On the command line, you can pass a compression header with cURL, like this:

-H "Accept-Encoding: gzip"

Although the request example is for gzip, the server may apply a different compression method.

Response Headers

These headers are inserted by the proxy server. They will be included with the other response headers from the remote site, unless it is an HTTPS request, in which case the headers are included with the Connection response.

X-ProxyMesh-IP

This header is included with every response, and its value is the IP used for the request. 

If the response header includes an IP you've listed for exclusion in X-ProxyMesh-Not-IP, click here for information.

X-ProxyMesh-IP-Not-Found

When an IP specified by the X-ProxyMesh-IP request header is no longer available, you will get a 502 response with a X-ProxyMesh-IP-Not-Found header containing the IP.

Response Codes

A remote site can return any number of response codes, including the same ones listed below, but for different reasons. It is important to inspect the response body in order to differentiate between whether the response came from the proxy server or the remote site.

• 402

• 403

• 407

• 408

• 502

• 503

• 509

HTTPS CONNECT Method with Custom Headers

HTTPS requests with custom headers require extra steps. Any request headers are encrypted unless you use the initial CONNECT method. For additional details, please see Proxy Server Requests over HTTPS.