Proxy Server Request & Response Headers
This article covers the following topics:
- Request headers that can be used to modify the default behavior of the ProxyMesh servers
- Response headers from the proxy server
- Response codes indicating possible errors or server issue
- The HTTPS CONNECT method used with unencrypted custom headers.
The ProxyMesh proxy servers accept a few custom headers that can be used to modify the default behavior. The proxy server will also return certain response codes when a request is not successful. All requests to the proxy server will return with an additional header,
X-ProxyMesh-IP, which contains the IP used to complete the request. Note that HTTPS requests require extra work to make use of the custom headers.
If you have any questions about these headers, or problems using them, please email ProxyMesh Support.
If no custom headers are given, the proxy server will choose an IP at random, and use a timeout of 20 seconds when waiting for a response from the remote site. You can use the headers described below to alter this behavior.
Use this header when you want to use a specific IP for a request. This IP should come from the
X-ProxyMesh-IP response header of a previous request. If the IP is no longer available, you will get a
502 error code with the
X-ProxyMesh-IP-Not-Found response header.
Use this header when you want to use a specific IP for a request, but you also want the request to succeed even if the IP is no longer available. This is like the
X-ProxyMesh-IP header, but when an IP is not available, a random IP is chosen instead. You can tell when this happens by comparing the
X-ProxyMesh-IP response header to the IP you sent with the
X-ProxyMesh-Prefer-IP header. If they are different, then the IP you tried to use is no longer available.
Use this header to exclude a specific IP from the available choices. The actual IP chosen will be random, as long as it is not the given IP. You can also send a comma list of IPs to exclude multiple IPs. You will get a
502 error code if no more IPs are available, after excluding the IPs sent in the header.
The open and world proxies have IPs in multiple countries. Use the
X-ProxyMesh-Country header to restrict the IP choices to a specific country. The value of the header should be a 2 character ISO country code, such as
RU. Check the status page for each proxy to see what country codes and how many IPs are currently available. You will get a
503 response code if you use a country code with no IPs.
Note: To connect to sites – especially in large numbers – in a specific geographical area, we recommend specifying proxies located near that area and, if possible, in the same domain. This can help speed responses to requests.
The proxy server has a default response timeout of 20 seconds. If the remote server does not respond in that time, you will get a
408 response code. If you need to wait longer for a complete response, use the
X-ProxyMesh-Timeout header to specify the number of seconds you want to wait.
These headers are inserted by the proxy server. They will be included with the other response headers from the remote site, unless it is an HTTPS request, in which case the headers are included with the Connection response.
This header is included with every response, and its value is the IP used for the request. If you'd like to use the same IP for a subsequent request, pass in this header unchanged.
When an IP specified by the
X-ProxyMesh-IP request header is no longer available, you will get a
502 response with a
X-ProxyMesh-IP-Not-Found header containing the IP.
A remote site can return any number of response codes, including the same ones listed below, but for different reasons. It is important to inspect the response body in order to differentiate between whether the response came from the proxy server or the remote site.
Your account has been de-activated due to cancellation or lack of payment, or you are not authorized on the particular proxy server. If your account is active, then look at modifying your proxy server authorizations.
The remote site has been blacklisted and can only be accessed thru the open proxy server or the world proxy server.
Your IP is not authorized or your Basic authorization header has an incorrect format, missing username, or bad password.
The IP requested in the
X-ProxyMesh-IP request header cannot be found. This response will include
X-ProxyMesh-IP as well as
X-ProxyMesh-IP-Not-Found both containing the same IP as the
X-ProxyMesh-IP request header. If these headers are not present in the response, then this response came from the remote site, not the proxy server.
There are a number of different situations that can generate a
503 error, and each situation produces a different response body. All of these situations are temporary, generally lasting a few minutes at most.
no available IP addresses: You have excluded all available IPs in the
X-ProxyMesh-Not-IPrequest header. You'll need to wait until the IPs rotate, or reduce the number of excluded IPs.
no proxies available: There are no outgoing IPs available. If you are using the
X-ProxyMesh-Countryheader, then you have chosen a country value with no IPs. Otherwise, this is due to an error in the IP rotation. This is rare, likely due to an issue with the hosting provider, and will generally be fixed within 1 hour.
proxy error: The proxy server cannot connect to an outgoing IP. This happens more often with the open proxy server, due to the unreliability of the open proxies.
server error: Something went wrong in the proxy server while trying to authenticate. This is very rare.
service unavailable: The proxy server cannot handle requests right now. This is rare, but can happen under high load situations.
too many connections: You have more than 100 open connections to the proxy server from a single IP. This puts undue load on the proxy server, and is a very inefficient use of resources. It is much better to use a shared connection or connection pool for proxy connections.
too many errors: Your requests have generated more than 60 response errors (with a status code of 400 or greater) over the past 30 seconds. You need to fix what you're doing to stop producing so many errors.
Any other response body likely indicates a rate limit response from the remote site. If this happens regularly, try using a different proxy server, or slowing down your crawl.
You have exceeded your bandwidth limit. This response will continue until your next bill has been processed, or you raise the limit.
HTTPS CONNECT Method with Custom Headers
HTTPS requests with custom headers are difficult because the actual request headers are encrypted. The only point at which unencrypted data is sent to the proxy server is with the initial CONNECT method. This is where you must insert the custom headers:
CONNECT example.com:443 HTTP/1.1 X-ProxyMesh-IP: 123.456.789.000 *actual request headers*
Similarly, the proxy server cannot inject an extra header into the final response. Instead, the
X-ProxyMesh-IP response header is injected immediately after the Connection response, which looks like this:
HTTP/1.1 200 Connection established X-ProxyMesh-IP: 123.456.789.000 *final response headers & body*
For additional details, please see Proxy Server Requests over HTTPS.