Proxy Server Requests over HTTPS

If you send most of your proxy server requests over HTTPS, this article is for you. HTTPS requires special steps to ensure that the proxy server can read and accept your credentials and also authenticate itself to the remote site. This article describes alternative methods of authentication for HTTPS requests.

And if you're using custom headers to control the IP address going to the remote site, these also need to be transmitted in a way that enables the proxy server to read them.

We'll discuss these elements of HTTPS requests in detail, and then put them all together in a transmission method.

Authentication

With an HTTPS request (as with HTTP), you need to authenticate your user agent to the proxy, and the proxy to the remote site. But with HTTPS, request headers, including custom headers, are encrypted when sent through the proxy.

Basic Access Authentication

Basic Access Authentication uses the username:password method to create a Proxy-Authorization header that looks like this:

In Windows:

Proxy-Authorization: Basic base64-encoded-username:password

In Unix/Linux:

export http_proxy=http://USERNAME:PASSWORD@PROXYHOST:PORT

But with HTTPS, the proxy server can't read encrypted data, including the Proxy-Authorization header, which authenticates your user agent to the proxy. To prevent encryption, that header must be sent with the initial CONNECT method, instead of with the rest of the request headers. This places it outside the encrypted data and leaves it decipherable. The Python Requests library is a Python client library that can do this.

IP-Based Authentication

As a reliable alternative, we recommend IP-based authentication. You can accomplish this with:

• The Authenticated IP Addresses function on your ProxyMesh dashboard, OR
• The ProxyMesh API.

Note: This method works only for authenticating your own IP address to the proxy, not for authenticating the proxy to the remote site. Any IP address you add to your account will be able to access the ProxyMesh proxy without a Proxy-Authorization header.

Authentication Header String

With HTTPS, if you want to control the IP address sent to the remote site, one method is to insert a random string in the Proxy-Authorization header. Then, instead of passing in an encoded username:password to the authentication header, you pass in username:string:password , where string is a random string that you generate. As long as you keep using the same string – assuming the outgoing IP is available – the proxy server will choose that same outgoing IP. If the IP is no longer available, then a new IP will be chosen by the proxy server, and will be used for subsequent requests with the same string .

You can use any string you choose. Below is a simple example in which Python code is used to generate a 5-character random string.

import random
import string
rand_string = ''.join(random.choices(string.ascii_letters + string.digits, k=5))

Most client libraries support the Basic access authentication method, which means the only change required in your code is to join the string to your username when providing the header value. This works with HTTP, but with HTTPS, there's a difference. For HTTPS requests, only some libraries, such as  Python Requests, support sending the Proxy-Authorization header with the initial CONNECT. Other client libraries may require custom code to make the Proxy-Authorization header work for HTTPS requests.

Proxy Header Masking

Our rotating proxy and world proxy servers prevent identifying headers from being passed along in requests. The following headers are removed in order to mask proxy identification by the remote site:

• Authorization
• From
• Proxy-Authorization
• Proxy-Connection
• Via
• X-Forwarded-For

Custom headers are also removed from the body of the HTTPS request. Please see the following section, which describes special handling to make custom headers readable to the proxy.

Custom Headers

You can use custom request headers to modify the default behavior of the ProxyMesh servers. Here's what you need to know about custom headers over HTTPS.

The proxy server can securely proxy HTTPS/SSL connections between you and an HTTPS server. All communication between your client/browser and the secure site is encrypted; the proxy server is only moving the data back and forth. If included in the HTTPS request data, custom headers are also encrypted. Since the proxy cannot inspect HTTPS requests, all custom ProxyMesh headers must be sent with the initial CONNECT method to place them outside the encrypted data.

Moreover, most client libraries will not support custom proxy server headers without add-ons or extensions. In the data stream, the initial CONNECT method establishes the only point where unencrypted data is sent to the proxy server. Here is a code example showing where you must insert custom headers:

CONNECT example.com:443 HTTP/1.1
X-ProxyMesh-IP: 123.456.789.000

*actual request headers*

Similarly, the proxy server cannot inject an extra header into the final response. Instead, the X-ProxyMesh-IP response header is injected immediately after the Connection response, which looks like this:

HTTP/1.1 200 Connection established
X-ProxyMesh-IP: 123.456.789.000

*final response headers & body*

Transmission: Putting It Together

Below is a general method for transmitting custom proxy server headers. With it, you hook into a socket as soon as you've made the initial CONNECT . The method involves these basic steps:

1. Open the socket directly in your code.
2. Send the initial CONNECT string, with optional custom proxy server headers.
3. Read the initial response to get the X-ProxyMesh-IP header.
4. Then continue with the normal request headers and response.

Depending on your client library, you may be able to pass the socket to the library in such a way that you can use the library as normal to complete the request.

To avoid use of a specific IP address:
Follow the transmission steps above, but include the IP in a custom X-ProxyMesh-Not-IP header.

Note on the Postman Platform

For web users gathering data with their own APIs, the Postman platform offers collaborative tools, a repository, and workspace to build and maintain web applications with APIs.

Be aware that, while the Headers configuration from Postman works well for HTTP requests, Postman cannot read a proxy header passed in with an encrypted request. Therefore, it passes in a custom request header undetected to the remote site, which will see the originating IP address within the encrypted request.

Please check out our help article Postman with ProxyMesh for steps to work around this limitation, obtain a custom header, and use it with a new HTTPS request..