High-Traffic Proxy Connections

ProxyMesh will support large numbers of short-interval, recurrent proxy server connections. All of our proxies have a limit of 100 concurrent connections per IP address.

Requests exceeding the limit result in 503 error responses. Therefore, if you are running queries with traffic exceeding 100 concurrent connections/IP/proxy, and if you cannot slow your crawl rate, you will need to use multiple proxies.

When high-level traffic involves a group of related URLs, best practice is to distribute your requests across multiple proxies. 

Scaling your requests across multiple proxies is best done by adding more proxies to your account. 

• Consider upgrading your subscription plan for access to more proxies. 
• Or, with the ProxyMesh 50 or ProxyMesh 100 plan, you also have the option of adding extra proxies beyond the number offered in each plan. 

On the 50 and 100 plans, you can add extra proxies to your account for $10/month each. 

So, let's say you were enrolled in the ProxyMesh 100 plan and you needed to run 1,280 concurrent requests per second. ProxyMesh 100 provides 11 proxies. 

By adding two extra proxies, you could scale the 1,280 requests across 13 proxies so that the number of concurrent requests on each proxy was well under 100. 

Your monthly cost would be $100.00 for the 100 plan and an additional $20.00 for the 2 extra proxies. 

Best Practices

Here are some recommended practices that can speed your proxy responses and minimize timeouts.

• Reduce the number of concurrent requests from a single IP. This could involve using an additional IP for crawling, or slowing down your crawl rate on your current requests.
• With added proxies, you have more connection strategies available, such as putting all of your authorized proxy hostnames in a list in your code or script, then randomly choosing one proxy for each request.
• To connect to sites – especially in large numbers – in a specific geographical area, use proxies located near that area and, if possible, in the same domain. For example, use European proxies to connect to European sites. Try configuring the X-ProxyMesh-Country request header for this result when using the open and world proxies.
• To optimize bandwidth use and control the total number of requests, minimize requests to pull images, JavaScript, and CSS files.

Apparent Server Outage Errors

Occasionally while running a high-traffic proxy connection, you may see an error message that seems to indicate a server outage.

Example:

Message: 'cURL error 56: Proxy CONNECT aborted due to timeout.'

This does not necessarily indicate a true outage on the proxy. There could have been network issues in between, or perhaps some of the outgoing IPs were temporarily blocked by the remote site.

Each geographical region has multiple IP addresses associated with it, representing multiple servers to handle requests. At any time, these IP addresses may be switched, making an individual IP unavailable; at this point another IP would be added to take over. 

Also, sometimes requests can be sent to an IP that is no longer available, resulting in issues such as this.

To reduce the impact of timeout errors, we suggest two steps:

1. Upgrade your plan and distribute your requests across multiple proxies.
2. Implement a retry strategy in your app, so that when a timeout occurs, you try a different proxy, and/or wait 1 second before retrying.

Interpreting 503 Error Responses

A 503 error response to a request can have several possible meanings. Two of these are:

• service unavailable The proxy server cannot handle requests right now. This is rare, but can happen under high load situations. For details, please see Non-Standard Error Messages.
• too Many Connections You have more than 100 open connections to the proxy server from a single IP -- an excessive load on the server. ..
• Other possible causes may be poor network connections between the proxy and the target site, or some form of IP blocking. If possible, try sending requests to the site from other proxies and see if that eliminates the error.

If you’ve gotten a lot of 503 errors when attempting to access certain sites, it’s possible that the target server is dropping the connection for some reason. It could be due to load, poor network connections between the proxy and the target site, or some kind of IP blocking. Or the remote site could be down for maintenance.

See if you can resolve the problem by trying other proxies for the site where you get 503 errors

Messages from the proxy will be in the response body when you receive a 503 error from the proxy server. The Proxy Status Page does not report how many concurrent connections you have per server. 

Your dashboard does offer a means of checking the overall load and condition of a given server. From the dashboard, just click on the name of the server to view a page with a variety of information about the server. Our article Proxy Status Page discusses the metrics displayed on that page.

If the server status is Available, a 503 response could mean you're exceeding the limit on concurrent connections, and may need to reconsider your high-traffic strategy.