Phantombuster Strategies with ProxyMesh

Phantombuster is a scraping platform to automate websites, providing many APIs for research on social media. Implementing Phantombuster with ProxyMesh can be an effective web strategy, combining key use cases of both.

For Phantombuster:

  • Growth hacking to help businesses acquire and retain customers
  • Cloud marketing with specialized APIs
  • Research and outreach for job recruitment
  • Social media research

…and for ProxyMesh:

  • Anonymity
  • Distribution of requests over multiple proxies
  • Scraping
  • Avoidance of rate-limits

This article describes how to install and implement Phantombuster on the proxy. You'll also find links to helpful Phantombuster sites.

This article assumes that you already have:

Creating a Proxy in Phantombuster

from Setting Up a Proxy in Phantombuster

  1. In the Phantombuster navigation bar, click on your name, then on IP & Proxies.
  2.  Click on New Proxy Pool to give the proxy a name of your choosing.
  3. Fill in your ProxyMesh information:
    1. Your proxy’s address (including port) as displayed in your ProxyMesh dashboard, e.g., de.proxymesh.com:31280
    2. Your login details.
  4. Click on Save Proxy (in Phantombuster).
  5. Set your API to launch with a proxy.
    1. Still in Phantombuster, go to the My APIs section of your dashboard and click on the Settings of the API you will use.
    2. Then scroll down to the Headless Browser options and pick Random Proxy from Pool.
  6. Save your settings.

Now you're ready to use Phantombuster and ProxyMesh together.

Best Practices

Here are some recommended practices that can speed your proxy responses and minimize timeouts.

  • Especially for social media research, we recommend you choose a proxy near your geographical location. Social media monitor the geographical locations of IP addresses. So, if you’re accessing social media via Phantombuster (with servers located in the western U.S.) and a second, far-distant address which is your actual location, then use a proxy in your own geographical area for the Phantombuster requests. This can help you avoid detection and bans.
  • Reduce the number of concurrent requests from a single IP. For example, use an additional IP for crawling, or slow down the crawl rate on your current requests.
  • Add proxies for more IPs to multiply connection strategies available.
  • To connect to sites – especially in large numbers – in a specific geographical area, use proxies located near that area and, if possible, in the same domain.
  • For optimal bandwidth use, minimize requests to pull images, JavaScript, and CSS files.

501 Error Messages

At times when using Phatombuster with ProxyMesh, you may receive a 501 error response with the following message body: "Message body: "Proxy accepts request but does not seem to support SSL (HTTP 501)"

Generally, you can ignore this message because your requests are actually working. However, if they persist, you can contact us for assistance.

Phantombuster with PhantomJS Headless & CasperJS

Be aware that PhantomJS, a headless web browser which is often used with CasperJS, is suspended and archived although version 2.1.1 is available for continued use. CasperJS is available as a testing framework and for scripting of full navigation scenarios in a simple interface. CasperJS also provides a download link for research.

This link leads you to a tutorial on web scraping with CasperJS and Phantombuster.

Headless Chrome

Similar to PhantomJS is Google's Headless Chrome, which enables automated control of web pages. With this tool, you can automate tasks, scripts, and user interface tests against a browser without opening the browser's user interface.

 Phantombuster has published a blog, Web Scraping in 2017: Advanced Headless Chrome Tips & Tricks, about ways to use Headless Chrome with Phantombuster.

Useful Links

You may want to follow these Phantombuster links for further details:

Still need help? Contact Us Contact Us