Free Proxy Scraper

Common errors

• fp.fp.FreeProxyException: There are no working proxies at this time.

  cause The library could not find any active proxies from its sources that meet the specified criteria (country, anonymity, HTTPS, timeout) after checking. Free proxies are often down or blocked.
  fix Relax your filtering criteria (e.g., remove `country_id`, `https=True`, or increase `timeout`). Try running the script again, as proxy lists update frequently. Consider implementing retry logic with a delay.

• urllib3.exceptions.ProxySchemeUnknown: Not supported proxy scheme None

  cause This error, fixed in version 1.0.1, indicates an issue where the library might return a proxy without a proper scheme (e.g., 'http://' or 'https://') or where `urllib3` fails to parse the proxy string.
  fix Update `free-proxy` to version 1.0.1 or higher to resolve this specific parsing issue. Ensure that the proxy string received is correctly formatted before passing it to `requests` or other HTTP clients.

• requests.exceptions.ProxyError: ('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 Forbidden'))

  cause The proxy server rejected the connection (e.g., 403 Forbidden, 407 Proxy Authentication Required) or failed to tunnel the request to the target website. This is common with unreliable free proxies or if the proxy requires authentication not supported by `free-proxy`.
  fix Retry with a different proxy. If using custom proxies, ensure credentials are correct. For `free-proxy`, this often means the chosen proxy is dead or blocked; try getting another one via `FreeProxy().get()`.

• requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

  cause The connection was abruptly terminated by either the proxy server or the target server. This can happen due to proxy instability, timeouts, or the target website detecting and blocking the proxy.
  fix Implement robust retry logic (e.g., using `requests` with `urllib3.Retry`). Increase the `timeout` parameter in `FreeProxy()` and your `requests.get()` call. Fetch a new proxy if errors persist.

Warnings

• breaking In version 1.0.5, the library changed its error handling mechanism from calling `sys.exit()` to raising a `FreeProxyException`.
  Fix: Update your code to catch `fp.fp.FreeProxyException` instead of relying on `SystemExit` or uncontrolled program termination. Example: `try: ... except FreeProxyException as e: print(e)`.
• gotcha Free proxies are inherently unreliable, often slow, short-lived, or quickly blacklisted. This can lead to frequent connection errors, timeouts, or HTTP status codes like 403 Forbidden or 429 Too Many Requests from target websites.
  Fix: Implement robust error handling, retry mechanisms, and consider using a pool of proxies. For critical applications, paid proxy services offer greater reliability and performance.
• gotcha Website structure changes on proxy source sites (e.g., sslproxies.org) can break the scraping logic (XPath selectors), leading to no proxies being found. This happened in version 1.0.3.
  Fix: Regularly update the `free-proxy` library to ensure compatibility with upstream proxy source websites. If you encounter issues, check the library's GitHub for recent updates or reported issues.
• gotcha Older versions (prior to 1.1.1) may have had issues with the `https` parameter, potentially failing to correctly filter or provide HTTPS proxies.
  Fix: Ensure you are using `free-proxy` version 1.1.1 or newer for reliable HTTPS proxy filtering.

Install

• pip install free-proxy Install latest version

Imports

• FreeProxy

  from fp.fp import FreeProxy

Quickstart

Initializes `FreeProxy` to get a working proxy. The first example fetches a random proxy and tests it with `requests`. The second example demonstrates fetching an HTTPS proxy specifically from the US or Brazil. Due to the nature of free proxies, reliability can vary, and retries may be necessary.

from fp.fp import FreeProxy
import requests
import os

# Get a random working proxy
proxy = FreeProxy(rand=True, timeout=1).get()

if proxy:
    print(f"Found working proxy: {proxy}")
    proxies = {
        "http": proxy,
        "https": proxy,
    }
    try:
        # Test the proxy with a simple request
        # Use a reliable target URL, e.g., ipify.org to check the proxy IP
        response = requests.get('https://api.ipify.org?format=json', proxies=proxies, timeout=5)
        if response.status_code == 200:
            print(f"Request through proxy successful. Your IP via proxy: {response.json()['ip']}")
        else:
            print(f"Request through proxy failed with status: {response.status_code}")
    except requests.exceptions.RequestException as e:
        print(f"Error making request through proxy: {e}")
else:
    print("No working proxies found at this time.")

# Example with specific country and HTTPS
# Note: Free proxies can be unreliable and may not always work as expected.
# You might need to run this multiple times or adjust parameters.
print("\nTrying to get an HTTPS proxy from US or BR...")
us_br_proxy = FreeProxy(country_id=['US', 'BR'], https=True, timeout=2, rand=True).get()
if us_br_proxy:
    print(f"Found US/BR HTTPS proxy: {us_br_proxy}")
else:
    print("No working US/BR HTTPS proxies found.")