Cloudscraper

Warnings

• breaking Cloudscraper v3.0.0 (released on GitHub, potentially available via `pip install cloudscraper>=3.0.0` or direct GitHub install) introduces breaking changes, including a minimum Python version requirement of 3.8+ (dropping Python 3.6 and 3.7 support) and significant dependency upgrades. Python 2 compatibility code has also been removed. If upgrading from an older 1.x or 2.x version, ensure your environment meets these new requirements.
  Fix: Upgrade Python to 3.8 or newer. Update all project dependencies to their latest stable versions. Review `cloudscraper`'s GitHub `UPGRADE_SUMMARY.md` for specific changes if migrating an existing codebase. Use `pip install --upgrade cloudscraper` or `pip install cloudscraper>=3.0.0`.
• gotcha Cloudflare continuously updates its anti-bot techniques, leading to an 'arms race' where `cloudscraper` versions can become outdated and cease to work against the latest Cloudflare protections. This often results in `403 Forbidden` errors or being stuck on a Cloudflare challenge page.
  Fix: Keep `cloudscraper` updated to the latest version. If issues persist, enable debug mode (`cloudscraper.create_scraper(debug=True)`) to diagnose. Consider adjusting scraper settings like `delay`, `browser` profile, or trying different JavaScript interpreters (e.g., Node.js if installed).
• gotcha While `cloudscraper` can handle JavaScript challenges, it does not natively solve CAPTCHA challenges (like reCAPTCHA or hCaptcha) that Cloudflare might present. Users will encounter blocks if CAPTCHAs are triggered.
  Fix: Integrate with a third-party CAPTCHA solving service (e.g., 2Captcha, Anti-Captcha) via `cloudscraper`'s built-in support. This typically requires an API key and an active subscription to the service.
• gotcha Cloudscraper is not a full-fledged browser and may not suffice for complex web scraping scenarios involving heavy client-side JavaScript rendering, dynamic content loaded via XHR, or highly sophisticated Cloudflare protection layers. It primarily tackles JavaScript challenges and cookie management.
  Fix: For highly dynamic or heavily protected websites, consider using headless browser automation tools like Selenium or Playwright, or utilizing specialized managed scraping APIs that handle these complexities.

Install

• pip install cloudscraper Install stable release

Imports

• cloudscraper

  import cloudscraper

  While `CloudScraper()` directly works, `create_scraper()` is the recommended factory function as it handles initial setup and configuration best, especially in newer versions.

Quickstart

The simplest way to use `cloudscraper` is by calling `cloudscraper.create_scraper()`, which returns a `CloudScraper` instance. This instance behaves similarly to a `requests.Session` object. Any requests made through this session to Cloudflare-protected websites will automatically attempt to bypass the anti-bot measures. The script includes a placeholder URL and demonstrates basic error handling for a 403 status code, which is common if Cloudflare still blocks the request. It's recommended to set a realistic User-Agent for better stealth.

import cloudscraper
import os

# Instantiate a CloudScraper session. This object works like a requests.Session
scraper = cloudscraper.create_scraper(
    # Optionally, provide a `requests` Session object to base it on
    # sess=requests.Session(),
    # Or configure an interpreter, e.g., 'nodejs' if installed for better performance
    # interpreter='nodejs'
)

# Make a GET request to a Cloudflare-protected site
# Replace 'http://somesite.com' with your target URL
# For demonstration, we'll use a placeholder URL or a test site
# You might need to set headers, e.g., a User-Agent, for more realistic requests
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36'}
url = os.environ.get('TARGET_URL', 'https://nowsecure.nl') # A common test site for bot detection

try:
    response = scraper.get(url, headers=headers)
    response.raise_for_status() # Raise an exception for bad status codes
    print(f"Successfully accessed {url} (Status: {response.status_code})")
    # print(response.text[:500]) # Print first 500 characters of content
except Exception as e:
    print(f"Failed to access {url}: {e}")
    if response.status_code == 403:
        print("Access denied (403 Forbidden). Cloudflare protection might be too strong or settings need adjustment.")