Selenium Wire

Warnings

• breaking In version 5.0.0, the `request.path` attribute changed to return only the path segment of the URL. To retrieve the full URL, use `request.url` instead. Additionally, empty request and response bodies are now returned as empty bytes (`b''`) rather than `None`.
  Fix: Update your code to use `request.url` for full URLs and expect `b''` for empty bodies.
• gotcha Selenium Wire uses its own proxy server for request interception. Therefore, you cannot configure proxies directly via Selenium's `DesiredCapabilities` or `Options` objects (e.g., `chrome_options.add_argument('--proxy-server=...')`). Instead, all proxy configurations must be passed through the `seleniumwire_options` dictionary when initialising the WebDriver.
  Fix: Pass proxy settings within the `seleniumwire_options` dictionary, e.g., `driver = webdriver.Chrome(seleniumwire_options={'proxy': {'http': 'http://user:pass@host:port'}})`.
• gotcha For HTTPS request decryption and interception on Linux and macOS, OpenSSL is a required dependency. While often pre-installed, its absence will prevent Selenium Wire from capturing HTTPS traffic effectively.
  Fix: Ensure OpenSSL is installed on your system. On Debian/Ubuntu: `sudo apt-get install openssl`. On macOS: `brew install openssl`.
• gotcha There have been reports of `ModuleNotFoundError: No module named blinker._saferef` with certain `blinker` versions. This dependency conflict can prevent Selenium Wire from starting.
  Fix: If encountering this error, try downgrading `blinker`: `pip install blinker==1.7.0`.
• gotcha Request and response capture is asynchronous. If you access `driver.requests` immediately after a page load or action, some requests or their responses might not yet be fully captured. You might need to implement explicit or implicit waits.
  Fix: Use Selenium's explicit waits (e.g., `WebDriverWait`) to wait for specific requests to appear or for the DOM to update, implying that network activity has settled. Alternatively, use `driver.wait_for_request()` for specific request patterns.

Install

• pip install selenium-wire Install Selenium Wire

Imports

• webdriver

  from seleniumwire import webdriver

  Importing `webdriver` from `seleniumwire` is essential to enable network interception and other extended features. Importing from `selenium` will result in a standard Selenium WebDriver without Selenium Wire's enhancements.
• ChromeOptions

  from selenium.webdriver.chrome.options import Options

  Standard Selenium Options classes are used for browser-specific configurations, which can then be passed to `seleniumwire.webdriver.Chrome()` or `Firefox()`.

Quickstart

This quickstart demonstrates how to initialize a Selenium Wire WebDriver for Chrome, navigate to a page, and then iterate through the captured network requests and their responses. It also includes an example of setting a request interceptor to modify headers for a specific URL, showing how to gain programmatic control over network traffic. Remember to have a compatible browser driver (e.g., ChromeDriver) installed and accessible in your system PATH or specified via `executable_path`.

import os
from seleniumwire import webdriver
from selenium.webdriver.chrome.options import Options

# Configure Chrome options (e.g., headless mode)
chrome_options = Options()
# chrome_options.add_argument('--headless')

# Configure Selenium Wire options (e.g., disable capture to start)
# For proxy with authentication, use env vars or direct string:
# PROXY_HOST = os.environ.get('PROXY_HOST', 'your_proxy_host')
# PROXY_PORT = os.environ.get('PROXY_PORT', '8080')
# PROXY_USER = os.environ.get('PROXY_USER', 'your_proxy_username')
# PROXY_PASS = os.environ.get('PROXY_PASS', 'your_proxy_password')
# seleniumwire_options = {
#     'proxy': {
#         'http': f'http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}',
#         'https': f'https://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}',
#         'no_proxy': 'localhost,127.0.0.1'
#     }
# }

# Create a new instance of the Chrome driver with Selenium Wire
driver = webdriver.Chrome(options=chrome_options, seleniumwire_options={'disable_capture': False})

try:
    # Go to a webpage
    driver.get('https://www.google.com')

    # Access requests via the `requests` attribute
    print('Captured Requests:')
    for request in driver.requests:
        if request.response:
            print(f'URL: {request.url}, Status: {request.response.status_code}, Content-Type: {request.response.headers.get("Content-Type")}')

    # Example: Intercept a request to add a header
    def interceptor(request):
        if request.url == 'https://www.google.com/':
            request.headers['X-Custom-Header'] = 'SeleniumWireTest'

    driver.request_interceptor = interceptor
    driver.get('https://httpbin.org/headers') # Navigate to a site that echoes headers
    print('\nIntercepted Request to httpbin.org/headers:')
    for request in driver.requests:
        if 'httpbin.org/headers' in request.url and request.response:
            print(f'URL: {request.url}, Status: {request.response.status_code}')
            # For demonstration, typically you'd parse response.body to confirm header
            print('Response body snippet (may contain X-Custom-Header if successfully applied):')
            try:
                print(request.response.body.decode('utf-8')[:200] + '...')
            except Exception as e:
                print(f'Could not decode response body: {e}')

finally:
    # Close the browser
    driver.quit()