HTTPX Throttle Cache Client

Common errors

• ModuleNotFoundError: No module named 'hishel'

  cause Attempting to use `cache_mode='Hishel-File'` or `cache_mode='Hishel-S3'` without installing the `hishel` library.
  fix Install the optional `hishel` dependency: `pip install hishel`.

• pyrate_limiter.exceptions.BucketFullException: Too many requests for bucket 'default'

  cause The rate limiter has prevented a request because it would exceed the configured `request_per_sec_limit` or `burst_limit`.
  fix This is expected behavior for rate limiting. To mitigate, decrease the request rate, increase `request_per_sec_limit`, or implement retry logic with backoff in your application.

• TypeError: __init__ got an unexpected keyword argument 'http2'

  cause Passing an `http2` argument directly to `HttpxThrottleCache` or its `http_client()` method in a way that is not supported by the underlying `httpx` version or `httpxthrottlecache` wrapper.
  fix Check the `httpxthrottlecache` documentation or `httpx` documentation for the correct way to enable HTTP/2 if needed. In `httpx`, HTTP/2 support is typically enabled by installing `httpx[http2]` and configuring the client with `http2=True` or `http1=False, http2=True` if supported by `httpxthrottlecache`'s current API. Ensure you've installed `httpx[http2]` if you intend to use HTTP/2.

Warnings

• breaking The default caching backend changed in v0.3.0. Previously, `hishel` was the default, but it was removed as a default due to breaking API changes. The new default is `FileCache`. If your application relied on `hishel` being the default, you must explicitly set `cache_mode="Hishel-File"` and ensure `hishel` is installed.
  Fix: For `hishel` caching, ensure `pip install hishel` and initialize `HttpxThrottleCache(cache_mode="Hishel-File", ...)`. Otherwise, update your code to account for `FileCache` behavior or explicitly choose a different `cache_mode`.
• breaking Version 0.3.5 bumps the internal `pyrate-limiter` dependency to 4.x. If you are also directly using `pyrate-limiter` in your project and have an older version installed, you might encounter compatibility issues. Older versions of `pyrate-limiter` are likely incompatible with `httpxthrottlecache>=0.3.5`.
  Fix: Ensure your `pyrate-limiter` installation is version 4.x or higher when using `httpxthrottlecache>=0.3.5`. If you need to maintain an older `pyrate-limiter` version, consider pinning `httpxthrottlecache<0.3.5`.
• gotcha The `FileCache` implementation does not perform any automatic cache cleanup. Cached files will persist on disk indefinitely unless manually removed by the user.
  Fix: Implement your own cache cleanup strategy (e.g., a scheduled job to clear old files from the `cache_dir`) if using `FileCache` and needing to manage disk space.

Install

• pip install httpxthrottlecache Install stable version

Imports

• HttpxThrottleCache

  from httpxthrottlecache import HttpxThrottleCache

Quickstart

Initializes the HttpxThrottleCache client with file-based caching and rate limiting. It demonstrates a synchronous GET request to an example URL, printing the status code and headers. Users can configure cache mode, directory, rate limits, user agent, and specific caching rules using regular expressions. The client can be used as a context manager to ensure proper resource handling.

import os
from httpxthrottlecache import HttpxThrottleCache

# Example usage with a dummy URL and basic configuration
# For real usage, replace 'https://api.example.com' and 'your_user_agent'
url = "https://httpbingo.org/get"

with HttpxThrottleCache(
    cache_mode="FileCache", # Use 'FileCache' or 'Hishel-File' (if hishel is installed)
    cache_dir="_my_cache",
    rate_limiter_enabled=True,
    request_per_sec_limit=5,
    user_agent=os.environ.get('MY_APP_USER_AGENT', 'httpxthrottlecache-example/1.0'),
    # Example cache_rules to cache all paths for an hour
    cache_rules={
        '.*': {
            '.*': 3600 # Cache all responses for 3600 seconds (1 hour)
        }
    }
) as manager:
    with manager.http_client() as client:
        response = client.get(url)
        print(f"Status Code: {response.status_code}")
        print(f"Response headers: {response.headers}")
        # Uncomment to see response body:
        # print(response.json())