Asynchronous Python HTTP for Humans

1.0.2 · active · verified Thu Apr 09

Requests-futures is a small add-on for the popular 'requests' HTTP library, enabling asynchronous HTTP requests. It leverages Python's `concurrent.futures` module to perform requests concurrently using either `ThreadPoolExecutor` or `ProcessPoolExecutor`. It provides a `FuturesSession` class that mimics `requests.Session` but returns `Future` objects, allowing non-blocking operations and retrieval of responses later. The current version is 1.0.2, and its release cadence is infrequent but active.

Warnings

gotcha Using `ProcessPoolExecutor` requires careful handling of pickling for the session and request objects. Python 2.x and < 3.4 are not supported for `ProcessPoolExecutor`. For Python 3.4+, an existing `requests.Session` instance *must* be passed when initializing `FuturesSession` with a `ProcessPoolExecutor`.
Fix: Ensure all components (session, request data, hooks) are picklable. For Python 3.4 using `ProcessPoolExecutor`, instantiate `FuturesSession(executor=ProcessPoolExecutor(), session=requests.Session())`. Python 3.5+ handles this more gracefully.
breaking Exceptions are not raised immediately when a request is made. Instead, they are 'shifted' to when `future.result()` is called. This changes the expected location for `try/except` blocks.
Fix: Wrap calls to `future.result()` in a `try/except` block to catch network errors or other request-related exceptions.
gotcha The default number of workers for the `ThreadPoolExecutor` has changed. Older versions might have defaulted to 2 workers, while recent versions (e.g., 1.0.2) default to 8 workers.
Fix: Always explicitly set `max_workers` when initializing `FuturesSession` with `ThreadPoolExecutor` to ensure predictable concurrency levels, e.g., `FuturesSession(executor=ThreadPoolExecutor(max_workers=10))`.
gotcha As with standard `requests`, omitting `timeout` parameters can lead to requests hanging indefinitely, especially critical in concurrent applications where one stalled request can impact many others.
Fix: Always specify a `timeout` when making requests, e.g., `session.get(url, timeout=5)`. This timeout applies to `future.result()` as well.

Install

pip install requests-futures Install latest version

Imports

FuturesSession

from requests_futures.sessions import FuturesSession

Quickstart

Initialize a FuturesSession, submit requests which return Future objects, then iterate through the Futures and call `.result()` to obtain the Response objects once they are ready. Error handling for network issues should wrap the `.result()` call.

from requests_futures.sessions import FuturesSession
import os

session = FuturesSession()

# Example: Send multiple GET requests concurrently
urls = [
    'http://httpbin.org/get?id=1',
    'http://httpbin.org/get?id=2',
    'http://httpbin.org/get?id=3',
]

futures = [session.get(url) for url in urls]

for i, future in enumerate(futures):
    try:
        response = future.result() # Blocks until the request completes
        print(f"Request {i+1} Status: {response.status_code}")
        print(f"Request {i+1} Content: {response.json()['args']}")
    except Exception as e:
        print(f"Request {i+1} failed: {e}")

view raw JSON →