GRequests: Asynchronous HTTP Requests with Gevent

Warnings

• gotcha Import `grequests` (and thus `gevent`) before `requests` or other networking libraries that might conflict with `gevent`'s monkey-patching. Incorrect import order can lead to unexpected blocking behavior or errors.
  Fix: Ensure `import grequests` is placed at the very beginning of your script or module, before any other imports that perform I/O operations.
• gotcha When using `grequests.map()`, failed requests (e.g., due to timeouts or connection errors) will result in `None` being present in the returned list of responses. You must explicitly handle these `None` values.
  Fix: Iterate through the results and check if each `response` object is `None` before attempting to access its attributes (e.g., `if response: print(response.status_code)`). Consider using an `exception_handler` with `grequests.map` for more granular error reporting.
• gotcha `grequests.imap()` returns a generator of responses, and the order in which responses are yielded is arbitrary; it does not correspond to the order of the input requests. This differs from `grequests.map()` which maintains order.
  Fix: If response order is critical, use `grequests.map()`. If processing responses as they arrive is sufficient and order is not important, `grequests.imap()` can be more performant. If you need the original index with `imap`-like behavior, use `grequests.imap_enumerated` (v0.7.0+), but note its `None` handling.
• gotcha Making a large number of concurrent requests without controlling the pool size (`size` parameter) can lead to 'Too many open files' errors, rate limiting by target servers, or servers closing connections prematurely.
  Fix: Use the `size` parameter in `grequests.map()` or `grequests.imap()` to limit the number of concurrent requests being made (e.g., `grequests.map(reqs, size=10)`). Adjust this value based on your system's capabilities and the target server's limitations.
• breaking The behavior of `grequests.imap` changed in version 0.4.0. While not explicitly detailed as a breaking change in releases, 'behavior changes' suggest that code relying on previous `imap` functionality might need adjustments.
  Fix: Review code using `grequests.imap` and test its behavior against version 0.4.0 or later to ensure it functions as expected. Refer to GitHub issue #111 if possible for specifics.
• deprecated The underlying `gevent` library has deprecated and removed support for older Python versions (e.g., Python 2.7, 3.6, and soon 3.9). While `grequests` itself doesn't always specify Python requirements, its compatibility is bound by `gevent`'s support.
  Fix: Upgrade to Python 3.7+ and ensure your `gevent` installation is compatible with your Python version. Consult `gevent`'s documentation for its specific Python compatibility matrix.

Install

• pip install grequests Install with pip

Imports

• grequests

  import grequests

  Due to gevent's monkey-patching, `grequests` (which imports `gevent`) should generally be imported before other networking libraries like `requests` to ensure proper asynchronous behavior.
• grequests.map

  responses = grequests.map(requests_list)

  Used to send a collection of Request objects concurrently and retrieve responses.
• grequests.imap

  for response in grequests.imap(requests_list):

  Returns a generator of responses, which can offer performance gains. Responses are not guaranteed to be in the order of the original requests.
• grequests.imap_enumerated

  for index, response in grequests.imap_enumerated(requests_list):

  Returns a generator yielding (index, response) tuples. Unlike `imap`, it yields `None` for failed requests. Introduced in v0.7.0.

Quickstart

This quickstart demonstrates how to use `grequests.map` and `grequests.imap_enumerated` to send multiple GET requests concurrently. It includes a custom exception handler and sets a timeout for individual requests. `grequests.map` returns a list of responses (or `None` for failed requests) in the order of the initial requests, while `grequests.imap_enumerated` yields (index, response) tuples as responses become available, not necessarily in the original request order.

import grequests
import time

urls = [
    'http://httpbin.org/delay/1',
    'http://httpbin.org/status/200',
    'http://httpbin.org/delay/3',
    'http://httpbin.org/status/500'
]

def exception_handler(request, exception):
    print(f"Request to {request.url} failed: {exception}")

start_time = time.time()

# Create a list of unsent Request objects
reqs = (
    grequests.get(u, timeout=2) for u in urls
)

# Send all requests concurrently using map
# Responses will be in the same order as requests, with None for failed ones
responses = grequests.map(reqs, exception_handler=exception_handler, size=5)

print(f"\n--- Responses (using map, took {time.time() - start_time:.2f}s) ---")
for response in responses:
    if response:
        print(f"URL: {response.url}, Status: {response.status_code}")
    else:
        print("Request failed or timed out.")

print("\n--- Responses (using imap_enumerated) ---")
# imap_enumerated yields (index, response) and includes Nones for failures
reqs_for_imap_enumerated = (
    grequests.get(u, timeout=2) for u in urls
)
for index, response in grequests.imap_enumerated(reqs_for_imap_enumerated, exception_handler=exception_handler, size=5):
    if response:
        print(f"Index: {index}, URL: {response.url}, Status: {response.status_code}")
    else:
        print(f"Index: {index}, Request failed or timed out.")