aiomultiprocess

raw JSON →
0.9.1 verified Fri Apr 24 auth: no python

aiomultiprocess is a Python library that provides an asynchronous version of the standard `multiprocessing` module, combining the benefits of `asyncio` for I/O-bound tasks and `multiprocessing` for CPU-bound tasks. It runs a full `asyncio` event loop on each child process, enabling high levels of concurrency and parallelism beyond the Global Interpreter Lock (GIL). The library is actively maintained, with its current version being 0.9.1.

pip install aiomultiprocess

Common errors

error ModuleNotFoundError: No module named 'aiomultiprocess' ↓
cause The 'aiomultiprocess' package is not installed in the Python environment.
fix
Install the package using pip: 'pip install aiomultiprocess'.
error RuntimeWarning: coroutine 'main' was never awaited ↓
cause The asynchronous function 'main()' was defined but not awaited, leading to a coroutine that was never executed.
fix
Ensure that the 'main()' coroutine is properly awaited by using 'asyncio.run(main())' in Python 3.7+, or 'loop = asyncio.get_event_loop(); loop.run_until_complete(main())' in Python 3.6.
error ModuleNotFoundError: No module named 'aiohttp' ↓
cause The 'aiohttp' package, which is a dependency for 'aiomultiprocess', is not installed in the Python environment.
fix
Install the 'aiohttp' package using pip: 'pip install aiohttp'.
error _pickle.PicklingError: Can't pickle <function ...>: attribute lookup ... failed ↓
cause Objects (especially functions or class methods) passed to `aiomultiprocess.Pool` or `Process` must be picklable. This error often occurs when trying to pickle local, nested, or dynamically created functions/objects, or certain complex objects that cannot be serialized across process boundaries.
fix
Ensure that functions and objects intended for use in child processes are defined at the top-level of a module, making them importable by the new process. Avoid using lambda functions or nested function definitions as targets for multiprocessing tasks. If passing complex objects, ensure they are serializable or use an initializer function for setup within the child processes.
error RuntimeError: Cannot run the event loop while another loop is running ↓
cause This error occurs when attempting to start a new `asyncio` event loop (e.g., with `asyncio.run()`) in a thread or process where an event loop is already active. This is a common `asyncio` pitfall that can occur when `aiomultiprocess` is misused, for example, if a child process inadvertently tries to create a new loop without proper handling.
fix
Ensure that asyncio.run() or loop.run_until_complete() is called only once per thread/process to manage the top-level event loop. In child processes managed by aiomultiprocess, the event loop is usually set up automatically; avoid explicitly calling asyncio.run() within the target functions executed by the pool or worker, unless you are deliberately managing separate loops in a highly specific way.

Warnings

breaking By default, aiomultiprocess uses 'spawned' processes on all platforms, which is different from the standard `multiprocessing` module's default 'forked' on Linux/macOS (for Python versions prior to 3.8). This means any objects or coroutines passed to child processes *must* be pickleable (importable from a fresh child process). Unpickleable objects will cause errors. ↓
fix Ensure all functions, classes, and global objects passed to worker processes are defined at the top-level of a module and are importable. If 'forked' behavior is strictly required (e.g., for sharing non-pickleable resources that are copied by fork), call `aiomultiprocess.set_start_method('fork')` before creating any workers or pools.
gotcha Global variables and shared state behave differently with spawned processes. Each child process gets its own independent memory space; changes to global variables in one process are not reflected in others. This can lead to unexpected behavior if not accounted for. ↓
fix Avoid relying on shared mutable global state. Use queues (`multiprocessing.Manager().Queue()`) for explicit inter-process communication or pass data explicitly as arguments and return values. For resource initialization per process, use the `initializer` and `initargs` parameters of the `Pool` constructor.
gotcha Exceptions raised within worker processes are automatically caught and re-raised in the main process as `ProxyException` objects. While convenient, this might obscure the original traceback or prevent in-worker exception handling (e.g., for logging or reporting). ↓
fix To handle exceptions within the worker process itself, provide an `exception_handler` callable to the `Pool` (or `Process`/`Worker`) constructor. This handler will be called with the exception object before it's propagated back to the main process.
gotcha Long-running tasks or high concurrency with `aiomultiprocess.Pool` can sometimes lead to 'OSError: [Errno 24] Too many open files' or memory leaks if worker processes are not periodically refreshed. ↓
fix Utilize the `maxtasksperchild` parameter when creating a `Pool`. Setting it to a positive integer will cause worker processes to exit and be respawned after completing the specified number of tasks, helping to release resources and prevent file handle exhaustion.
gotcha If you are using an alternative `asyncio` event loop implementation (e.g., `uvloop`), `aiomultiprocess` will not automatically use it in child processes. The default `asyncio` loop will be used instead, potentially leading to suboptimal performance. ↓
fix Explicitly specify the event loop initializer using the `loop_initializer` parameter in the `Pool` (or `Process`/`Worker`) constructor. For `uvloop`, this would be `loop_initializer=uvloop.new_event_loop`.

Install compatibility last tested: 2026-04-24

runtime status import time mem disk
3.10-alpine 0.32s 7.4MB 17.9M
3.10-slim 0.18s 7.4MB 18M
3.11-alpine 0.40s 8.4MB 19.8M
3.11-slim 0.30s 8.4MB 20M
3.12-alpine 0.71s 8.7MB 11.7M
3.12-slim 0.60s 8.7MB 12M
3.13-alpine 0.72s 9.2MB 11.3M
3.13-slim 0.57s 9.2MB 12M
3.9-alpine 0.29s 7.2MB 17.4M
3.9-slim 0.20s 7.2MB 18M

Imports

Quickstart last tested: 2026-04-25

This quickstart demonstrates using `aiomultiprocess.Pool` to fetch multiple URLs concurrently across several processes. It defines an asynchronous function `fetch_url_content` that uses `aiohttp` to make an HTTP GET request. The `main` function then creates a `Pool` and uses its `map` method to distribute the `fetch_url_content` coroutine calls across the worker processes, iterating over results as they complete.

import asyncio
from aiohttp import ClientSession
from aiomultiprocess import Pool

async def fetch_url_content(url: str) -> str:
    """An example async coroutine to be run by the pool."""
    async with ClientSession() as session:
        async with session.get(url) as response:
            response.raise_for_status() # Raise an exception for bad status codes
            return await response.text()

async def main():
    urls = [
        "https://www.google.com",
        "https://www.python.org",
        "https://docs.python.org/3/library/asyncio.html",
        "https://www.wikipedia.org"
    ]

    print(f"Fetching {len(urls)} URLs using aiomultiprocess Pool...")
    async with Pool(processes=2) as pool: # Use 2 processes for demonstration
        # Use pool.map to apply the coroutine to each URL
        async for result in pool.map(fetch_url_content, urls):
            if result: # Check if result is not None (e.g., if an exception was caught by handler)
                print(f"Fetched content size: {len(result)} bytes for one URL.")
            else:
                print("Failed to fetch content for a URL.")

if __name__ == '__main__':
    # Ensure aiohttp is installed for this example: pip install aiohttp
    try:
        asyncio.run(main())
    except ImportError:
        print("Please install aiohttp for this example: pip install aiohttp")
    except Exception as e:
        print(f"An error occurred: {e}")
last verified Fri Apr 24
next check Wed Jul 08