{"id":1896,"library":"aiomultiprocess","title":"aiomultiprocess","description":"aiomultiprocess is a Python library that provides an asynchronous version of the standard `multiprocessing` module, combining the benefits of `asyncio` for I/O-bound tasks and `multiprocessing` for CPU-bound tasks. It runs a full `asyncio` event loop on each child process, enabling high levels of concurrency and parallelism beyond the Global Interpreter Lock (GIL). The library is actively maintained, with its current version being 0.9.1.","status":"active","version":"0.9.1","language":"python","source_language":"en","source_url":"https://github.com/omnilib/aiomultiprocess","tags":["asyncio","multiprocessing","concurrency","worker-pool","python3","gil-break"],"install":[{"cmd":"pip install aiomultiprocess","lang":"bash","label":"Install with pip"}],"dependencies":[{"reason":"Requires Python 3.8 or newer for full functionality and compatibility.","package":"python","optional":false},{"reason":"Commonly used in examples for asynchronous network requests within worker processes.","package":"aiohttp","optional":true}],"imports":[{"note":"The primary class for managing a pool of worker processes.","symbol":"Pool","correct":"from aiomultiprocess import Pool"},{"note":"For running a single coroutine in a dedicated subprocess, similar to `multiprocessing.Process`.","symbol":"Process","correct":"from aiomultiprocess import Process"},{"note":"Similar to `Process` but specifically designed to return results from a coroutine task in a subprocess.","symbol":"Worker","correct":"from aiomultiprocess import Worker"}],"quickstart":{"code":"import asyncio\nfrom aiohttp import ClientSession\nfrom aiomultiprocess import Pool\n\nasync def fetch_url_content(url: str) -> str:\n    \"\"\"An example async coroutine to be run by the pool.\"\"\"\n    async with ClientSession() as session:\n        async with session.get(url) as response:\n            response.raise_for_status() # Raise an exception for bad status codes\n            return await response.text()\n\nasync def main():\n    urls = [\n        \"https://www.google.com\",\n        \"https://www.python.org\",\n        \"https://docs.python.org/3/library/asyncio.html\",\n        \"https://www.wikipedia.org\"\n    ]\n\n    print(f\"Fetching {len(urls)} URLs using aiomultiprocess Pool...\")\n    async with Pool(processes=2) as pool: # Use 2 processes for demonstration\n        # Use pool.map to apply the coroutine to each URL\n        async for result in pool.map(fetch_url_content, urls):\n            if result: # Check if result is not None (e.g., if an exception was caught by handler)\n                print(f\"Fetched content size: {len(result)} bytes for one URL.\")\n            else:\n                print(\"Failed to fetch content for a URL.\")\n\nif __name__ == '__main__':\n    # Ensure aiohttp is installed for this example: pip install aiohttp\n    try:\n        asyncio.run(main())\n    except ImportError:\n        print(\"Please install aiohttp for this example: pip install aiohttp\")\n    except Exception as e:\n        print(f\"An error occurred: {e}\")","lang":"python","description":"This quickstart demonstrates using `aiomultiprocess.Pool` to fetch multiple URLs concurrently across several processes. It defines an asynchronous function `fetch_url_content` that uses `aiohttp` to make an HTTP GET request. The `main` function then creates a `Pool` and uses its `map` method to distribute the `fetch_url_content` coroutine calls across the worker processes, iterating over results as they complete."},"warnings":[{"fix":"Ensure all functions, classes, and global objects passed to worker processes are defined at the top-level of a module and are importable. If 'forked' behavior is strictly required (e.g., for sharing non-pickleable resources that are copied by fork), call `aiomultiprocess.set_start_method('fork')` before creating any workers or pools.","message":"By default, aiomultiprocess uses 'spawned' processes on all platforms, which is different from the standard `multiprocessing` module's default 'forked' on Linux/macOS (for Python versions prior to 3.8). This means any objects or coroutines passed to child processes *must* be pickleable (importable from a fresh child process). Unpickleable objects will cause errors.","severity":"breaking","affected_versions":"<=0.9.1"},{"fix":"Avoid relying on shared mutable global state. Use queues (`multiprocessing.Manager().Queue()`) for explicit inter-process communication or pass data explicitly as arguments and return values. For resource initialization per process, use the `initializer` and `initargs` parameters of the `Pool` constructor.","message":"Global variables and shared state behave differently with spawned processes. Each child process gets its own independent memory space; changes to global variables in one process are not reflected in others. This can lead to unexpected behavior if not accounted for.","severity":"gotcha","affected_versions":"<=0.9.1"},{"fix":"To handle exceptions within the worker process itself, provide an `exception_handler` callable to the `Pool` (or `Process`/`Worker`) constructor. This handler will be called with the exception object before it's propagated back to the main process.","message":"Exceptions raised within worker processes are automatically caught and re-raised in the main process as `ProxyException` objects. While convenient, this might obscure the original traceback or prevent in-worker exception handling (e.g., for logging or reporting).","severity":"gotcha","affected_versions":"<=0.9.1"},{"fix":"Utilize the `maxtasksperchild` parameter when creating a `Pool`. Setting it to a positive integer will cause worker processes to exit and be respawned after completing the specified number of tasks, helping to release resources and prevent file handle exhaustion.","message":"Long-running tasks or high concurrency with `aiomultiprocess.Pool` can sometimes lead to 'OSError: [Errno 24] Too many open files' or memory leaks if worker processes are not periodically refreshed.","severity":"gotcha","affected_versions":"<=0.9.1"},{"fix":"Explicitly specify the event loop initializer using the `loop_initializer` parameter in the `Pool` (or `Process`/`Worker`) constructor. For `uvloop`, this would be `loop_initializer=uvloop.new_event_loop`.","message":"If you are using an alternative `asyncio` event loop implementation (e.g., `uvloop`), `aiomultiprocess` will not automatically use it in child processes. The default `asyncio` loop will be used instead, potentially leading to suboptimal performance.","severity":"gotcha","affected_versions":"<=0.9.1"}],"env_vars":null,"last_verified":"2026-05-20T13:02:22.951Z","next_check":"2026-07-08T00:00:00.000Z","problems":[{"fix":"Install the package using pip: 'pip install aiomultiprocess'.","cause":"The 'aiomultiprocess' package is not installed in the Python environment.","error":"ModuleNotFoundError: No module named 'aiomultiprocess'"},{"fix":"Ensure that the 'main()' coroutine is properly awaited by using 'asyncio.run(main())' in Python 3.7+, or 'loop = asyncio.get_event_loop(); loop.run_until_complete(main())' in Python 3.6.","cause":"The asynchronous function 'main()' was defined but not awaited, leading to a coroutine that was never executed.","error":"RuntimeWarning: coroutine 'main' was never awaited"},{"fix":"Install the 'aiohttp' package using pip: 'pip install aiohttp'.","cause":"The 'aiohttp' package, which is a dependency for 'aiomultiprocess', is not installed in the Python environment.","error":"ModuleNotFoundError: No module named 'aiohttp'"},{"fix":"Ensure that functions and objects intended for use in child processes are defined at the top-level of a module, making them importable by the new process. Avoid using `lambda` functions or nested function definitions as targets for multiprocessing tasks. If passing complex objects, ensure they are serializable or use an initializer function for setup within the child processes.","cause":"Objects (especially functions or class methods) passed to `aiomultiprocess.Pool` or `Process` must be picklable. This error often occurs when trying to pickle local, nested, or dynamically created functions/objects, or certain complex objects that cannot be serialized across process boundaries.","error":"_pickle.PicklingError: Can't pickle <function ...>: attribute lookup ... failed"},{"fix":"Ensure that `asyncio.run()` or `loop.run_until_complete()` is called only once per thread/process to manage the top-level event loop. In child processes managed by `aiomultiprocess`, the event loop is usually set up automatically; avoid explicitly calling `asyncio.run()` within the target functions executed by the pool or worker, unless you are deliberately managing separate loops in a highly specific way.","cause":"This error occurs when attempting to start a new `asyncio` event loop (e.g., with `asyncio.run()`) in a thread or process where an event loop is already active. This is a common `asyncio` pitfall that can occur when `aiomultiprocess` is misused, for example, if a child process inadvertently tries to create a new loop without proper handling.","error":"RuntimeError: Cannot run the event loop while another loop is running"}],"ecosystem":"pypi","meta_description":null,"install_score":null,"install_tag":null,"quickstart_score":null,"quickstart_tag":null,"pypi_latest":"0.9.1","cli_name":"","cli_version":null,"type":"library","homepage":null,"github":"https://github.com/omnilib/aiomultiprocess","docs":"https://aiomultiprocess.omnilib.dev/en/latest/","changelog":null,"pypi":"https://pypi.org/project/aiomultiprocess/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null,"categories":["workflow","devops"],"install_checks":{"last_tested":"2026-05-20","tag":null,"tag_description":null,"installed_version":"0.9.1","pypi_latest":"0.9.1","is_stale":false,"results":[{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":"aiomultiprocess","exit_code":0,"wheel_type":"wheel","failure_reason":null,"import_side_effects":"clean","install_time_s":null,"import_time_s":0.22,"mem_mb":7.4,"disk_size":"17.9M"},{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":"aiomultiprocess","exit_code":0,"wheel_type":null,"failure_reason":null,"import_side_effects":null,"install_time_s":null,"import_time_s":0.32,"mem_mb":7.4,"disk_size":"17.9M"},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":"aiomultiprocess","exit_code":0,"wheel_type":"wheel","failure_reason":null,"import_side_effects":"clean","install_time_s":1.5,"import_time_s":0.15,"mem_mb":7.4,"disk_size":"18M"},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":"aiomultiprocess","exit_code":0,"wheel_type":null,"failure_reason":null,"import_side_effects":null,"install_time_s":null,"import_time_s":0.18,"mem_mb":7.4,"disk_size":"18M"},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":"aiomultiprocess","exit_code":0,"wheel_type":"wheel","failure_reason":null,"import_side_effects":"clean","install_time_s":null,"import_time_s":0.31,"mem_mb":8.4,"disk_size":"19.8M"},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":"aiomultiprocess","exit_code":0,"wheel_type":null,"failure_reason":null,"import_side_effects":null,"install_time_s":null,"import_time_s":0.4,"mem_mb":8.4,"disk_size":"19.8M"},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":"aiomultiprocess","exit_code":0,"wheel_type":"wheel","failure_reason":null,"import_side_effects":"clean","install_time_s":1.6,"import_time_s":0.27,"mem_mb":8.4,"disk_size":"20M"},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":"aiomultiprocess","exit_code":0,"wheel_type":null,"failure_reason":null,"import_side_effects":null,"install_time_s":null,"import_time_s":0.3,"mem_mb":8.4,"disk_size":"20M"},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":"aiomultiprocess","exit_code":0,"wheel_type":"wheel","failure_reason":null,"import_side_effects":"clean","install_time_s":null,"import_time_s":0.55,"mem_mb":8.7,"disk_size":"11.7M"},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":"aiomultiprocess","exit_code":0,"wheel_type":null,"failure_reason":null,"import_side_effects":null,"install_time_s":null,"import_time_s":0.71,"mem_mb":8.7,"disk_size":"11.7M"},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":"aiomultiprocess","exit_code":0,"wheel_type":"wheel","failure_reason":null,"import_side_effects":"clean","install_time_s":1.4,"import_time_s":0.48,"mem_mb":8.7,"disk_size":"12M"},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":"aiomultiprocess","exit_code":0,"wheel_type":null,"failure_reason":null,"import_side_effects":null,"install_time_s":null,"import_time_s":0.6,"mem_mb":8.7,"disk_size":"12M"},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":"aiomultiprocess","exit_code":0,"wheel_type":"wheel","failure_reason":null,"import_side_effects":"clean","install_time_s":null,"import_time_s":0.55,"mem_mb":9.2,"disk_size":"11.4M"},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":"aiomultiprocess","exit_code":0,"wheel_type":null,"failure_reason":null,"import_side_effects":null,"install_time_s":null,"import_time_s":0.72,"mem_mb":9.2,"disk_size":"11.3M"},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":"aiomultiprocess","exit_code":0,"wheel_type":"wheel","failure_reason":null,"import_side_effects":"clean","install_time_s":1.5,"import_time_s":0.5,"mem_mb":9.2,"disk_size":"12M"},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":"aiomultiprocess","exit_code":0,"wheel_type":null,"failure_reason":null,"import_side_effects":null,"install_time_s":null,"import_time_s":0.57,"mem_mb":9.2,"disk_size":"12M"},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":"aiomultiprocess","exit_code":0,"wheel_type":"wheel","failure_reason":null,"import_side_effects":"clean","install_time_s":null,"import_time_s":0.2,"mem_mb":7.2,"disk_size":"17.4M"},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":"aiomultiprocess","exit_code":0,"wheel_type":null,"failure_reason":null,"import_side_effects":null,"install_time_s":null,"import_time_s":0.29,"mem_mb":7.2,"disk_size":"17.4M"},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":"aiomultiprocess","exit_code":0,"wheel_type":"wheel","failure_reason":null,"import_side_effects":"clean","install_time_s":1.8,"import_time_s":0.16,"mem_mb":7.2,"disk_size":"18M"},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":"aiomultiprocess","exit_code":0,"wheel_type":null,"failure_reason":null,"import_side_effects":null,"install_time_s":null,"import_time_s":0.2,"mem_mb":7.2,"disk_size":"18M"}]},"quickstart_checks":{"last_tested":"2026-04-25","tag":null,"tag_description":null,"results":[{"runtime":"python:3.10-alpine","exit_code":1},{"runtime":"python:3.10-slim","exit_code":1},{"runtime":"python:3.11-alpine","exit_code":1},{"runtime":"python:3.11-slim","exit_code":1},{"runtime":"python:3.12-alpine","exit_code":1},{"runtime":"python:3.12-slim","exit_code":1},{"runtime":"python:3.13-alpine","exit_code":1},{"runtime":"python:3.13-slim","exit_code":1},{"runtime":"python:3.9-alpine","exit_code":1},{"runtime":"python:3.9-slim","exit_code":1}]},"_links":{"self":"https://checklist.day/api/registry/aiomultiprocess","v1":"https://checklist.day/v1/registry/aiomultiprocess","v1_install":"https://checklist.day/v1/registry/aiomultiprocess/install","v1_imports":"https://checklist.day/v1/registry/aiomultiprocess/imports","v1_compatibility":"https://checklist.day/v1/registry/aiomultiprocess/compatibility","v1_quickstart":"https://checklist.day/v1/registry/aiomultiprocess/quickstart","docs":"https://checklist.day/docs"}}