{"id":917,"library":"pathos","title":"pathos","description":"pathos is a framework for heterogeneous computing that provides tools for parallel graph management and execution. It offers a consistent high-level interface for configuring and launching parallel computations across diverse resources, aiming to extend user code to parallel and distributed computing with minimal refactoring. The library is currently at version 0.3.5 and has a consistent release cadence with minor versions released every few months, typically adding incremental features and dependency updates.","status":"active","version":"0.3.5","language":"python","source_language":"en","source_url":"https://github.com/uqfoundation/pathos","tags":["parallel-processing","multiprocessing","distributed-computing","high-performance-computing","framework","serialization"],"install":[{"cmd":"pip install pathos","lang":"bash","label":"Standard Install"}],"dependencies":[{"reason":"Runtime requirement","package":"python","optional":false,"min_version":"3.9"},{"reason":"Advanced serialization for objects and functions","package":"dill","optional":false,"min_version":"0.4.1"},{"reason":"Utilities for filesystem exploration and automated builds","package":"pox","optional":false,"min_version":"0.3.7"},{"reason":"Distributed and parallel Python","package":"ppft","optional":false,"min_version":"1.7.8"},{"reason":"Enhanced multiprocessing and multithreading capabilities","package":"multiprocess","optional":false,"min_version":"0.70.19"}],"imports":[{"note":"While 'pathos.multiprocessing' is an extension of Python's built-in 'multiprocessing', direct import from 'multiprocessing' will lack pathos's enhancements like advanced serialization and multi-argument map support. The 'as Pool' alias is common practice for brevity.","wrong":"from multiprocessing import Pool","symbol":"ProcessingPool","correct":"from pathos.multiprocessing import ProcessingPool as Pool"},{"note":"This provides a convenient interface for creating a pool of worker processes, similar to ProcessingPool.","symbol":"ParallelPool","correct":"from pathos.pools import ParallelPool"}],"quickstart":{"code":"from pathos.multiprocessing import ProcessingPool as Pool\n\ndef calculate_power(base, exponent):\n    return base ** exponent\n\nif __name__ == '__main__':\n    bases = [1, 2, 3, 4, 5]\n    exponents = [2, 3, 2, 4, 3]\n\n    # Initialize a pool with a number of worker processes (e.g., 4)\n    pool = Pool(nodes=4)\n\n    # Use the map method to apply calculate_power in parallel\n    # pathos's map directly accepts multiple iterables for multiple arguments\n    results = pool.map(calculate_power, bases, exponents)\n\n    print(f\"Bases: {bases}\")\n    print(f\"Exponents: {exponents}\")\n    print(f\"Parallel Results: {results}\")\n\n    # Don't forget to close and join the pool when done\n    pool.close()\n    pool.join()","lang":"python","description":"This example demonstrates how to use `pathos.multiprocessing.ProcessingPool` to parallelize a function with multiple arguments using its enhanced `map` method. The `nodes` parameter configures the number of worker processes. Ensure the `if __name__ == '__main__':` block is used for multiprocessing compatibility."},"warnings":[{"fix":"Upgrade your Python environment to version 3.9 or higher. For compatibility with older Python versions, install an earlier `pathos` release (e.g., `pip install 'pathos<0.3.5'` for Python 3.8, or `pip install 'pathos<0.3.1'` for Python 3.7).","message":"The minimum required Python version for `pathos` has progressively increased. As of version 0.3.5, Python 3.9 or newer is required.","severity":"breaking","affected_versions":"0.3.5+"},{"fix":"Design your parallel functions to return explicit results from worker processes. If shared mutable state is absolutely necessary, consider using `multiprocess.Manager` objects (e.g., `Manager().dict()`, `Manager().list()`) or explicit inter-process communication mechanisms like queues or pipes, understanding their inherent overhead.","message":"When using `pathos.multiprocessing.ProcessingPool`, objects passed to worker processes are *copied* (serialized and deserialized) rather than shared in memory. This means modifications to these objects within a worker process will not reflect in the original object in the parent process or other workers.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Pass each argument iterable as a separate argument to the `pool.map()` call. For a function `f(a, b)`, you can call `pool.map(f, iterable_a, iterable_b)` directly, where `iterable_a` and `iterable_b` are sequences of arguments for `a` and `b` respectively.","message":"`pathos`'s `map` methods (e.g., `ProcessingPool.map`) directly accept multiple iterables as arguments for functions with multiple parameters, which differs from the standard `multiprocessing.Pool.map` signature that expects a single iterable of arguments. Users accustomed to `multiprocessing` might attempt workarounds like `itertools.starmap` or tuple unpacking in the target function, which are unnecessary and potentially less efficient with `pathos`.","severity":"gotcha","affected_versions":"All versions"},{"fix":"This is a feature of `pathos` that solves a common `multiprocessing` problem. If you encounter serialization issues with standard `multiprocessing`, `pathos` is designed to handle such complex objects transparently. Ensure `dill` is correctly installed and `pathos`'s pool implementations are utilized.","message":"`pathos` leverages `dill` for object serialization, which is significantly more powerful than Python's default `pickle` used by standard `multiprocessing`. This allows `pathos` to reliably serialize and transfer complex objects, lambda functions, nested functions, and class methods to worker processes, which often cause `PicklingError` exceptions with plain `multiprocessing`.","severity":"gotcha","affected_versions":"All versions"}],"env_vars":null,"last_verified":"2026-05-12T21:17:06.118Z","next_check":"2026-06-27T00:00:00.000Z","problems":[{"fix":"Ensure `dill` and `multiprocess` are correctly installed: `pip install dill multiprocess`. On Windows, a C++ compiler might be required for `multiprocess` to compile correctly. Explicitly import from `pathos.multiprocessing` (e.g., `from pathos.multiprocessing import ProcessingPool`). If passing class methods, define them at the top level of a module or make them static/class methods.","cause":"This error or '_pickle.PicklingError' often occurs when pathos's multiprocessing falls back to Python's standard pickle module, which cannot serialize complex objects like local functions, lambda functions, or class methods, instead of using its more powerful dill serializer via multiprocess. This can happen if multiprocess is not correctly installed or compiled.","error":"Can't pickle local object"},{"fix":"Install `pathos` and its dependencies using pip: `pip install pathos`. If issues persist, try upgrading setuptools (`pip install --upgrade setuptools`) then reinstalling `pathos`. Also, ensure that `multiprocess` and `dill` are explicitly installed: `pip install multiprocess dill`.","cause":"This error (or `ImportError` for older Python versions) indicates that the `pathos` library or its `multiprocess` dependency is not installed correctly, or an older version is being used where the module path was different.","error":"ModuleNotFoundError: No module named 'pathos.multiprocessing'"},{"fix":"Manually manage the pool lifecycle by calling `pool.close()` and `pool.join()` explicitly after the parallel computation is complete, instead of solely relying on the `with` statement's implicit exit.","cause":"This error often occurs when a `pathos` pool (especially `ProcessPool`) is used within a `with` statement, and Python attempts to shut down processes while resources are still being accessed or cleaned up, leading to issues with module imports during interpreter shutdown.","error":"ImportError: sys.meta_path is None, Python is likely shutting down"}],"ecosystem":"pypi","meta_description":null,"install_score":100,"install_tag":"verified","quickstart_score":null,"quickstart_tag":null,"pypi_latest":"0.3.5","cli_name":"","install_checks":{"last_tested":"2026-05-12","tag":"verified","tag_description":"installs cleanly on critical runtimes, fast import, recently tested","results":[{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":null,"import_time_s":0.2,"mem_mb":7.4,"disk_size":"21.0M"},{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.22,"mem_mb":7.4,"disk_size":"21.0M"},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":1.8,"import_time_s":0.17,"mem_mb":7.4,"disk_size":"21M"},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.16,"mem_mb":7.4,"disk_size":"21M"},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":null,"import_time_s":0.28,"mem_mb":7.9,"disk_size":"23.7M"},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.31,"mem_mb":7.9,"disk_size":"23.7M"},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":1.9,"import_time_s":0.26,"mem_mb":7.9,"disk_size":"24M"},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.26,"mem_mb":7.9,"disk_size":"24M"},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":null,"import_time_s":0.3,"mem_mb":8,"disk_size":"15.5M"},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.28,"mem_mb":8,"disk_size":"15.5M"},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":1.8,"import_time_s":0.28,"mem_mb":8,"disk_size":"16M"},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.29,"mem_mb":8,"disk_size":"16M"},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":null,"import_time_s":0.26,"mem_mb":8.5,"disk_size":"15.3M"},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.27,"mem_mb":8.5,"disk_size":"15.2M"},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":1.9,"import_time_s":0.28,"mem_mb":8.5,"disk_size":"16M"},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.29,"mem_mb":8.5,"disk_size":"16M"},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":null,"import_time_s":0.2,"mem_mb":7.2,"disk_size":"20.5M"},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.22,"mem_mb":7.2,"disk_size":"20.5M"},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":2.1,"import_time_s":0.2,"mem_mb":7.2,"disk_size":"21M"},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.2,"mem_mb":7.2,"disk_size":"21M"}]},"quickstart_checks":{"last_tested":"2026-04-24","tag":null,"tag_description":null,"results":[{"runtime":"python:3.10-alpine","exit_code":0},{"runtime":"python:3.10-slim","exit_code":0},{"runtime":"python:3.11-alpine","exit_code":0},{"runtime":"python:3.11-slim","exit_code":0},{"runtime":"python:3.12-alpine","exit_code":0},{"runtime":"python:3.12-slim","exit_code":0},{"runtime":"python:3.13-alpine","exit_code":0},{"runtime":"python:3.13-slim","exit_code":0},{"runtime":"python:3.9-alpine","exit_code":0},{"runtime":"python:3.9-slim","exit_code":0}]}}