pathos

0.3.5 verified Tue May 12 auth: no python install: verified

pathos is a framework for heterogeneous computing that provides tools for parallel graph management and execution. It offers a consistent high-level interface for configuring and launching parallel computations across diverse resources, aiming to extend user code to parallel and distributed computing with minimal refactoring. The library is currently at version 0.3.5 and has a consistent release cadence with minor versions released every few months, typically adding incremental features and dependency updates.

pip install pathos

Common errors

error Can't pickle local object ↓

cause This error or '_pickle.PicklingError' often occurs when pathos's multiprocessing falls back to Python's standard pickle module, which cannot serialize complex objects like local functions, lambda functions, or class methods, instead of using its more powerful dill serializer via multiprocess. This can happen if multiprocess is not correctly installed or compiled.

fix

Ensure dill and multiprocess are correctly installed: pip install dill multiprocess. On Windows, a C++ compiler might be required for multiprocess to compile correctly. Explicitly import from pathos.multiprocessing (e.g., from pathos.multiprocessing import ProcessingPool). If passing class methods, define them at the top level of a module or make them static/class methods.

error ModuleNotFoundError: No module named 'pathos.multiprocessing' ↓

cause This error (or `ImportError` for older Python versions) indicates that the `pathos` library or its `multiprocess` dependency is not installed correctly, or an older version is being used where the module path was different.

fix

Install pathos and its dependencies using pip: pip install pathos. If issues persist, try upgrading setuptools (pip install --upgrade setuptools) then reinstalling pathos. Also, ensure that multiprocess and dill are explicitly installed: pip install multiprocess dill.

error ImportError: sys.meta_path is None, Python is likely shutting down ↓

cause This error often occurs when a `pathos` pool (especially `ProcessPool`) is used within a `with` statement, and Python attempts to shut down processes while resources are still being accessed or cleaned up, leading to issues with module imports during interpreter shutdown.

fix

Manually manage the pool lifecycle by calling pool.close() and pool.join() explicitly after the parallel computation is complete, instead of solely relying on the with statement's implicit exit.

Warnings

breaking The minimum required Python version for `pathos` has progressively increased. As of version 0.3.5, Python 3.9 or newer is required. ↓

fix Upgrade your Python environment to version 3.9 or higher. For compatibility with older Python versions, install an earlier `pathos` release (e.g., `pip install 'pathos<0.3.5'` for Python 3.8, or `pip install 'pathos<0.3.1'` for Python 3.7).

gotcha When using `pathos.multiprocessing.ProcessingPool`, objects passed to worker processes are *copied* (serialized and deserialized) rather than shared in memory. This means modifications to these objects within a worker process will not reflect in the original object in the parent process or other workers. ↓

fix Design your parallel functions to return explicit results from worker processes. If shared mutable state is absolutely necessary, consider using `multiprocess.Manager` objects (e.g., `Manager().dict()`, `Manager().list()`) or explicit inter-process communication mechanisms like queues or pipes, understanding their inherent overhead.

gotcha `pathos`'s `map` methods (e.g., `ProcessingPool.map`) directly accept multiple iterables as arguments for functions with multiple parameters, which differs from the standard `multiprocessing.Pool.map` signature that expects a single iterable of arguments. Users accustomed to `multiprocessing` might attempt workarounds like `itertools.starmap` or tuple unpacking in the target function, which are unnecessary and potentially less efficient with `pathos`. ↓

fix Pass each argument iterable as a separate argument to the `pool.map()` call. For a function `f(a, b)`, you can call `pool.map(f, iterable_a, iterable_b)` directly, where `iterable_a` and `iterable_b` are sequences of arguments for `a` and `b` respectively.

gotcha `pathos` leverages `dill` for object serialization, which is significantly more powerful than Python's default `pickle` used by standard `multiprocessing`. This allows `pathos` to reliably serialize and transfer complex objects, lambda functions, nested functions, and class methods to worker processes, which often cause `PicklingError` exceptions with plain `multiprocessing`. ↓

fix This is a feature of `pathos` that solves a common `multiprocessing` problem. If you encounter serialization issues with standard `multiprocessing`, `pathos` is designed to handle such complex objects transparently. Ensure `dill` is correctly installed and `pathos`'s pool implementations are utilized.

Install compatibility verified last tested: 2026-05-12

python os / libc status wheel install import disk

3.10 alpine (musl) wheel - 0.20s 21.0M

3.10 alpine (musl) - - 0.22s 21.0M

3.10 slim (glibc) wheel 1.8s 0.17s 21M

3.10 slim (glibc) - - 0.16s 21M

3.11 alpine (musl) wheel - 0.28s 23.7M

3.11 alpine (musl) - - 0.31s 23.7M

3.11 slim (glibc) wheel 1.9s 0.26s 24M

3.11 slim (glibc) - - 0.26s 24M

3.12 alpine (musl) wheel - 0.30s 15.5M

3.12 alpine (musl) - - 0.28s 15.5M

3.12 slim (glibc) wheel 1.8s 0.28s 16M

3.12 slim (glibc) - - 0.29s 16M

3.13 alpine (musl) wheel - 0.26s 15.3M

3.13 alpine (musl) - - 0.27s 15.2M

3.13 slim (glibc) wheel 1.9s 0.28s 16M

3.13 slim (glibc) - - 0.29s 16M

3.9 alpine (musl) wheel - 0.20s 20.5M

3.9 alpine (musl) - - 0.22s 20.5M

3.9 slim (glibc) wheel 2.1s 0.20s 21M

3.9 slim (glibc) - - 0.20s 21M

Imports

ProcessingPool
wrong
```
from multiprocessing import Pool
```
correct
```
from pathos.multiprocessing import ProcessingPool as Pool
```
While 'pathos.multiprocessing' is an extension of Python's built-in 'multiprocessing', direct import from 'multiprocessing' will lack pathos's enhancements like advanced serialization and multi-argument map support. The 'as Pool' alias is common practice for brevity.
ParallelPool
```
from pathos.pools import ParallelPool
```
This provides a convenient interface for creating a pool of worker processes, similar to ProcessingPool.

Quickstart last tested: 2026-04-24

This example demonstrates how to use `pathos.multiprocessing.ProcessingPool` to parallelize a function with multiple arguments using its enhanced `map` method. The `nodes` parameter configures the number of worker processes. Ensure the `if __name__ == '__main__':` block is used for multiprocessing compatibility.

from pathos.multiprocessing import ProcessingPool as Pool

def calculate_power(base, exponent):
    return base ** exponent

if __name__ == '__main__':
    bases = [1, 2, 3, 4, 5]
    exponents = [2, 3, 2, 4, 3]

    # Initialize a pool with a number of worker processes (e.g., 4)
    pool = Pool(nodes=4)

    # Use the map method to apply calculate_power in parallel
    # pathos's map directly accepts multiple iterables for multiple arguments
    results = pool.map(calculate_power, bases, exponents)

    print(f"Bases: {bases}")
    print(f"Exponents: {exponents}")
    print(f"Parallel Results: {results}")

    # Don't forget to close and join the pool when done
    pool.close()
    pool.join()