MPIRE - MultiProcessing Is Really Easy

2.10.2 · active · verified Fri Apr 10

MPIRE is a Python package that simplifies multiprocessing, offering a faster and more user-friendly alternative to the standard `multiprocessing` module. It provides an intuitive API with map-like functions, support for worker state, progress bars (via tqdm), worker insights, and efficient handling of shared objects. The library is actively maintained with regular updates and aims to make parallelizing CPU-bound tasks straightforward and performant.

Warnings

breaking The `func_pointer` parameter in `map` functions was renamed to `func`.
Fix: Update your code to use `func` instead of `func_pointer` when calling `WorkerPool.map`, `map_unordered`, `imap`, or `imap_unordered`.
deprecated The `restart_workers` parameter was deprecated and subsequently removed.
Fix: Remove the `restart_workers` parameter. Consider using `worker_lifespan` in the `WorkerPool` constructor for similar functionality of restarting workers after a specified number of tasks.
deprecated The `enable_insights` parameter moved from `map` functions to the `WorkerPool` constructor.
Fix: Pass `enable_insights=True` to the `WorkerPool` constructor instead of individual `map` calls.
gotcha Copy-on-write shared objects, a key performance feature, are only available when using the `fork` start method, which is not supported on Windows.
Fix: On Windows, shared objects will be copied once per worker instead of using copy-on-write. This might still be more efficient than copying per task, but understand the performance implications compared to Linux/macOS.
gotcha Mixing multiple `map` calls (e.g., `pool.map` followed by `pool.imap`) can raise an error due to internal state management.
Fix: Avoid mixing different types of `map` calls within the same `WorkerPool` instance. If you need to use different map styles, create separate `WorkerPool` instances for each.
gotcha When using `dill` as the serialization backend on Windows, exceptions raised by workers or exit functions might print additional `OSError` messages to the terminal.
Fix: These `OSError` messages can generally be safely ignored as they are related to internal cleanup processes.
gotcha The progress bar feature is not supported when using the `threading` start method on Windows.
Fix: If running on Windows and needing a progress bar, use a different start method (e.g., `spawn` or `forkserver` if available, though `fork` is not on Windows) or consider disabling the progress bar.

Install

pip install mpire Install MPIRE

Imports

WorkerPool
```
from mpire import WorkerPool
```
MPIRE's core class is `WorkerPool`, which extends `multiprocessing.Pool` with additional features and a different API. Directly using `multiprocessing.Pool` will not provide MPIRE's enhancements.

Quickstart

This quickstart demonstrates how to use `mpire.WorkerPool` to parallelize a simple function across multiple processes. It initializes a pool with 4 workers and uses the `map` function, similar to Python's built-in `map` but executed in parallel.

import time
from mpire import WorkerPool

def my_function(x):
    time.sleep(0.01) # Simulate some work
    return x * 2

# Run with 4 worker processes
with WorkerPool(n_jobs=4) as pool:
    results = pool.map(my_function, range(100))
    print(f"Results: {results[:5]}...")

view raw JSON →