Multiprocess

0.70.19 · active · verified Sat Mar 28

Multiprocess is a Python library that serves as a friendly fork of the standard `multiprocessing` module, primarily providing enhanced and more robust serialization capabilities through the use of `dill`. It aims to be a drop-in replacement for `multiprocessing` in many scenarios, offering better handling of complex objects and functions. The library is actively maintained, with regular releases (several per year) addressing updates and Python version compatibility. The current version is 0.70.19.

Warnings

breaking Python 3.7 and older versions are no longer formally supported. Version 0.70.19 requires Python >=3.9. Previous versions had different minimum requirements (e.g., 0.70.18 required >=3.8, 0.70.16 dropped 3.7).
Fix: Ensure your environment uses Python 3.9 or newer. Upgrade your Python interpreter if necessary.
breaking The minimum required version for the `dill` dependency has steadily increased across `multiprocess` releases. Version 0.70.19 requires `dill >=0.4.1`. Older `multiprocess` versions may have different `dill` requirements.
Fix: Upgrade your `dill` package to the latest version (`pip install --upgrade dill`) or at least to the version specified by your `multiprocess` installation.
gotcha When using `multiprocess` (or standard `multiprocessing`), the code that spawns child processes must be protected inside an `if __name__ == '__main__':` block. This is crucial for Windows compatibility and to prevent infinite recursion when child processes import the main script.
Fix: Always wrap your process-spawning logic within `if __name__ == '__main__':`.
gotcha For 'frozen executables' (e.g., created with PyInstaller, cx_Freeze), `multiprocess` requires explicit early import of `multiprocess.forking` to set up proper process spawning mechanisms, otherwise processes may fail to start or behave unexpectedly.
Fix: Include `import multiprocess.forking` at the very beginning of your main script when creating frozen executables.
gotcha While `multiprocess` aims to be a 'drop-in replacement' and offers 'better serialization' compared to the standard `multiprocessing` module, users migrating or relying on very specific `pickle` behaviors might encounter subtle differences. `multiprocess` uses `dill` which has broader serialization capabilities, but this can also lead to different behavior for edge cases or non-standard objects.
Fix: Thoroughly test your application after switching from `multiprocessing` to `multiprocess`, especially if you pass complex or custom objects between processes. Review `dill`'s documentation for its serialization specifics.

Install

pip install multiprocess Install latest version

Imports

Process
```
from multiprocess import Process
```
While often a drop-in replacement, explicit import from `multiprocess` is required to leverage its enhanced features.
Pool
```
from multiprocess import Pool
```
To utilize the `multiprocess` Pool with its serialization advantages, import directly from `multiprocess`.
Queue
```
from multiprocess import Queue
```

Quickstart

This quickstart demonstrates how to create and manage processes using `multiprocess.Process`. Each worker function runs in its own process, executing the `worker_function` with a given name. The `if __name__ == "__main__":` block is essential for proper process spawning on certain operating systems (especially Windows) and to prevent recursive imports.

import multiprocess
import os
import time

def worker_function(name):
    """A function to be run in a separate process."""
    print(f"Process {os.getpid()}: Hello, {name}!")
    time.sleep(0.5)
    print(f"Process {os.getpid()}: Goodbye, {name}!")

if __name__ == "__main__":
    print(f"Main process: {os.getpid()}")
    processes = []
    names = ["Alice", "Bob", "Charlie"]

    for name in names:
        p = multiprocess.Process(target=worker_function, args=(name,))
        processes.append(p)
        p.start()

    for p in processes:
        p.join()

    print("Main process: All workers finished.")

view raw JSON →