{"id":535,"library":"multiprocess","title":"Multiprocess","description":"Multiprocess is a Python library that serves as a friendly fork of the standard `multiprocessing` module, primarily providing enhanced and more robust serialization capabilities through the use of `dill`. It aims to be a drop-in replacement for `multiprocessing` in many scenarios, offering better handling of complex objects and functions. The library is actively maintained, with regular releases (several per year) addressing updates and Python version compatibility. The current version is 0.70.19.","status":"active","version":"0.70.19","language":"python","source_language":"en","source_url":"https://github.com/uqfoundation/multiprocess","tags":["multiprocessing","serialization","concurrency","parallel-processing","dill"],"install":[{"cmd":"pip install multiprocess","lang":"bash","label":"Install latest version"}],"dependencies":[{"reason":"Provides enhanced serialization for complex Python objects; it is a core dependency and its minimum version requirement increases with multiprocess updates.","package":"dill","optional":false}],"imports":[{"note":"While often a drop-in replacement, explicit import from `multiprocess` is required to leverage its enhanced features.","wrong":"from multiprocessing import Process","symbol":"Process","correct":"from multiprocess import Process"},{"note":"To utilize the `multiprocess` Pool with its serialization advantages, import directly from `multiprocess`.","wrong":"from multiprocessing import Pool","symbol":"Pool","correct":"from multiprocess import Pool"},{"symbol":"Queue","correct":"from multiprocess import Queue"}],"quickstart":{"code":"import multiprocess\nimport os\nimport time\n\ndef worker_function(name):\n    \"\"\"A function to be run in a separate process.\"\"\"\n    print(f\"Process {os.getpid()}: Hello, {name}!\")\n    time.sleep(0.5)\n    print(f\"Process {os.getpid()}: Goodbye, {name}!\")\n\nif __name__ == \"__main__\":\n    print(f\"Main process: {os.getpid()}\")\n    processes = []\n    names = [\"Alice\", \"Bob\", \"Charlie\"]\n\n    for name in names:\n        p = multiprocess.Process(target=worker_function, args=(name,))\n        processes.append(p)\n        p.start()\n\n    for p in processes:\n        p.join()\n\n    print(\"Main process: All workers finished.\")","lang":"python","description":"This quickstart demonstrates how to create and manage processes using `multiprocess.Process`. Each worker function runs in its own process, executing the `worker_function` with a given name. The `if __name__ == \"__main__\":` block is essential for proper process spawning on certain operating systems (especially Windows) and to prevent recursive imports."},"warnings":[{"fix":"Ensure your environment uses Python 3.9 or newer. Upgrade your Python interpreter if necessary.","message":"Python 3.7 and older versions are no longer formally supported. Version 0.70.19 requires Python >=3.9. Previous versions had different minimum requirements (e.g., 0.70.18 required >=3.8, 0.70.16 dropped 3.7).","severity":"breaking","affected_versions":"0.70.16 and later"},{"fix":"Upgrade your `dill` package to the latest version (`pip install --upgrade dill`) or at least to the version specified by your `multiprocess` installation.","message":"The minimum required version for the `dill` dependency has steadily increased across `multiprocess` releases. Version 0.70.19 requires `dill >=0.4.1`. Older `multiprocess` versions may have different `dill` requirements.","severity":"breaking","affected_versions":"All versions"},{"fix":"Always wrap your process-spawning logic within `if __name__ == '__main__':`.","message":"When using `multiprocess` (or standard `multiprocessing`), the code that spawns child processes must be protected inside an `if __name__ == '__main__':` block. This is crucial for Windows compatibility and to prevent infinite recursion when child processes import the main script.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Include `import multiprocess.forking` at the very beginning of your main script when creating frozen executables.","message":"For 'frozen executables' (e.g., created with PyInstaller, cx_Freeze), `multiprocess` requires explicit early import of `multiprocess.forking` to set up proper process spawning mechanisms, otherwise processes may fail to start or behave unexpectedly.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Thoroughly test your application after switching from `multiprocessing` to `multiprocess`, especially if you pass complex or custom objects between processes. Review `dill`'s documentation for its serialization specifics.","message":"While `multiprocess` aims to be a 'drop-in replacement' and offers 'better serialization' compared to the standard `multiprocessing` module, users migrating or relying on very specific `pickle` behaviors might encounter subtle differences. `multiprocess` uses `dill` which has broader serialization capabilities, but this can also lead to different behavior for edge cases or non-standard objects.","severity":"gotcha","affected_versions":"All versions"}],"env_vars":null,"last_verified":"2026-05-12T14:45:10.316Z","next_check":"2026-06-26T00:00:00.000Z","problems":[{"fix":"Use the process-safe synchronization primitives provided by the `multiprocess` library, such as `multiprocess.Lock()` or `multiprocess.Queue()`, instead of their `threading` or standard `queue` module counterparts. Ensure any shared state or objects are designed to be picklable or are managed via `multiprocess.Manager`. For instances where you are creating threads within a child process, create the `_thread.lock` object *inside* the child process function, not in the parent.","cause":"This error occurs because low-level threading synchronization primitives, like `_thread.lock` objects, cannot be serialized (pickled) and transferred between separate processes. This often happens when passing `threading.Lock` or `queue.Queue` (from the `queue` module, which is thread-safe) directly to `multiprocess` workers, rather than using the process-safe equivalents from the `multiprocess` module itself.","error":"TypeError: can't pickle _thread.lock objects"},{"fix":"Wrap all code that creates new processes within an `if __name__ == '__main__':` block. If creating a frozen executable, also include `multiprocess.freeze_support()` at the very beginning of the main script, although this is often not strictly necessary for unfrozen scripts.","cause":"This error is common on Windows and macOS (which default to the 'spawn' start method) when code that creates child processes (like `multiprocess.Process` or `multiprocess.Pool`) is executed directly at the top level of a script, rather than being protected within an `if __name__ == '__main__':` block. Child processes must import the main module to run, and if the process creation code is at the top level, it leads to an infinite recursion of process spawning.","error":"RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase. This probably means that you are not using fork to start your child processes and you have forgotten to use the proper idiom in the main module: if __name__ == '__main__': freeze_support()"},{"fix":"Ensure that functions or methods intended for multiprocessing are defined at the top level of a module, or within a class whose instances are properly structured for serialization. If a lambda function is causing the issue, refactor it into a regular, globally defined function. For class methods, ensure the class itself is picklable and the method does not rely on non-picklable internal state that isn't handled by `__getstate__` and `__setstate__` methods.","cause":"Although `multiprocess` uses `dill` for enhanced serialization, which is generally more capable than `pickle` at handling local objects, closures, and lambda functions, this error can still occur in specific complex scenarios. It typically means that `dill` was unable to serialize an object (often a function or a method) that was defined locally within another function or is not accessible in the global scope of the module when the child process tries to unpickle it.","error":"AttributeError: Can't pickle local object '...' (e.g., 'lambda', 'MyClass.<locals>.my_function')"},{"fix":"Install the `multiprocess` library using pip: `pip install multiprocess`. Verify the installation by running `python -c \"import multiprocess; print(multiprocess.__version__)\"` in your terminal. If using a virtual environment, ensure it's activated before installation.","cause":"This error indicates that the `multiprocess` library is either not installed in the Python environment being used or Python cannot find it in its `sys.path`.","error":"ModuleNotFoundError: No module named 'multiprocess'"}],"ecosystem":"pypi","meta_description":null,"install_score":100,"install_tag":"verified","quickstart_score":80,"quickstart_tag":"verified","pypi_latest":null,"install_checks":{"last_tested":"2026-05-12","tag":"verified","tag_description":"installs cleanly on critical runtimes, fast import, recently tested","results":[{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.24,"mem_mb":7.1,"disk_size":"19.7M"},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.18,"mem_mb":7.2,"disk_size":"20M"},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.37,"mem_mb":7.6,"disk_size":"22.3M"},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.28,"mem_mb":7.7,"disk_size":"23M"},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.27,"mem_mb":7.7,"disk_size":"14.1M"},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.28,"mem_mb":7.8,"disk_size":"15M"},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.26,"mem_mb":8.2,"disk_size":"13.8M"},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.35,"mem_mb":8.3,"disk_size":"14M"},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.21,"mem_mb":6.9,"disk_size":"19.2M"},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.22,"mem_mb":7,"disk_size":"20M"}]},"quickstart_checks":{"last_tested":"2026-04-23","tag":"verified","tag_description":"quickstart runs on critical runtimes, recently tested","results":[{"runtime":"python:3.10-alpine","exit_code":0},{"runtime":"python:3.10-slim","exit_code":0},{"runtime":"python:3.11-alpine","exit_code":0},{"runtime":"python:3.11-slim","exit_code":0},{"runtime":"python:3.12-alpine","exit_code":0},{"runtime":"python:3.12-slim","exit_code":0},{"runtime":"python:3.13-alpine","exit_code":0},{"runtime":"python:3.13-slim","exit_code":0},{"runtime":"python:3.9-alpine","exit_code":0},{"runtime":"python:3.9-slim","exit_code":0}]}}