{"id":4622,"library":"loky","title":"Loky - Robust Process Pool Executor","description":"Loky provides a robust, cross-platform, and cross-version implementation of Python's `concurrent.futures.ProcessPoolExecutor`. It enhances multiprocessing by offering reusable executors, transparent `cloudpickle` integration for complex object serialization, and deadlock-free process management, addressing common pitfalls in parallel Python computing. The library is actively maintained, with its current version being 3.5.6. It primarily follows a minor release cadence with bug fixes and improvements.","status":"active","version":"3.5.6","language":"en","source_language":"en","source_url":"https://github.com/joblib/loky","tags":["multiprocessing","concurrency","process pool","executor","joblib","cloudpickle"],"install":[{"cmd":"pip install loky","lang":"bash","label":"Install latest version"}],"dependencies":[{"reason":"Requires Python 3.9 or higher for the latest versions.","package":"python","optional":false},{"reason":"Optional dependency that enables serialization of a wider range of objects (e.g., lambda functions, interactively defined functions), especially those in the __main__ module, avoiding common pickling errors.","package":"cloudpickle","optional":true},{"reason":"Optional dependency used for early detection of memory leaks in worker processes.","package":"psutil","optional":true}],"imports":[{"note":"This is the recommended entry point for most common use cases, providing a managed, reusable process pool.","symbol":"get_reusable_executor","correct":"from loky import get_reusable_executor"},{"note":"This provides a direct, robust replacement for `concurrent.futures.ProcessPoolExecutor` with enhanced error handling.","symbol":"ProcessPoolExecutor","correct":"from loky import ProcessPoolExecutor"},{"note":"Loky has its own `set_start_method` function which *must* be used instead of `multiprocessing.set_start_method` to ensure proper behavior and compatibility with Loky's internal process management.","wrong":"import multiprocessing; multiprocessing.set_start_method('spawn')","symbol":"set_start_method","correct":"from loky import set_start_method"}],"quickstart":{"code":"import os\nfrom loky import get_reusable_executor\n\ndef worker_function(x):\n    # Simulate some work\n    pid = os.getpid()\n    return f\"Processed {x} by PID {pid}\"\n\nif __name__ == \"__main__\":\n    # Using get_reusable_executor for managed process pool\n    with get_reusable_executor(max_workers=2) as executor:\n        results = list(executor.map(worker_function, range(5)))\n    print(results)\n\n    # Direct ProcessPoolExecutor usage (similar to concurrent.futures)\n    from loky import ProcessPoolExecutor\n    with ProcessPoolExecutor(max_workers=2) as executor:\n        results_direct = list(executor.map(worker_function, range(5, 10)))\n    print(results_direct)","lang":"python","description":"This quickstart demonstrates the two primary ways to use Loky: `get_reusable_executor()` for a managed and persistent pool, and `ProcessPoolExecutor()` for a direct `concurrent.futures`-like experience. The `if __name__ == \"__main__\":` block is included for robust multiprocessing execution across different operating systems."},"warnings":[{"fix":"Always import and use `loky.set_start_method()` when configuring the process start method for Loky executors.","message":"Loky's `set_start_method` is incompatible with `multiprocessing.set_start_method`. Attempting to use the standard library's function will not configure Loky's process startup correctly and can lead to unexpected behavior or errors.","severity":"breaking","affected_versions":"All versions"},{"fix":"For scenarios requiring strict process termination and resource cleanup, consider if `get_reusable_executor()`'s persistence model is suitable. If not, explicitly manage process lifecycle and ensure all resources are released or cleaned up outside the main process before attempting directory changes or deletions.","message":"When using `loky.get_reusable_executor()` on Windows, worker processes are kept alive for reuse, which can prevent cleanup operations (e.g., `os.chdir()` or deleting temporary directories) if they were used within the worker context. Even `loky.ProcessPoolExecutor()` may leave one process active after explicit shutdown.","severity":"gotcha","affected_versions":"All versions, particularly on Windows"},{"fix":"For performance-critical applications, profile the serialization overhead. If it's a bottleneck, optimize objects to be standard-picklable or explore custom reducers for specific data types to minimize `cloudpickle`'s impact. Consider setting `LOKY_PICKLER=pickle` if `cloudpickle`'s broader serialization capabilities are not needed.","message":"While `loky` transparently integrates `cloudpickle` to serialize non-picklable objects, this serialization can introduce performance overhead compared to Python's standard `pickle` module, especially for very large objects or high-frequency task submission.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Be aware of the `fork+exec` behavior. If you encounter issues with shared memory or inherited resources, ensure all necessary state is explicitly passed to worker processes rather than relying on implicit inheritance. If working with very old Python versions or specific `multiprocessing.Pool` expectations, consider how this difference might affect your application.","message":"Loky on POSIX systems (e.g., Linux, macOS) uses `fork+exec` for all processes to ensure consistent and robust spawn behavior, which is safer when interacting with third-party libraries (e.g., OpenMP, macOS Accelerate) compared to `multiprocessing.Pool`'s default `fork` (or pre-Python 3.8 macOS `fork`). This difference might subtly alter behavior if your code relies on `fork` without `exec` semantics.","severity":"gotcha","affected_versions":"All versions on POSIX systems"}],"env_vars":null,"last_verified":"2026-04-12T00:00:00.000Z","next_check":"2026-07-11T00:00:00.000Z"}