Loky - Robust Process Pool Executor

Warnings

• breaking Loky's `set_start_method` is incompatible with `multiprocessing.set_start_method`. Attempting to use the standard library's function will not configure Loky's process startup correctly and can lead to unexpected behavior or errors.
  Fix: Always import and use `loky.set_start_method()` when configuring the process start method for Loky executors.
• gotcha When using `loky.get_reusable_executor()` on Windows, worker processes are kept alive for reuse, which can prevent cleanup operations (e.g., `os.chdir()` or deleting temporary directories) if they were used within the worker context. Even `loky.ProcessPoolExecutor()` may leave one process active after explicit shutdown.
  Fix: For scenarios requiring strict process termination and resource cleanup, consider if `get_reusable_executor()`'s persistence model is suitable. If not, explicitly manage process lifecycle and ensure all resources are released or cleaned up outside the main process before attempting directory changes or deletions.
• gotcha While `loky` transparently integrates `cloudpickle` to serialize non-picklable objects, this serialization can introduce performance overhead compared to Python's standard `pickle` module, especially for very large objects or high-frequency task submission.
  Fix: For performance-critical applications, profile the serialization overhead. If it's a bottleneck, optimize objects to be standard-picklable or explore custom reducers for specific data types to minimize `cloudpickle`'s impact. Consider setting `LOKY_PICKLER=pickle` if `cloudpickle`'s broader serialization capabilities are not needed.
• gotcha Loky on POSIX systems (e.g., Linux, macOS) uses `fork+exec` for all processes to ensure consistent and robust spawn behavior, which is safer when interacting with third-party libraries (e.g., OpenMP, macOS Accelerate) compared to `multiprocessing.Pool`'s default `fork` (or pre-Python 3.8 macOS `fork`). This difference might subtly alter behavior if your code relies on `fork` without `exec` semantics.
  Fix: Be aware of the `fork+exec` behavior. If you encounter issues with shared memory or inherited resources, ensure all necessary state is explicitly passed to worker processes rather than relying on implicit inheritance. If working with very old Python versions or specific `multiprocessing.Pool` expectations, consider how this difference might affect your application.

Install

• pip install loky Install latest version

Imports

• get_reusable_executor

  from loky import get_reusable_executor

  This is the recommended entry point for most common use cases, providing a managed, reusable process pool.
• ProcessPoolExecutor

  from loky import ProcessPoolExecutor

  This provides a direct, robust replacement for `concurrent.futures.ProcessPoolExecutor` with enhanced error handling.
• set_start_method

  from loky import set_start_method

  Loky has its own `set_start_method` function which *must* be used instead of `multiprocessing.set_start_method` to ensure proper behavior and compatibility with Loky's internal process management.

Quickstart

This quickstart demonstrates the two primary ways to use Loky: `get_reusable_executor()` for a managed and persistent pool, and `ProcessPoolExecutor()` for a direct `concurrent.futures`-like experience. The `if __name__ == "__main__":` block is included for robust multiprocessing execution across different operating systems.

import os
from loky import get_reusable_executor

def worker_function(x):
    # Simulate some work
    pid = os.getpid()
    return f"Processed {x} by PID {pid}"

if __name__ == "__main__":
    # Using get_reusable_executor for managed process pool
    with get_reusable_executor(max_workers=2) as executor:
        results = list(executor.map(worker_function, range(5)))
    print(results)

    # Direct ProcessPoolExecutor usage (similar to concurrent.futures)
    from loky import ProcessPoolExecutor
    with ProcessPoolExecutor(max_workers=2) as executor:
        results_direct = list(executor.map(worker_function, range(5, 10)))
    print(results_direct)