Billiard: Improved Python Multiprocessing

4.2.4 · active · verified Sat Mar 28

billiard is a robust fork of the Python 2.7 `multiprocessing` package, actively maintained by the Celery project. It provides numerous improvements and bugfixes over the standard library's `multiprocessing` module, offering enhanced process-based parallelism for Python applications. It aims to address specific challenges and performance bottlenecks, particularly in distributed task queue systems like Celery, where it serves as a core dependency. The library is under active development with a consistent release cadence.

Warnings

breaking Python 3.7 or newer is now required. Older Python versions (e.g., Python 3.6 and earlier) are no longer supported since `billiard` v4.0.0.
Fix: Upgrade your Python environment to 3.7 or higher.
breaking On macOS, `billiard` (like `multiprocessing` in Python 3.8+) defaults to the 'spawn' start method instead of 'fork' for safety. Relying on the 'fork' method, particularly in multi-threaded contexts, can lead to crashes.
Fix: Ensure your code is 'spawn'-safe (i.e., doesn't rely on inherited state, and uses the `if __name__ == '__main__':` guard). If 'fork' is strictly necessary and you understand the risks, configure the start method explicitly (e.g., `billiard.set_start_method('fork')`).
gotcha The `if __name__ == '__main__':` block is critical for any code using `billiard` processes. Without it, especially on Windows or when using 'spawn' or 'forkserver' start methods, child processes will re-import the main module and can lead to infinite recursion and process spawning.
Fix: Always wrap the code that creates `Process` or `Pool` objects and their associated logic within an `if __name__ == '__main__':` block.
gotcha Avoid using the standard `multiprocessing` module within Celery tasks that are managed by `billiard` (Celery's default). This can lead to nested process creation, which may cause unexpected behavior and issues.
Fix: If parallel execution is needed within a Celery task, consider restructuring the workflow to use Celery's native group/chain/chord primitives or ensure that any internal parallelization is carefully managed to avoid conflicts with `billiard`'s process management.
deprecated `SIGUSR2` was removed from `TERMSIGS_DEFAULT`, which defines signals that terminate processes. This change might affect custom signal handling logic.
Fix: Review any custom signal handling logic and remove reliance on `SIGUSR2` for process termination if you are using `billiard`'s default signal sets.

Install

pip install billiard Install latest stable version

Imports

Process
```
from billiard import Process
```
Queue
```
from billiard import Queue
```
Pool
```
from billiard import Pool
```
While `billiard` is a fork of `multiprocessing`, importing directly from `multiprocessing` will use the standard library version, not `billiard`'s improved version. Explicitly import from `billiard` to use its features.

Quickstart

This quickstart demonstrates how to create and use a process pool with `billiard.Pool` to execute a function across multiple processes. The `if __name__ == '__main__':` guard is essential for proper functioning, especially on Windows and with certain start methods, to prevent infinite process spawning.

import os
from billiard import Pool

def worker_function(x):
    """A simple function to be executed by pool workers."""
    print(f"Worker PID {os.getpid()}: Processing {x}")
    return x * x

if __name__ == '__main__':
    # It's crucial to use the if __name__ == '__main__': guard
    # especially on Windows or when using 'spawn' start methods.
    print(f"Main process PID: {os.getpid()}")
    
    with Pool(processes=3) as pool:
        # Map the worker_function to a list of inputs
        results = pool.map(worker_function, range(5))
        print(f"Results: {results}")

    # Alternatively, you can explicitly close and join the pool
    # pool = Pool(processes=2)
    # async_result = pool.apply_async(worker_function, (10,))
    # print(f"Async result: {async_result.get()}")
    # pool.close()
    # pool.join()

view raw JSON →