Billiard: Improved Python Multiprocessing

raw JSON →
4.2.4 verified Tue May 12 auth: no python install: verified quickstart: verified

billiard is a robust fork of the Python 2.7 `multiprocessing` package, actively maintained by the Celery project. It provides numerous improvements and bugfixes over the standard library's `multiprocessing` module, offering enhanced process-based parallelism for Python applications. It aims to address specific challenges and performance bottlenecks, particularly in distributed task queue systems like Celery, where it serves as a core dependency. The library is under active development with a consistent release cadence.

pip install billiard

Common errors

error ModuleNotFoundError: No module named 'billiard' ↓
cause The 'billiard' library is not installed in the current Python environment, or the environment is not correctly activated, preventing the Python interpreter from finding the package.
fix
Install the 'billiard' library using pip: pip install billiard or pip3 install billiard if you have multiple Python versions.
error TypeError: can't pickle _thread.lock objects ↓
cause This error occurs when attempting to pass non-serializable objects (like _thread.lock objects, local functions, or complex stateful class instances) between processes. The 'billiard' library, like 'multiprocessing', uses pickling to transfer objects for certain process start methods (e.g., 'spawn' on Windows or for `Pool` arguments).
fix
Ensure that all objects passed to 'billiard' processes (e.g., as arguments to a Process target function or elements in a Queue) are picklable. This often involves defining functions at the top level of a module, making classes and their methods independently picklable, or restructuring the data to remove unpicklable components.
error AssertionError: daemonic processes are not allowed to have children ↓
cause A daemon process (a background process that exits when its parent does, commonly used by `billiard` pools and Celery workers) is attempting to create its own child processes. Python's `multiprocessing` and `billiard` modules prevent this to avoid creating orphaned processes.
fix
Redesign the application to avoid nesting process creation. If a background task needs to perform concurrent operations, it should use threads instead of creating new processes. In a Celery context, this means ensuring your tasks do not themselves invoke billiard.Process or multiprocessing.Process.
error billiard.exceptions.WorkerLostError: Worker exited prematurely: ↓
cause A `billiard` worker process, frequently seen in Celery deployments, terminated unexpectedly. This can be caused by unhandled exceptions within a task, the worker exceeding its allocated memory (killed by the operating system's OOM killer), hitting a hard or soft time limit, or receiving a termination signal.
fix
Inspect detailed worker logs for the specific exception or signal that led to the premature exit. Increase system memory or adjust resource limits if out-of-memory errors are indicated. Implement robust error handling within your tasks and configure Celery's task_soft_time_limit and task_time_limit settings appropriately.

Warnings

breaking Python 3.7 or newer is now required. Older Python versions (e.g., Python 3.6 and earlier) are no longer supported since `billiard` v4.0.0. ↓
fix Upgrade your Python environment to 3.7 or higher.
breaking On macOS, `billiard` (like `multiprocessing` in Python 3.8+) defaults to the 'spawn' start method instead of 'fork' for safety. Relying on the 'fork' method, particularly in multi-threaded contexts, can lead to crashes. ↓
fix Ensure your code is 'spawn'-safe (i.e., doesn't rely on inherited state, and uses the `if __name__ == '__main__':` guard). If 'fork' is strictly necessary and you understand the risks, configure the start method explicitly (e.g., `billiard.set_start_method('fork')`).
gotcha The `if __name__ == '__main__':` block is critical for any code using `billiard` processes. Without it, especially on Windows or when using 'spawn' or 'forkserver' start methods, child processes will re-import the main module and can lead to infinite recursion and process spawning. ↓
fix Always wrap the code that creates `Process` or `Pool` objects and their associated logic within an `if __name__ == '__main__':` block.
gotcha Avoid using the standard `multiprocessing` module within Celery tasks that are managed by `billiard` (Celery's default). This can lead to nested process creation, which may cause unexpected behavior and issues. ↓
fix If parallel execution is needed within a Celery task, consider restructuring the workflow to use Celery's native group/chain/chord primitives or ensure that any internal parallelization is carefully managed to avoid conflicts with `billiard`'s process management.
deprecated `SIGUSR2` was removed from `TERMSIGS_DEFAULT`, which defines signals that terminate processes. This change might affect custom signal handling logic. ↓
fix Review any custom signal handling logic and remove reliance on `SIGUSR2` for process termination if you are using `billiard`'s default signal sets.

Install compatibility verified last tested: 2026-05-12

python os / libc status wheel install import disk
3.10 alpine (musl) wheel - 0.05s 18.4M
3.10 alpine (musl) - - 0.05s 18.4M
3.10 slim (glibc) wheel 1.5s 0.03s 19M
3.10 slim (glibc) - - 0.03s 19M
3.11 alpine (musl) wheel - 0.07s 20.5M
3.11 alpine (musl) - - 0.08s 20.5M
3.11 slim (glibc) wheel 1.6s 0.07s 21M
3.11 slim (glibc) - - 0.07s 21M
3.12 alpine (musl) wheel - 0.07s 12.3M
3.12 alpine (musl) - - 0.07s 12.3M
3.12 slim (glibc) wheel 1.5s 0.06s 13M
3.12 slim (glibc) - - 0.07s 13M
3.13 alpine (musl) wheel - 0.07s 12.0M
3.13 alpine (musl) - - 0.07s 11.9M
3.13 slim (glibc) wheel 1.5s 0.06s 13M
3.13 slim (glibc) - - 0.06s 12M
3.9 alpine (musl) wheel - 0.05s 17.9M
3.9 alpine (musl) - - 0.05s 17.9M
3.9 slim (glibc) wheel 1.8s 0.04s 18M
3.9 slim (glibc) - - 0.04s 18M

Imports

Quickstart verified last tested: 2026-04-24

This quickstart demonstrates how to create and use a process pool with `billiard.Pool` to execute a function across multiple processes. The `if __name__ == '__main__':` guard is essential for proper functioning, especially on Windows and with certain start methods, to prevent infinite process spawning.

import os
from billiard import Pool

def worker_function(x):
    """A simple function to be executed by pool workers."""
    print(f"Worker PID {os.getpid()}: Processing {x}")
    return x * x

if __name__ == '__main__':
    # It's crucial to use the if __name__ == '__main__': guard
    # especially on Windows or when using 'spawn' start methods.
    print(f"Main process PID: {os.getpid()}")
    
    with Pool(processes=3) as pool:
        # Map the worker_function to a list of inputs
        results = pool.map(worker_function, range(5))
        print(f"Results: {results}")

    # Alternatively, you can explicitly close and join the pool
    # pool = Pool(processes=2)
    # async_result = pool.apply_async(worker_function, (10,))
    # print(f"Async result: {async_result.get()}")
    # pool.close()
    # pool.join()
last verified Tue May 12
next check Fri Jun 26