{"id":615,"library":"billiard","title":"Billiard: Improved Python Multiprocessing","description":"billiard is a robust fork of the Python 2.7 `multiprocessing` package, actively maintained by the Celery project. It provides numerous improvements and bugfixes over the standard library's `multiprocessing` module, offering enhanced process-based parallelism for Python applications. It aims to address specific challenges and performance bottlenecks, particularly in distributed task queue systems like Celery, where it serves as a core dependency. The library is under active development with a consistent release cadence.","status":"active","version":"4.2.4","language":"python","source_language":"en","source_url":"https://github.com/celery/billiard","tags":["multiprocessing","concurrency","celery","process-pool","parallel-processing"],"install":[{"cmd":"pip install billiard","lang":"bash","label":"Install latest stable version"}],"dependencies":[{"reason":"Required for `max_memory_per_child` functionality, especially on Windows.","package":"psutil","optional":true}],"imports":[{"symbol":"Process","correct":"from billiard import Process"},{"symbol":"Queue","correct":"from billiard import Queue"},{"note":"While `billiard` is a fork of `multiprocessing`, importing directly from `multiprocessing` will use the standard library version, not `billiard`'s improved version. Explicitly import from `billiard` to use its features.","wrong":"from multiprocessing import Pool","symbol":"Pool","correct":"from billiard import Pool"}],"quickstart":{"code":"import os\nfrom billiard import Pool\n\ndef worker_function(x):\n    \"\"\"A simple function to be executed by pool workers.\"\"\"\n    print(f\"Worker PID {os.getpid()}: Processing {x}\")\n    return x * x\n\nif __name__ == '__main__':\n    # It's crucial to use the if __name__ == '__main__': guard\n    # especially on Windows or when using 'spawn' start methods.\n    print(f\"Main process PID: {os.getpid()}\")\n    \n    with Pool(processes=3) as pool:\n        # Map the worker_function to a list of inputs\n        results = pool.map(worker_function, range(5))\n        print(f\"Results: {results}\")\n\n    # Alternatively, you can explicitly close and join the pool\n    # pool = Pool(processes=2)\n    # async_result = pool.apply_async(worker_function, (10,))\n    # print(f\"Async result: {async_result.get()}\")\n    # pool.close()\n    # pool.join()","lang":"python","description":"This quickstart demonstrates how to create and use a process pool with `billiard.Pool` to execute a function across multiple processes. The `if __name__ == '__main__':` guard is essential for proper functioning, especially on Windows and with certain start methods, to prevent infinite process spawning."},"warnings":[{"fix":"Upgrade your Python environment to 3.7 or higher.","message":"Python 3.7 or newer is now required. Older Python versions (e.g., Python 3.6 and earlier) are no longer supported since `billiard` v4.0.0.","severity":"breaking","affected_versions":">=4.0.0"},{"fix":"Ensure your code is 'spawn'-safe (i.e., doesn't rely on inherited state, and uses the `if __name__ == '__main__':` guard). If 'fork' is strictly necessary and you understand the risks, configure the start method explicitly (e.g., `billiard.set_start_method('fork')`).","message":"On macOS, `billiard` (like `multiprocessing` in Python 3.8+) defaults to the 'spawn' start method instead of 'fork' for safety. Relying on the 'fork' method, particularly in multi-threaded contexts, can lead to crashes.","severity":"breaking","affected_versions":">=4.3.0rc1 (future), effectively >=4.0.0 (due to underlying Python changes)"},{"fix":"Always wrap the code that creates `Process` or `Pool` objects and their associated logic within an `if __name__ == '__main__':` block.","message":"The `if __name__ == '__main__':` block is critical for any code using `billiard` processes. Without it, especially on Windows or when using 'spawn' or 'forkserver' start methods, child processes will re-import the main module and can lead to infinite recursion and process spawning.","severity":"gotcha","affected_versions":"All versions"},{"fix":"If parallel execution is needed within a Celery task, consider restructuring the workflow to use Celery's native group/chain/chord primitives or ensure that any internal parallelization is carefully managed to avoid conflicts with `billiard`'s process management.","message":"Avoid using the standard `multiprocessing` module within Celery tasks that are managed by `billiard` (Celery's default). This can lead to nested process creation, which may cause unexpected behavior and issues.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Review any custom signal handling logic and remove reliance on `SIGUSR2` for process termination if you are using `billiard`'s default signal sets.","message":"`SIGUSR2` was removed from `TERMSIGS_DEFAULT`, which defines signals that terminate processes. This change might affect custom signal handling logic.","severity":"deprecated","affected_versions":">=4.2.1"}],"env_vars":null,"last_verified":"2026-05-12T16:42:55.696Z","next_check":"2026-06-26T00:00:00.000Z","problems":[{"fix":"Install the 'billiard' library using pip: `pip install billiard` or `pip3 install billiard` if you have multiple Python versions.","cause":"The 'billiard' library is not installed in the current Python environment, or the environment is not correctly activated, preventing the Python interpreter from finding the package.","error":"ModuleNotFoundError: No module named 'billiard'"},{"fix":"Ensure that all objects passed to 'billiard' processes (e.g., as arguments to a `Process` target function or elements in a `Queue`) are picklable. This often involves defining functions at the top level of a module, making classes and their methods independently picklable, or restructuring the data to remove unpicklable components.","cause":"This error occurs when attempting to pass non-serializable objects (like _thread.lock objects, local functions, or complex stateful class instances) between processes. The 'billiard' library, like 'multiprocessing', uses pickling to transfer objects for certain process start methods (e.g., 'spawn' on Windows or for `Pool` arguments).","error":"TypeError: can't pickle _thread.lock objects"},{"fix":"Redesign the application to avoid nesting process creation. If a background task needs to perform concurrent operations, it should use threads instead of creating new processes. In a Celery context, this means ensuring your tasks do not themselves invoke `billiard.Process` or `multiprocessing.Process`.","cause":"A daemon process (a background process that exits when its parent does, commonly used by `billiard` pools and Celery workers) is attempting to create its own child processes. Python's `multiprocessing` and `billiard` modules prevent this to avoid creating orphaned processes.","error":"AssertionError: daemonic processes are not allowed to have children"},{"fix":"Inspect detailed worker logs for the specific exception or signal that led to the premature exit. Increase system memory or adjust resource limits if out-of-memory errors are indicated. Implement robust error handling within your tasks and configure Celery's `task_soft_time_limit` and `task_time_limit` settings appropriately.","cause":"A `billiard` worker process, frequently seen in Celery deployments, terminated unexpectedly. This can be caused by unhandled exceptions within a task, the worker exceeding its allocated memory (killed by the operating system's OOM killer), hitting a hard or soft time limit, or receiving a termination signal.","error":"billiard.exceptions.WorkerLostError: Worker exited prematurely:"}],"ecosystem":"pypi","meta_description":null,"install_score":100,"install_tag":"verified","quickstart_score":80,"quickstart_tag":"verified","pypi_latest":"4.2.4","install_checks":{"last_tested":"2026-05-12","tag":"verified","tag_description":"installs cleanly on critical runtimes, fast import, recently tested","results":[{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":" $EXIT -eq 0 ","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":null,"import_time_s":0.05,"mem_mb":1.4,"disk_size":"18.4M"},{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.05,"mem_mb":1.4,"disk_size":"18.4M"},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":" $EXIT -eq 0 ","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":1.5,"import_time_s":0.03,"mem_mb":1.7,"disk_size":"19M"},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.03,"mem_mb":1.7,"disk_size":"19M"},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":" $EXIT -eq 0 ","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":null,"import_time_s":0.07,"mem_mb":1.9,"disk_size":"20.5M"},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.08,"mem_mb":1.9,"disk_size":"20.5M"},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":" $EXIT -eq 0 ","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":1.6,"import_time_s":0.07,"mem_mb":1.9,"disk_size":"21M"},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.07,"mem_mb":1.9,"disk_size":"21M"},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":" $EXIT -eq 0 ","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":null,"import_time_s":0.07,"mem_mb":1.8,"disk_size":"12.3M"},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.07,"mem_mb":1.8,"disk_size":"12.3M"},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":" $EXIT -eq 0 ","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":1.5,"import_time_s":0.06,"mem_mb":1.8,"disk_size":"13M"},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.07,"mem_mb":1.8,"disk_size":"13M"},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":" $EXIT -eq 0 ","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":null,"import_time_s":0.07,"mem_mb":2,"disk_size":"12.0M"},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.07,"mem_mb":2,"disk_size":"11.9M"},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":" $EXIT -eq 0 ","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":1.5,"import_time_s":0.06,"mem_mb":1.9,"disk_size":"13M"},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.06,"mem_mb":1.9,"disk_size":"12M"},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":" $EXIT -eq 0 ","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":null,"import_time_s":0.05,"mem_mb":1.4,"disk_size":"17.9M"},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.05,"mem_mb":1.4,"disk_size":"17.9M"},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":" $EXIT -eq 0 ","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":1.8,"import_time_s":0.04,"mem_mb":1.7,"disk_size":"18M"},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.04,"mem_mb":1.7,"disk_size":"18M"}]},"quickstart_checks":{"last_tested":"2026-04-24","tag":"verified","tag_description":"quickstart runs on critical runtimes, recently tested","results":[{"runtime":"python:3.10-alpine","exit_code":0},{"runtime":"python:3.10-slim","exit_code":0},{"runtime":"python:3.11-alpine","exit_code":0},{"runtime":"python:3.11-slim","exit_code":0},{"runtime":"python:3.12-alpine","exit_code":0},{"runtime":"python:3.12-slim","exit_code":0},{"runtime":"python:3.13-alpine","exit_code":0},{"runtime":"python:3.13-slim","exit_code":0},{"runtime":"python:3.9-alpine","exit_code":0},{"runtime":"python:3.9-slim","exit_code":0}]}}