{"id":7435,"library":"multiprocessing-logging","title":"Multiprocessing Logging","description":"Multiprocessing-logging is a Python library that provides a handler to centralize logging from child processes created by the `multiprocessing` module to the main process. This prevents log messages from becoming garbled when multiple processes attempt to write to the same stream or file concurrently. It currently supports Python 3.9+ and is primarily tested on Linux.","status":"active","version":"0.3.4","language":"en","source_language":"en","source_url":"https://github.com/jruere/multiprocessing-logging","tags":["logging","multiprocessing","fork","posix"],"install":[{"cmd":"pip install multiprocessing-logging","lang":"bash","label":"Install stable version"}],"dependencies":[],"imports":[{"note":"Main function to enable multiprocessing-safe logging.","symbol":"install_mp_handler","correct":"from multiprocessing_logging import install_mp_handler"}],"quickstart":{"code":"import logging\nimport multiprocessing\nimport time\nfrom multiprocessing_logging import install_mp_handler\nimport os\n\ndef worker_function(name):\n    logger = logging.getLogger()\n    logger.info(f\"Worker {name} starting (PID: {os.getpid()})\")\n    time.sleep(0.5) # Simulate work\n    logger.info(f\"Worker {name} finishing (PID: {os.getpid()})\")\n\nif __name__ == \"__main__\":\n    # Set up basic logging for the main process first\n    logging.basicConfig(\n        level=os.environ.get('LOG_LEVEL', 'INFO'),\n        format='%(asctime)s - %(processName)s - %(levelname)s - %(message)s'\n    )\n    logger = logging.getLogger()\n    logger.info(\"Main process starting.\")\n\n    # IMPORTANT: install_mp_handler BEFORE creating any worker processes or Pool\n    # This library only works with the 'fork' start method.\n    # On some systems (e.g., macOS Python 3.8+, Python 3.14+ on POSIX), \n    # 'fork' is not the default or is considered unsafe.\n    # You might need to explicitly set 'fork' if your system supports it:\n    # multiprocessing.set_start_method('fork', force=True)\n    \n    try:\n        install_mp_handler(logger=logger)\n        logger.info(\"Multiprocessing logging handler installed.\")\n\n        processes = []\n        for i in range(3):\n            p = multiprocessing.Process(target=worker_function, args=(f\"Task-{i}\",))\n            processes.append(p)\n            p.start()\n\n        for p in processes:\n            p.join()\n\n        logger.info(\"All worker processes completed.\")\n    except AssertionError as e:\n        logger.error(f\"Failed to install multiprocessing handler: {e}. Check multiprocessing start method.\")\n    except Exception as e:\n        logger.error(f\"An unexpected error occurred: {e}\")\n\n    logger.info(\"Main process finishing.\")","lang":"python","description":"This quickstart demonstrates how to set up `multiprocessing-logging` with Python's standard `logging` module. Configure your root logger with `logging.basicConfig` first, then call `install_mp_handler()` before any `multiprocessing.Process` or `multiprocessing.Pool` instances are created. Note the critical warning about the 'fork' start method requirement."},"warnings":[{"fix":"On compatible POSIX systems, explicitly set the start method to 'fork' using `multiprocessing.set_start_method('fork', force=True)` at the very beginning of your main script, before any processes are created. Ensure your environment supports 'fork' safely.","message":"This library *only* works with the `fork` start method for `multiprocessing` and will raise an `AssertionError` otherwise. Python 3.8+ on macOS, and Python 3.14+ on POSIX systems, changed the default `start_method` to `spawn` or `forkserver`, which are incompatible. It explicitly does not work on Windows.","severity":"breaking","affected_versions":"All versions"},{"fix":"As a palliative, avoid continuously creating new processes. Instead, create a `multiprocessing.Pool` once and reuse it for your tasks.","message":"The reliance on the `fork` start method is inherently unsafe when the parent process is also multithreaded. This can lead to a low probability of the application hanging when creating new processes.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Ensure `multiprocessing_logging.install_mp_handler()` is placed early in your `if __name__ == '__main__':` block, after your main process logging is configured but before any child processes are spawned.","message":"The `install_mp_handler()` function must be called *after* `logging.basicConfig()` (or other logger configuration) but *before* any `multiprocessing.Process` or `multiprocessing.Pool` instances are created or used. Failing to do so can result in logs not being properly captured from child processes or errors during handler installation.","severity":"gotcha","affected_versions":"All versions"}],"env_vars":null,"last_verified":"2026-04-16T00:00:00.000Z","next_check":"2026-07-15T00:00:00.000Z","problems":[{"fix":"Ensure `multiprocessing_logging.install_mp_handler()` is correctly called in the main process before any child processes are spawned, and that the root logger (or the logger passed to the handler) has appropriate handlers configured.","cause":"Multiple processes are attempting to write to the same log handler (e.g., a file or console) without proper synchronization, or child processes are not configured to send logs to the main process.","error":"Log messages from child processes are garbled, interleaved, or missing."},{"fix":"If your environment supports 'fork' and you understand its implications, explicitly set the start method at the very beginning of your script: `import multiprocessing; multiprocessing.set_start_method('fork', force=True)`.","cause":"Your operating system (e.g., Windows, macOS from Python 3.8+, or Linux from Python 3.14+) uses a default multiprocessing start method ('spawn' or 'forkserver') that is incompatible with `multiprocessing-logging`.","error":"AssertionError: This module only works with the 'fork' start method."},{"fix":"Avoid continuously creating new processes. Instead, use a `multiprocessing.Pool` and initialize it once, then reuse it for your tasks. This limits the number of `fork` operations.","cause":"The `multiprocessing-logging` library relies on the `fork` start method, which can be unsafe when the parent process is multithreaded, leading to resource contention or deadlocks.","error":"Application hangs or deadlocks when creating or joining multiprocessing processes, especially with threads."}]}