Multiprocessing Logging

Common errors

• Log messages from child processes are garbled, interleaved, or missing.

  cause Multiple processes are attempting to write to the same log handler (e.g., a file or console) without proper synchronization, or child processes are not configured to send logs to the main process.
  fix Ensure `multiprocessing_logging.install_mp_handler()` is correctly called in the main process before any child processes are spawned, and that the root logger (or the logger passed to the handler) has appropriate handlers configured.

• AssertionError: This module only works with the 'fork' start method.

  cause Your operating system (e.g., Windows, macOS from Python 3.8+, or Linux from Python 3.14+) uses a default multiprocessing start method ('spawn' or 'forkserver') that is incompatible with `multiprocessing-logging`.
  fix If your environment supports 'fork' and you understand its implications, explicitly set the start method at the very beginning of your script: `import multiprocessing; multiprocessing.set_start_method('fork', force=True)`.

• Application hangs or deadlocks when creating or joining multiprocessing processes, especially with threads.

  cause The `multiprocessing-logging` library relies on the `fork` start method, which can be unsafe when the parent process is multithreaded, leading to resource contention or deadlocks.
  fix Avoid continuously creating new processes. Instead, use a `multiprocessing.Pool` and initialize it once, then reuse it for your tasks. This limits the number of `fork` operations.

Warnings

• breaking This library *only* works with the `fork` start method for `multiprocessing` and will raise an `AssertionError` otherwise. Python 3.8+ on macOS, and Python 3.14+ on POSIX systems, changed the default `start_method` to `spawn` or `forkserver`, which are incompatible. It explicitly does not work on Windows.
  Fix: On compatible POSIX systems, explicitly set the start method to 'fork' using `multiprocessing.set_start_method('fork', force=True)` at the very beginning of your main script, before any processes are created. Ensure your environment supports 'fork' safely.
• gotcha The reliance on the `fork` start method is inherently unsafe when the parent process is also multithreaded. This can lead to a low probability of the application hanging when creating new processes.
  Fix: As a palliative, avoid continuously creating new processes. Instead, create a `multiprocessing.Pool` once and reuse it for your tasks.
• gotcha The `install_mp_handler()` function must be called *after* `logging.basicConfig()` (or other logger configuration) but *before* any `multiprocessing.Process` or `multiprocessing.Pool` instances are created or used. Failing to do so can result in logs not being properly captured from child processes or errors during handler installation.
  Fix: Ensure `multiprocessing_logging.install_mp_handler()` is placed early in your `if __name__ == '__main__':` block, after your main process logging is configured but before any child processes are spawned.

Install

• pip install multiprocessing-logging Install stable version

Imports

• install_mp_handler

  from multiprocessing_logging import install_mp_handler

  Main function to enable multiprocessing-safe logging.

Quickstart

This quickstart demonstrates how to set up `multiprocessing-logging` with Python's standard `logging` module. Configure your root logger with `logging.basicConfig` first, then call `install_mp_handler()` before any `multiprocessing.Process` or `multiprocessing.Pool` instances are created. Note the critical warning about the 'fork' start method requirement.

import logging
import multiprocessing
import time
from multiprocessing_logging import install_mp_handler
import os

def worker_function(name):
    logger = logging.getLogger()
    logger.info(f"Worker {name} starting (PID: {os.getpid()})")
    time.sleep(0.5) # Simulate work
    logger.info(f"Worker {name} finishing (PID: {os.getpid()})")

if __name__ == "__main__":
    # Set up basic logging for the main process first
    logging.basicConfig(
        level=os.environ.get('LOG_LEVEL', 'INFO'),
        format='%(asctime)s - %(processName)s - %(levelname)s - %(message)s'
    )
    logger = logging.getLogger()
    logger.info("Main process starting.")

    # IMPORTANT: install_mp_handler BEFORE creating any worker processes or Pool
    # This library only works with the 'fork' start method.
    # On some systems (e.g., macOS Python 3.8+, Python 3.14+ on POSIX), 
    # 'fork' is not the default or is considered unsafe.
    # You might need to explicitly set 'fork' if your system supports it:
    # multiprocessing.set_start_method('fork', force=True)
    
    try:
        install_mp_handler(logger=logger)
        logger.info("Multiprocessing logging handler installed.")

        processes = []
        for i in range(3):
            p = multiprocessing.Process(target=worker_function, args=(f"Task-{i}",))
            processes.append(p)
            p.start()

        for p in processes:
            p.join()

        logger.info("All worker processes completed.")
    except AssertionError as e:
        logger.error(f"Failed to install multiprocessing handler: {e}. Check multiprocessing start method.")
    except Exception as e:
        logger.error(f"An unexpected error occurred: {e}")

    logger.info("Main process finishing.")