{"id":4809,"library":"tqdm-multiprocess","title":"tqdm-multiprocess","description":"tqdm-multiprocess is a Python library that facilitates easy integration of `tqdm` progress bars and logging redirection into multiprocessing workflows. It allows multiple worker processes to display their individual progress bars and log messages cleanly through the main process, along with a global progress bar for aggregate monitoring. The current version is 0.0.11, and its release cadence appears to be feature-driven rather than time-boxed.","status":"active","version":"0.0.11","language":"en","source_language":"en","source_url":"https://github.com/EleutherAI/tqdm-multiprocess","tags":["tqdm","multiprocessing","progress-bar","logging","parallel-processing"],"install":[{"cmd":"pip install tqdm-multiprocess","lang":"bash","label":"Install stable version"}],"dependencies":[{"reason":"Core functionality for progress bars.","package":"tqdm"},{"reason":"Used for cross-platform terminal coloring, likely for logging output.","package":"colorama"}],"imports":[{"symbol":"TqdmMultiProcessPool","correct":"from tqdm_multiprocess import TqdmMultiProcessPool"}],"quickstart":{"code":"import time\nimport logging\nfrom tqdm_multiprocess import TqdmMultiProcessPool\nfrom tqdm import tqdm\n\nlogging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')\n\ndef worker_function(task_id, sleep_duration, tqdm_func, global_tqdm):\n    # Initialize a worker-specific tqdm bar\n    worker_bar = tqdm_func(total=10, desc=f'Task {task_id}')\n    logging.info(f'Task {task_id}: Starting...')\n    for i in range(10):\n        time.sleep(sleep_duration) # Simulate work\n        worker_bar.update(1)\n        global_tqdm.update(1) # Update the global progress bar as well\n    logging.info(f'Task {task_id}: Finished.')\n    worker_bar.close()\n    return f'Task {task_id} completed'\n\nif __name__ == '__main__':\n    # Number of processes to use\n    PROCESS_COUNT = 2\n    # Total number of small steps across all tasks for the global bar\n    TOTAL_GLOBAL_STEPS = 2 * 10 # 2 tasks * 10 steps each\n\n    # Initialize a global tqdm bar (optional, can be None)\n    global_pbar = tqdm(total=TOTAL_GLOBAL_STEPS, desc='Global Progress', position=0)\n\n    # Create a list of tasks: (function_to_run, (args_tuple))\n    tasks = [\n        (worker_function, (1, 0.2)), # task_id 1, sleep 0.2s\n        (worker_function, (2, 0.3))  # task_id 2, sleep 0.3s\n    ]\n\n    # Create the TqdmMultiProcessPool\n    with TqdmMultiProcessPool(PROCESS_COUNT) as pool:\n        # Map tasks to the pool\n        results = pool.map(\n            process_count=PROCESS_COUNT,\n            global_tqdm=global_pbar,\n            task_list=tasks,\n            error_callback=lambda x: logging.error(f'Error: {x}'),\n            done_callback=lambda x: logging.info(f'Done: {x}')\n        )\n\n    global_pbar.close()\n    logging.info(f'All tasks completed. Results: {results}')","lang":"python","description":"This quickstart demonstrates how to use `TqdmMultiProcessPool` to run multiple tasks in parallel, each with its own `tqdm` progress bar, while also updating a global progress bar. It also showcases how logging from worker processes is redirected to the main process. Note the specific arguments `tqdm_func` and `global_tqdm` that must be passed to worker functions."},"warnings":[{"fix":"Always initialize worker-specific `tqdm` instances as `worker_bar = tqdm_func(total=total_steps, desc='...')` and call `worker_bar.update(1)` within your loop.","message":"Worker `tqdm` instances (those created with `tqdm_func`) do not support iterators. You must explicitly initialize them with `total=...` and update manually using `.update()`.","severity":"gotcha","affected_versions":"All versions (0.0.11)"},{"fix":"Update your `tqdm` progress bars less frequently (e.g., every few iterations) if performance issues are observed. The library author indicates a future attempt to implement a lock-free ringbuffer to improve this.","message":"Due to performance limitations of Python's default `multiprocessing.Queue`, frequent updates to global or worker `tqdm` objects can 'flood' the main process, leading to slow or non-responsive progress bars.","severity":"gotcha","affected_versions":"All versions (0.0.11)"},{"fix":"Ensure your worker function signature is `def worker_func(..., tqdm_func, global_tqdm):` and use `tqdm_func` to create any `tqdm` bars within the worker.","message":"Functions passed to `TqdmMultiProcessPool` must accept `tqdm_func` and `global_tqdm` as their last two arguments, even if `global_tqdm` is set to `None` when creating the pool.","severity":"gotcha","affected_versions":"All versions (0.0.11)"}],"env_vars":null,"last_verified":"2026-04-12T00:00:00.000Z","next_check":"2026-07-11T00:00:00.000Z"}