tqdm-multiprocess

0.0.11 · active · verified Sun Apr 12

tqdm-multiprocess is a Python library that facilitates easy integration of `tqdm` progress bars and logging redirection into multiprocessing workflows. It allows multiple worker processes to display their individual progress bars and log messages cleanly through the main process, along with a global progress bar for aggregate monitoring. The current version is 0.0.11, and its release cadence appears to be feature-driven rather than time-boxed.

Warnings

gotcha Worker `tqdm` instances (those created with `tqdm_func`) do not support iterators. You must explicitly initialize them with `total=...` and update manually using `.update()`.
Fix: Always initialize worker-specific `tqdm` instances as `worker_bar = tqdm_func(total=total_steps, desc='...')` and call `worker_bar.update(1)` within your loop.
gotcha Due to performance limitations of Python's default `multiprocessing.Queue`, frequent updates to global or worker `tqdm` objects can 'flood' the main process, leading to slow or non-responsive progress bars.
Fix: Update your `tqdm` progress bars less frequently (e.g., every few iterations) if performance issues are observed. The library author indicates a future attempt to implement a lock-free ringbuffer to improve this.
gotcha Functions passed to `TqdmMultiProcessPool` must accept `tqdm_func` and `global_tqdm` as their last two arguments, even if `global_tqdm` is set to `None` when creating the pool.
Fix: Ensure your worker function signature is `def worker_func(..., tqdm_func, global_tqdm):` and use `tqdm_func` to create any `tqdm` bars within the worker.

Install

pip install tqdm-multiprocess Install stable version

Imports

TqdmMultiProcessPool

from tqdm_multiprocess import TqdmMultiProcessPool

Quickstart

This quickstart demonstrates how to use `TqdmMultiProcessPool` to run multiple tasks in parallel, each with its own `tqdm` progress bar, while also updating a global progress bar. It also showcases how logging from worker processes is redirected to the main process. Note the specific arguments `tqdm_func` and `global_tqdm` that must be passed to worker functions.

import time
import logging
from tqdm_multiprocess import TqdmMultiProcessPool
from tqdm import tqdm

logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')

def worker_function(task_id, sleep_duration, tqdm_func, global_tqdm):
    # Initialize a worker-specific tqdm bar
    worker_bar = tqdm_func(total=10, desc=f'Task {task_id}')
    logging.info(f'Task {task_id}: Starting...')
    for i in range(10):
        time.sleep(sleep_duration) # Simulate work
        worker_bar.update(1)
        global_tqdm.update(1) # Update the global progress bar as well
    logging.info(f'Task {task_id}: Finished.')
    worker_bar.close()
    return f'Task {task_id} completed'

if __name__ == '__main__':
    # Number of processes to use
    PROCESS_COUNT = 2
    # Total number of small steps across all tasks for the global bar
    TOTAL_GLOBAL_STEPS = 2 * 10 # 2 tasks * 10 steps each

    # Initialize a global tqdm bar (optional, can be None)
    global_pbar = tqdm(total=TOTAL_GLOBAL_STEPS, desc='Global Progress', position=0)

    # Create a list of tasks: (function_to_run, (args_tuple))
    tasks = [
        (worker_function, (1, 0.2)), # task_id 1, sleep 0.2s
        (worker_function, (2, 0.3))  # task_id 2, sleep 0.3s
    ]

    # Create the TqdmMultiProcessPool
    with TqdmMultiProcessPool(PROCESS_COUNT) as pool:
        # Map tasks to the pool
        results = pool.map(
            process_count=PROCESS_COUNT,
            global_tqdm=global_pbar,
            task_list=tasks,
            error_callback=lambda x: logging.error(f'Error: {x}'),
            done_callback=lambda x: logging.info(f'Done: {x}')
        )

    global_pbar.close()
    logging.info(f'All tasks completed. Results: {results}')

view raw JSON →