p_tqdm: Parallel Processing with Progress Bars

1.4.2 · active · verified Thu Apr 16

p_tqdm is a Python library that simplifies parallel processing by wrapping `pathos.multiprocessing` with `tqdm` progress bars. It provides a flexible parallel map functionality, supporting lambda functions, nested functions, and class methods, while displaying a clear progress bar with estimated time to completion. The library is actively maintained, with version 1.4.2 being the current release as of August 2024, and receives updates to ensure compatibility and address issues.

Common errors

```
TypeError: __init__() takes 1 positional argument but 2 were given
```
cause This error often arises from issues with `dill` (used by `pathos`) when trying to pickle certain objects, especially class methods or complex closures, for multiprocessing.

fix Ensure the function passed to `p_tqdm` is a top-level function. If using class methods, convert them to static or standalone functions, or ensure the class itself is picklable. Consider using `multiprocess.Pool` with `tqdm.tqdm` manually if `p_tqdm`'s `pathos` integration proves problematic for complex objects, or simplify the function being pickled.
```
ModuleNotFoundError: No module named 'collections.abc'
```
cause Older versions of `p-tqdm` (prior to v1.4.0) used deprecated imports from the `collections` module. This can cause `ModuleNotFoundError` or `DeprecationWarning` on Python 3.9+ environments. [GitHub release notes]

fix Upgrade `p-tqdm` to version 1.4.0 or higher: `pip install --upgrade p-tqdm`. This version includes a fix for `collections` compatibility with Python 3.9+.

Warnings

breaking p_tqdm versions prior to 1.3 dropped support for Python 2.x and versions prior to 3.5. Ensure you are using Python 3.5 or newer.
Fix: Upgrade your Python interpreter to 3.5+ or use p_tqdm v1.3 or higher. `pip install --upgrade p-tqdm`.
breaking The handling of non-list iterables changed significantly in v1.3.3. Previously, single items or `num_iters` might have been implicitly repeated; now, singleton items (and the `num_iters` kwarg) should no longer be passed directly. Use `functools.partial` for repeated arguments.
Fix: For single, repeated arguments, use `functools.partial`. For example, instead of `p_map(func, iterable, fixed_arg)`, use `p_map(partial(func, fixed_arg=fixed_arg), iterable)`.
gotcha Using `print()` statements inside functions processed by `p_tqdm` (or `tqdm`) can disrupt the progress bar's display, leading to messy output.
Fix: Use `tqdm.write('Your message')` instead of `print()` to ensure messages are printed without interfering with the progress bar.
gotcha If you are using generators or iterables without a known `len()`, the progress bar will not accurately display the total or estimated time remaining unless you explicitly provide the `total` argument.
Fix: Pass the `total` keyword argument to `p_map` (or other `p_tqdm` functions) with the expected number of iterations: `p_map(func, iterable, total=expected_count)`.

Install

pip install p-tqdm PyPI

Imports

p_map
```
from p_tqdm import p_map
```
p_imap
```
from p_tqdm import p_imap
```
p_umap
```
from p_tqdm import p_umap
```
p_uimap
```
from p_tqdm import p_uimap
```

Quickstart

This example demonstrates how to use `p_map` to apply a function in parallel to elements from multiple iterables, displaying a progress bar. The `my_function` simulates a task by sleeping briefly and then adding two numbers.

from p_tqdm import p_map
import time

def my_function(x, y):
    time.sleep(0.01) # Simulate work
    return x + y

l1 = [i for i in range(100)]
l2 = [i * 2 for i in range(100)]

results = p_map(my_function, l1, l2)
print(f"First 5 results: {results[:5]}")

view raw JSON →