Pandarallel

Warnings

• gotcha Pandarallel can require up to twice the memory of standard Pandas operations. Ensure your system has sufficient RAM, especially for large datasets.
  Fix: Monitor memory usage. For data larger than available memory, consider alternatives like Dask or PySpark.
• gotcha On Windows, functions passed to `pandarallel` must be self-contained and should not depend on external resources (e.g., global variables, complex closures) due to Python's `multiprocessing` 'spawn' start method.
  Fix: Define functions at the top level of the module. For complex scenarios, consider using Windows Subsystem for Linux (WSL) or refactor functions to be entirely self-contained.
• gotcha Parallelization introduces overhead. For small datasets or very fast operations, `pandarallel` might not provide a speedup, or could even be slower than native Pandas.
  Fix: Benchmark performance with and without `pandarallel` for your specific use case to determine if parallelization is beneficial.
• gotcha Pandarallel scales best with the number of *physical* CPU cores, not necessarily logical cores (hyperthreading). Setting `nb_workers` higher than physical cores may not yield further performance gains.
  Fix: For optimal performance, set `nb_workers` to your system's number of physical CPU cores or allow `pandarallel` to determine it automatically.
• gotcha The `shm_size_mb` parameter in `pandarallel.initialize()` is deprecated and should no longer be used.
  Fix: Remove `shm_size_mb` from `pandarallel.initialize()` calls. Memory file system usage is now controlled by `use_memory_fs`.
• gotcha Pandarallel can sometimes get stuck without raising errors if all physical cores are heavily utilized by other background processes. The progress bar might stop updating.
  Fix: Monitor system CPU usage. Try reducing `nb_workers` to leave some cores free, or ensure your environment has sufficient idle CPU resources.
• gotcha Functions defined locally (e.g., inside another function) or using closures may lead to `AttributeError: Can't pickle local object` errors, a common issue with Python's multiprocessing.
  Fix: Ensure functions passed to `pandarallel` methods are defined at the top level of a module, not nested within other functions.

Install

• pip install pandarallel Install with pip

Imports

• pandarallel

  from pandarallel import pandarallel

Quickstart

This quickstart demonstrates how to initialize pandarallel and then use `parallel_apply` on a Pandas Series. It includes a simple, CPU-bound function to showcase the parallelization effect. The `nb_workers` is explicitly set to the CPU count for clarity, and a progress bar is enabled.

import pandas as pd
from pandarallel import pandarallel
import os

# Initialize pandarallel. It defaults to using all available CPU cores.
# progress_bar=True is often useful to visualize progress.
pandarallel.initialize(nb_workers=os.cpu_count(), progress_bar=True)

# Create a sample DataFrame with some data
data = {'col1': range(1_000_000), 'col2': [f'item_{i}' for i in range(1_000_000)]}
df = pd.DataFrame(data)

# Define a CPU-bound function to apply
def example_computation(x):
    # Simulate a computationally intensive task
    res = 0
    for i in range(50):
        res += (x * i) ** 0.5
    return res

# Apply the function in parallel using pandarallel's parallel_apply
# This replaces df['col1'].apply(example_computation)
print("Starting parallel computation...")
df['result'] = df['col1'].parallel_apply(example_computation)

print("Computation complete. First 5 rows of the DataFrame with results:")
print(df.head())