Swifter

1.4.0 · active · verified Thu Apr 09

Swifter is a Python package designed to significantly speed up `apply` operations on pandas DataFrames and Series. It achieves this by automatically determining the fastest available method for applying a function, leveraging vectorized pandas operations, Dask for parallel processing, or multi-threading/multi-processing. Swifter integrates directly into pandas objects, offering a seamless way to optimize user-defined functions without extensive code changes. The current version is 1.4.0, released in July 2023, and the project maintains an active development and release cadence.

Warnings

gotcha Avoid using swifter with functions that modify external variables. Swifter performs 'sample applies' to optimize performance, which can lead to erroneous modifications of external variables in addition to the final apply operation.
Fix: Ensure that functions passed to `swifter.apply()` are pure functions without side effects on external state.
gotcha When `swifter` is called from a forked process, its progress bar may become confused. It is advisable to disable the progress bar in such scenarios.
Fix: Disable the progress bar using `swifter.disable_progress_bar()` or by setting the `SWIFTER_PROGRESS_BAR` environment variable to `False` when running in forked processes.
gotcha For compatibility with Modin DataFrames, `modin.pandas` must be imported *before* `swifter`, or `swifter.register_modin()` must be called explicitly after importing both.
Fix: Order imports as `import modin.pandas as pd; import swifter` or call `swifter.register_modin()` if `swifter` is imported first.
breaking Swifter relies on recent features of the pandas extension API. Older versions of pandas (e.g., pre-1.0) may not be fully compatible or may cause unexpected behavior.
Fix: Always ensure you are using a recent and updated version of pandas (`pip install -U pandas`).
gotcha When using Dask as a backend for large datasets, `swifter` is limited to `axis=1` (row-wise application) for `df.swifter.apply()`. Attempting `axis=0` with large Dask-backed DataFrames may not use Dask or might result in errors.
Fix: Structure your `apply` operations to be row-wise (`axis=1`) when dealing with large datasets that would trigger Dask usage, or consider alternative Dask-native operations for column-wise transforms.

Install

pip install swifter Install base package
pip install -U pandas swifter[notebook] Install with notebook progress bar
pip install -U swifter[groupby] Install with groupby.apply dependencies (includes Ray)
conda install -c conda-forge swifter Install via Conda

Imports

swifter
```
import swifter
```
Imports swifter to add the `.swifter` accessor to pandas objects.
pandas
```
import pandas as pd
```
Standard import for pandas functionality, required for swifter integration.

Quickstart

This quickstart demonstrates how to apply a custom, computationally intensive function to a pandas Series and DataFrame using `swifter.apply()`. Swifter automatically chooses the most efficient execution backend (vectorized, Dask, or multiprocessing) based on the data size and function complexity.

import pandas as pd
import swifter

# Create a sample DataFrame
df = pd.DataFrame({
    'value': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
    'category': ['A', 'B', 'A', 'C', 'B', 'A', 'C', 'B', 'A', 'C']
})

# Define a custom function
def complex_calculation(x):
    import time
    time.sleep(0.001) # Simulate a time-consuming operation
    return x * x + 1

# Use swifter.apply() on a Series
df['squared_value'] = df['value'].swifter.apply(complex_calculation)

# Use swifter.apply() on a DataFrame (row-wise)
df['sum_squared'] = df.swifter.apply(lambda row: row['value']**2 + row['value'], axis=1)

print(df.head())

view raw JSON →