Fugue: Abstraction Layer for Distributed Computing

0.9.7 · active · verified Thu Apr 09

Fugue is a Python library that provides a unified interface for defining data workflows, allowing them to run seamlessly on Pandas, Spark, Dask, Ray, and other distributed computing engines without code changes. It's designed to make data pipelines more portable and testable. The current version is 0.9.7, and it maintains an active release cadence with frequent patches and updates.

Warnings

breaking Fugue SQL and related functions/dependencies were moved to an optional `[sql]` extra package in version 0.9.0. Users relying on these features without explicit installation will encounter `ImportError`.
Fix: Ensure `fugue[sql]` is installed: `pip install 'fugue[sql]'`.
gotcha Fugue has strict Python version requirements. From 0.9.0 onwards, it generally requires Python 3.10 or higher. Using older Python versions will lead to installation and runtime errors.
Fix: Upgrade your Python environment to 3.10 or newer.
gotcha There have been several patches related to Pandas compatibility (e.g., specific version pinning for Pandas<3, then unpinning). Users might face issues if their Pandas version is incompatible with the installed Fugue version, especially when using Pandas 2.x.
Fix: Always install Fugue with its recommended extras (e.g., `pip install fugue[spark]`) to ensure compatible dependency versions are pulled. If issues persist, try upgrading/downgrading Pandas to align with Fugue's tested range or refer to Fugue's release notes for specific constraints.
deprecated The `DataFrame` class was moved from `fugue.dataframe` to `fugue.collections.dataframe`. While the old path might still work for some time due to redirects, it's considered deprecated.
Fix: Update imports from `from fugue.dataframe import DataFrame` to `from fugue.collections.dataframe import DataFrame`.
gotcha Compatibility with distributed engines (Spark, Ray, Dask) is continuously updated. Older Fugue versions might not work with the latest versions of these engines, and vice-versa. Users should monitor Fugue release notes for compatibility fixes.
Fix: Keep Fugue updated to the latest patch version alongside your distributed engine. If specific compatibility issues arise, check the Fugue GitHub issues or documentation for known conflicts.

Install

pip install fugue Base installation
pip install 'fugue[spark]' For Spark integration
pip install 'fugue[dask]' For Dask integration
pip install 'fugue[ray]' For Ray integration
pip install 'fugue[sql]' For FugueSQL features

Imports

FugueWorkflow
```
from fugue import FugueWorkflow
```
fugue_transform
```
from fugue.api import fugue_transform
```
DataFrame
```
from fugue.collections.dataframe import DataFrame
```
The `DataFrame` class moved to `fugue.collections.dataframe` in newer versions.

Quickstart

This quickstart demonstrates defining a simple Fugue workflow. It creates a DataFrame, applies a transformation function (`map_to_string`) that converts a numeric column to a string, and then executes the workflow, printing the result to the console. The `schema` parameter in `transform` is crucial for explicit schema definition.

from fugue import FugueWorkflow
import pandas as pd

def map_to_string(df: pd.DataFrame) -> pd.DataFrame:
    return df.assign(value_str=df['value'].astype(str))

with FugueWorkflow() as dag:
    df = dag.df([{"id": 1, "value": 10}, {"id": 2, "value": 20}])
    result = df.transform(map_to_string, schema="*,value_str:str")
    result.show()

view raw JSON →