Daft Distributed Dataframes

Warnings

• breaking The original package `getdaft` was renamed to `daft`. Older installations or scripts referencing `getdaft` will fail.
  Fix: Use `pip install daft` and update imports from `import getdaft` to `import daft`.
• gotcha Daft DataFrames are lazy by default. Operations like `select`, `where`, `with_column` build a query plan but do not execute or fetch data until an action such as `collect()`, `show()`, `to_pandas()`, or `write_*()` is explicitly called.
  Fix: Remember to call an action method (e.g., `.collect().to_pandas()`) to materialize and retrieve data from a DataFrame.
• gotcha There is another distinct Python package named `daft-pgm` (for Probabilistic Graphical Models). Ensure you install and import the correct `daft` library for distributed dataframes, which is `daft` from the `Eventual-Inc/Daft` project.
  Fix: Verify you are installing `daft` (not `daft-pgm`) and importing `daft`.
• breaking The default `ddof` parameter for the `stddev` aggregation function changed to `1`. Previously, it might have defaulted to `0` or behaved inconsistently depending on the underlying implementation.
  Fix: Explicitly set `ddof` when calling `stddev()` if you rely on a specific degree of freedom correction (e.g., `df.agg.stddev(ddof=0)`).
• breaking Significant internal refactors have occurred, especially concerning interactions with Apache Arrow libraries (migrating from `arrow2` to `arrow-rs`). While primarily internal, users with highly customized integrations or those relying on specific Arrow-related behaviors might experience subtle changes or compatibility issues.
  Fix: Review existing code for direct interaction with `arrow2` or specific Arrow table/array implementations. Test thoroughly after upgrading, especially for data loading and serialization/deserialization workflows.

Install

• pip install daft Basic Installation
• pip install "daft[openai]" With OpenAI integration
• pip install "daft[ray,aws]" With Ray and AWS integration

Imports

• daft

  import daft

  The package was renamed from `getdaft` to `daft`.

Quickstart

This quickstart demonstrates loading a dataset from Hugging Face, inspecting the lazy DataFrame schema, selecting columns, and materializing a small subset into a Pandas DataFrame to view the data.

import daft
import os

# Load an e-commerce dataset from Hugging Face
# Requires 'daft[huggingface]' to be installed if not already part of your setup
df = daft.read_huggingface("calmgoose/amazon-product-data-2020")

# Inspect the schema (Daft is lazy, so no data is fetched yet)
print("Schema:")
print(df)

# Select a few columns and materialize the first 5 rows to see data
df_subset = df.select(df["Product Name"], df["Category"]).limit(5)
result = df_subset.collect()

print("\nFirst 5 rows:")
print(result.to_pandas())