fastparquet

Warnings

• breaking The `fastparquet` project is being retired and is incompatible with `pandas` 3.0 and newer versions. Pandas 3.0 now explicitly depends on `pyarrow`, superseding `fastparquet` for many common workflows. Users should target `pandas<3.0` for continued use or migrate to `pyarrow`.
  Fix: Downgrade Pandas to a version less than 3.0, or migrate your Parquet I/O operations to use PyArrow or other alternatives like Dask with the PyArrow engine.
• gotcha Performance can be significantly impacted by the presence of NULL values and variable-length string encoding in your data. For optimal performance, consider using sentinel values (e.g., NaN) for data types that support them, or fixed-length strings where compatible with your ecosystem.
  Fix: Pre-process data to minimize NULLs or use appropriate sentinel values. Consider fixed-length string types if supported by your data and downstream systems.
• gotcha When installing `fastparquet` via `pip`, it's advisable to install `numpy` first to aid the dependency resolver. If pre-compiled wheels are not available for your system/Python version, or when installing directly from the GitHub repository, a C compiler toolchain and `cython` are required for compilation.
  Fix: `pip install numpy` before `pip install fastparquet`. Ensure you have a C compiler (e.g., GCC on Linux, Xcode command-line tools on macOS, MSVC on Windows) and `pip install cython` if encountering compilation errors.

Install

• pip install fastparquet PyPI
• conda install -c conda-forge fastparquet Conda

Imports

• ParquetFile

  from fastparquet import ParquetFile

  Used for reading existing Parquet files.
• write

  from fastparquet import write

  Used for writing Pandas DataFrames to Parquet files.

Quickstart

This quickstart demonstrates how to create a Pandas DataFrame, write it to a Parquet file using `fastparquet.write`, and then read the data back into a new DataFrame using `fastparquet.ParquetFile.to_pandas()`. It also includes basic file cleanup.

import pandas as pd
from fastparquet import write, ParquetFile
import os

# Create a sample DataFrame
df = pd.DataFrame({
    'col1':,
    'col2': ['A', 'B', 'C', 'D'],
    'col3': [True, False, True, False]
})

filename = "example.parquet"

# Write the DataFrame to a Parquet file with Snappy compression
write(filename, df, compression='SNAPPY')
print(f"DataFrame successfully written to '{filename}'.")

# Read the Parquet file back into a DataFrame
pf = ParquetFile(filename)
df_read = pf.to_pandas()
print(f"DataFrame successfully read from '{filename}':")
print(df_read)

# Clean up the created file
os.remove(filename)