DataComPy

Warnings

• breaking `LegacySparkCompare` and `SparkPandasCompare` classes were removed in version 0.17.0. For Spark DataFrame comparison, use `SparkSQLCompare` (from `datacompy.spark.sql`) or the general `datacompy.is_match` function with the appropriate backend setup (often leveraging Fugue).
  Fix: Migrate to `datacompy.spark.sql.SparkSQLCompare` or `datacompy.is_match` with Fugue backend. Consult the documentation for specific backend usage.
• gotcha The default `datacompy.Compare` (Pandas native) requires DataFrames to fit into memory. Comparing very large datasets may lead to out-of-memory errors. For larger-than-memory datasets, consider using the Spark SQL or Polars backend implementations, which are more performant for big data.
  Fix: For large datasets, use `datacompy` with Spark, Polars, or Fugue backends after installing the relevant extras (e.g., `pip install datacompy[spark]`).
• gotcha When comparing floating-point numbers, minor precision differences can cause mismatches even if values are conceptually the same. Always use `abs_tol` (absolute tolerance) and/or `rel_tol` (relative tolerance) parameters in `datacompy.Compare` to account for these expected deviations.
  Fix: Set appropriate `abs_tol` and `rel_tol` values when initializing the `Compare` object (e.g., `abs_tol=0.0001`, `rel_tol=0.01`).
• gotcha DataComPy's duplicate row matching logic can be 'naïve' if `join_columns` do not uniquely identify rows. If many duplicates exist, `datacompy` sorts by other fields to create a temporary ID, which might not align with desired matching.
  Fix: Ensure `join_columns` provide sufficient granularity to uniquely identify rows, or pre-process duplicates if specific matching behavior is required.
• gotcha If 'DATACOMPY_NULL' exists as a legitimate string value in your `join_columns`, it can conflict with how `datacompy` internally handles null values during duplicate matching, potentially causing merge failures.
  Fix: Either rename or replace 'DATACOMPY_NULL' values in your join columns before comparison, or fill nulls with a different sentinel value of your choice.
• deprecated DataComPy is actively moving towards a v1 release. The `0.19.x` branch will receive only dependency updates and critical bug fixes, with no new features. Future `v1` releases may introduce breaking changes as development targets `v1` branches (`develop` and `main`).
  Fix: Monitor GitHub releases and documentation for `v1` migration guides when upgrading from `0.19.x`.
• gotcha Python 3.12 and above currently have limited support with Spark and Ray within the Fugue backend. Pandas and Polars comparisons should work fine.
  Fix: If using Spark or Ray with Fugue, consider using Python versions <3.12 until full compatibility is announced. Otherwise, Pandas and Polars backends are supported.

Install

• pip install datacompy Basic Installation
• pip install datacompy[spark] For Spark DataFrame comparison
• pip install datacompy[polars] For Polars DataFrame comparison
• pip install datacompy[fugue] For Fugue-supported backends (Dask, DuckDB, Ray, etc.)
• pip install datacompy[snowflake] For Snowflake (Snowpark) DataFrame comparison

Imports

• Compare

  import datacompy
  compare = datacompy.Compare(...)

• Compare

  from datacompy.core import Compare
  compare = Compare(...)

• SparkCompare

  from datacompy.spark.sql import SparkSQLCompare

  LegacySparkCompare and SparkPandasCompare were removed in v0.17.0. The recommended approach for Spark is SparkSQLCompare, or using the generic `datacompy.is_match` with Fugue for various backends.

Quickstart

This quickstart demonstrates how to compare two Pandas DataFrames using `datacompy.Compare`. It initializes two sample DataFrames, then creates a `Compare` object specifying the join column and tolerance levels for numeric comparisons. Finally, it prints a comprehensive report of differences and checks for an overall match.

import pandas as pd
import datacompy
from io import StringIO

data1 = """acct_id,dollar_amt,name
1,123.45,Alice
2,67.89,Bob
3,99.99,Charlie
4,10.00,David
"""
df1 = pd.read_csv(StringIO(data1))

data2 = """acct_id,dollar_amt,name
1,123.45,Alice
2,67.90,Bobbert
3,99.99,Charlie
5,11.00,Eve
"""
df2 = pd.read_csv(StringIO(data2))

# Perform comparison, joining on 'acct_id'
# abs_tol and rel_tol are crucial for float comparisons
compare = datacompy.Compare(
    df1, df2,
    join_columns='acct_id',
    abs_tol=0.001, # Absolute tolerance for numeric fields
    rel_tol=0.0    # Relative tolerance for numeric fields
)

# Print a detailed report
print(compare.report())

# Check if dataframes match entirely
print(f"DataFrames match: {compare.matches()}")