Chispa

Warnings

• breaking Chispa versions prior to 0.12.0 may not be fully compatible with PySpark 4.x. Version 0.12.0 explicitly adds support for Spark 4.x, so users upgrading their PySpark environment should ensure they are on Chispa 0.12.0 or newer.
  Fix: Upgrade chispa to 0.12.0 or a later version: `pip install --upgrade chispa`
• deprecated The internal `bcolors` utility module was removed in `v0.11.0`. While primarily an internal refactor, direct imports of `chispa.bcolors` will now fail.
  Fix: Remove any direct imports of `chispa.bcolors`. If custom console coloring is needed, use a dedicated library or Python's `colorama`.
• gotcha By default, `assert_df_equality` performs a strict comparison, expecting identical schemas (including nullability and metadata), column order, and row order. Divergences in any of these aspects will cause an assertion failure unless corresponding `ignore_*` flags (e.g., `ignore_nullable`, `ignore_column_order`, `ignore_row_order`, `ignore_metadata`) are explicitly set to `True`.
  Fix: Review the specific comparison requirements. Use `ignore_nullable=True`, `ignore_metadata=True`, `ignore_column_order=True`, or `ignore_row_order=True` as appropriate for your test case. For floating-point comparisons, consider `assert_approx_df_equality` or the `precision` argument.
• gotcha Prior to `v0.11.1`, a bug existed in `assert_df_equality` where using both `ignore_columns` and `ignore_row_order` simultaneously could lead to incorrect DataFrame comparisons due to faulty row ordering logic. This was resolved in `v0.11.1`.
  Fix: Upgrade chispa to version 0.11.1 or newer to ensure correct behavior when combining `ignore_columns` and `ignore_row_order`.

Install

• pip install chispa Install latest version

Imports

• assert_df_equality

  from chispa.dataframe_comparer import assert_df_equality

• assert_column_equality

  from chispa.column_comparer import assert_column_equality

• assert_approx_df_equality

  from chispa.dataframe_comparer import assert_approx_df_equality

  Used for comparing DataFrames with floating point numbers, allowing for a specified precision.

Quickstart

This quickstart demonstrates how to use `chispa.dataframe_comparer.assert_df_equality` to compare two PySpark DataFrames. It initializes a SparkSession, creates two DataFrames (one identical, one with a subtle difference), and then uses `assert_df_equality` to check their equivalence, showcasing both a passing and a failing scenario with Chispa's descriptive error messages.

import os
from pyspark.sql import SparkSession
from chispa.dataframe_comparer import assert_df_equality

# Initialize SparkSession (for local testing, typically done in a pytest fixture)
spark = SparkSession.builder.appName("ChispaQuickstart").getOrCreate()

# Create an 'actual' DataFrame
data_actual = [
    ("john", 1),
    ("jane", 2),
    ("doe", 3)
]
df_actual = spark.createDataFrame(data_actual, ["name", "id"])

# Create an 'expected' DataFrame (identical for a passing test)
data_expected = [
    ("john", 1),
    ("jane", 2),
    ("doe", 3)
]
df_expected = spark.createDataFrame(data_expected, ["name", "id"])

# Assert equality - this test should pass
try:
    assert_df_equality(df_actual, df_expected)
    print("Assertion Passed: df_actual and df_expected are equal.")
except Exception as e:
    print(f"Assertion Failed: {e}")

# Create a different 'expected' DataFrame to demonstrate a failing test
data_expected_fail = [
    ("john", 1),
    ("jane", 99), # Intentional difference
    ("doe", 3)
]
df_expected_fail = spark.createDataFrame(data_expected_fail, ["name", "id"])

# Assert equality - this test should fail, showing descriptive error
try:
    assert_df_equality(df_actual, df_expected_fail)
    print("Assertion Passed (unexpectedly): df_actual and df_expected_fail are equal.")
except Exception as e:
    print(f"Assertion Failed (as expected): {e}")

# Stop SparkSession
spark.stop()