Soda Core Spark Integration (Legacy)

Warnings

• breaking The `soda-core-spark` package has been officially deprecated. It, along with `Soda SQL`, has been replaced by `Soda Core` as the unified solution for data quality testing.
  Fix: Migrate to `soda-core` and use its native Apache Spark connection capabilities. This involves installing `soda-core` and configuring your `Scan` object with a Spark session.
• breaking Soda Core v4 (released January 28, 2026) introduces 'Data Contracts' as the default method for defining data quality rules, replacing the previous 'checks language' syntax. This is a significant breaking change for users migrating from older versions of Soda Core or `soda-core-spark`.
  Fix: Review the Soda Core v4 documentation for migration guidance and examples of the new Data Contract format. Pin your `soda-core` dependency to a v3 version (e.g., `soda-core==3.5.6`) if you are not ready to upgrade to v4.
• gotcha Soda Core v3 (which is the relevant version for migrating from `soda-core-spark`) has known compatibility limitations. Specifically, it does not support Apache Spark 4.0 or Python 3.12.
  Fix: Ensure your environment uses compatible versions of Spark (e.g., Spark 3.x) and Python (e.g., Python 3.11 or lower) when working with `soda-core` v3. Check the official Soda Core documentation for the latest compatibility matrix.
• gotcha When using Soda Core with Spark DataFrames, you typically need to run Soda programmatically and register DataFrames as temporary views for checks to be executed.
  Fix: After creating or loading your Spark DataFrame, use `df.createOrReplaceTempView("your_temp_view_name")` to make it accessible to Soda scans. Then, define your checks in YAML (or Data Contracts in v4) against this temporary view name.

Install

• pip install soda-core-spark Install deprecated package

Imports

• scan

  from sodaspark import scan

  The `sodaspark` module was part of the deprecated `soda-core-spark` for programmatic scans. For `soda-core` (the replacement), use `from soda.scan import Scan`.

Quickstart

This example demonstrates how to perform data quality checks on a Spark DataFrame using the deprecated `soda-core-spark` library (`sodaspark`). It initializes a Spark session, creates a sample DataFrame, defines data quality checks in a YAML string, and executes the scan programmatically. Please note that for modern usage, you should migrate to `soda-core`.

import os
from pyspark.sql import SparkSession
from sodaspark import scan

# Initialize Spark Session
spark_session = SparkSession.builder.appName("SodaSparkExample").getOrCreate()

# Create a sample DataFrame
df = spark_session.createDataFrame([
    {"id": "1", "name": "Alice", "age": 30},
    {"id": "2", "name": "Bob", "age": None},
    {"id": "3", "name": "Charlie", "age": 35},
    {"id": "4", "name": "David", "age": 22}
])

# Define data quality checks in YAML format
# For deprecated soda-spark, checks are passed as a string.
# For modern Soda Core, these would typically be in a separate .yml file.
scan_definition = """
table_name: my_dataframe
metrics:
  - row_count
  - missing_count(age)
  - avg(age)
checks:
  - row_count > 0
  - missing_count(age) < 1
  - avg(age) between 20 and 40
"""

# Execute the scan
# Note: data_source_name should be set if connecting to Soda Cloud,
# but for local programmatic scans, it's often 'spark_df' by default.
scan_results = scan.execute(
    data_frame=df, 
    scan_definition=scan_definition,
    data_source_name="spark_df" # Can be customized
)

print("Scan Results:")
print(scan_results.get_json_representation())

# Stop Spark Session
spark_session.stop()

# IMPORTANT: This quickstart uses the deprecated `sodaspark` library.
# For current Spark integration, please refer to Soda Core documentation and use
# `from soda.scan import Scan` and `scan.add_spark_session(...)`.