Spark Expectations

Spark Expectations is a Python library by Nike-Inc that facilitates in-flight data quality (DQ) checks within Apache Spark jobs. It enables validation of data against defined rules (row-level, aggregate, and query-based) as data is processed, ensuring only quality data reaches its destination. Erroneous records are quarantined into a separate error table, and aggregated metrics are provided. The library is actively maintained with regular updates; the current version is 2.9.1.