Soda Core DuckDB Connector

3.5.6 · active · verified Mon Apr 13

soda-core-duckdb is a Python connector that enables Soda Core, an open-source data quality and data contract verification engine, to connect and run data quality checks against DuckDB databases. It facilitates defining data quality expectations in YAML (SodaCL) and executing scans programmatically or via CLI to validate data. The library is actively maintained as part of the broader Soda Core ecosystem, which sees frequent updates and new feature releases.

Warnings

breaking Soda Core v4 introduced a breaking change, moving from a 'checks language' (used in `checks.yml`) to a 'Data Contracts-based syntax' (`contract.yml`). Users upgrading from Soda Core v3.x to v4.x will need to migrate their data quality definitions.
Fix: Refer to Soda Core's official migration guides when upgrading to v4 to convert existing `checks.yml` files to the new `contract.yml` format. For `soda-core-duckdb 3.5.6`, continue using the checks language.
gotcha When defining DuckDB connections in `configuration.yml` for Soda Core v3, some users have reported issues when using the `database` key to specify the DuckDB file path. Using `path` instead often resolves the problem.
Fix: If `database: 'your_file.duckdb'` causes issues, try `connection: path: 'your_file.duckdb'` in your data source configuration YAML.
gotcha Specific versions of `soda-core` (and by extension `soda-core-duckdb`) might have strict `duckdb` version requirements. For example, `soda-core` v3.5.0 relaxed its `duckdb` dependency to `<1.1.0`.
Fix: Always check the `soda-core` and `soda-core-duckdb` release notes or PyPI `requires` section for exact `duckdb` version compatibility. Pin your `duckdb` dependency if encountering conflicts.
gotcha An `ImportError: dlopen(.../site-packages/google/protobuf/pyext/_message.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace` can occur due to a transitive dependency from `opentelemetry` that gathers OSS usage statistics in Soda Core 3.x.
Fix: This issue is often resolved by updating `protobuf` or related dependencies. If the issue persists, consult Soda Core's troubleshooting documentation for specific dependency pinning or exclusion advice.

Install

pip install soda-core-duckdb Install stable version

Imports

Scan
```
from soda.scan import Scan
```
Programmatic interaction with Soda Core for DuckDB is typically done through the central `Scan` class from `soda.scan`, not directly from the connector package.

Quickstart

This quickstart demonstrates how to set up an in-memory DuckDB database, define data quality checks using SodaCL in a `checks.yml` file, and then execute a programmatic scan using the `soda.scan.Scan` class to validate the data. It checks for a positive row count, a specific number of missing values in a column, and the total column count.

import os
import duckdb
from soda.scan import Scan

# 1. Create a dummy DuckDB database and a table
con = duckdb.connect(database=':memory:', read_only=False)
con.execute("CREATE TABLE my_table (id INTEGER, name VARCHAR);")
con.execute("INSERT INTO my_table VALUES (1, 'Alice'), (2, 'Bob'), (3, NULL);")

# 2. Define a data source configuration (optional for in-memory, but good practice)
# This would typically be in a configuration.yml file
# ds_config_content = """
# data_source my_duckdb:
#   type: duckdb
#   connection:
#     database: ':memory:'
# """

# 3. Define SodaCL checks in a checks.yml file
checks_content = """
checks for my_table:
  - row_count > 0
  - missing_count(name) = 1
  - column_count = 2
"""

with open('checks.yml', 'w') as f:
    f.write(checks_content)

# 4. Programmatically run a Soda scan
scan = Scan()
scan.add_duckdb_connection(con)
scan.set_data_source_name('my_duckdb_source') # Logical name for the data source
scan.add_sodacl_yaml_files(file_paths=['checks.yml'])

print('Running Soda scan...')
scan.execute()

if scan.has_failures():
    print('Scan failed!')
    # Optionally, you can assert or raise an error
    # scan.assert_no_checks_fail()
else:
    print('Scan successful: all checks passed or warned.')

print(scan.get_logs_text())

# Clean up temporary files
os.remove('checks.yml')
con.close()

view raw JSON →