Collate SQL Lineage

2.1.0 · active · verified Sun Apr 12

Collate SQL Lineage is a Python tool designed to analyze SQL statements and extract their data lineage, mapping relationships between tables and columns. It builds upon `sqllineage` for core lineage parsing and `sqlfluff` for robust SQL parsing and linting. The library provides a programmatic interface for integrating lineage analysis into data governance, compliance, and impact analysis workflows. It is currently at version 2.1.0 and maintains an active development and release cadence.

Warnings

breaking Version 2.0.0 introduced a significant API refactor, deprecating direct function calls for lineage analysis in favor of the `CollateSQLLineageRunner` class. Code written for versions prior to 2.0.0 will no longer work.
Fix: Migrate existing code to use `CollateSQLLineageRunner` by instantiating the class and calling its `run()` method with the SQL string. For example, `runner = CollateSQLLineageRunner(); result = runner.run(sql=my_sql)`. Old direct function calls will raise an `AttributeError` or `ImportError`.
gotcha Lineage extraction relies on `sqllineage` and `sqlfluff`, which may have limitations in fully parsing highly complex, non-standard, or niche SQL dialects. Edge cases or unsupported syntax might lead to incomplete or incorrect lineage results.
Fix: Test with representative SQL samples. For problematic queries, simplify them where possible or report issues to `collate-sqllineage` or its underlying dependencies. Enable verbose logging in `CollateSQLLineageRunner(config={'verbose': True})` to get more diagnostic information on parsing failures.
gotcha While `collate-sqllineage` specifies dependency versions (`sqllineage`, `sqlfluff`), future updates to these underlying libraries might introduce subtle breaking changes in parsing behavior or output format that could impact lineage results, even if `collate-sqllineage`'s direct API remains stable.
Fix: For production environments, pin `collate-sqllineage` and its direct dependencies to specific versions. Regularly test against new dependency versions in development before deploying to ensure consistent lineage results. Review `collate-sqllineage`'s release notes for compatibility updates.

Install

pip install collate-sqllineage Install stable version

Imports

CollateSQLLineageRunner
```
from collate_sqllineage.runner import CollateSQLLineageRunner
```
As of v2.0.0, direct function calls like `run_lineage` were removed in favor of the `CollateSQLLineageRunner` class.

Quickstart

This quickstart demonstrates how to initialize `CollateSQLLineageRunner` and use its `run()` method to extract data lineage from a sample SQL `INSERT` statement. The output includes source and target tables, with the full result containing more granular details like column-level lineage.

from collate_sqllineage.runner import CollateSQLLineageRunner

# Example SQL statement to analyze
sql_statement = """
INSERT INTO target_schema.target_table (col_a, col_b)
SELECT
    source_schema.source_table_1.col_x,
    source_schema.source_table_2.col_y
FROM
    source_schema.source_table_1
JOIN
    source_schema.source_table_2 ON source_schema.source_table_1.id = source_schema.source_table_2.id
WHERE
    source_schema.source_table_1.status = 'active';
"""

# Initialize the runner. Configuration can be passed here (e.g., for verbose logging).
# runner = CollateSQLLineageRunner(config={'verbose': True})
runner = CollateSQLLineageRunner()

# Run the lineage analysis
lineage_result = runner.run(sql=sql_statement)

# Print the extracted lineage information
print("--- Extracted SQL Lineage ---")
print(f"Source Tables: {lineage_result.get('tables', {}).get('source', [])}")
print(f"Target Tables: {lineage_result.get('tables', {}).get('target', [])}")

# The 'lineage_result' dictionary contains more detailed information including columns, statements, etc.
# print(lineage_result)

view raw JSON →