DiscoverX - Lakehouse Mapping and Search

Common errors

• ModuleNotFoundError: No module named 'discoverx'

  cause The `dbl-discoverx` package is either not installed or, in a Databricks notebook, the Python kernel has not been restarted after installation.
  fix Run `%pip install dbl-discoverx` (or `pip install dbl-discoverx` in non-Databricks environments) and then `dbutils.library.restartPython()` if in a Databricks notebook.

• Py4JJavaError: An error occurred while calling o.s.sql.functions.regexp_extract. Invalid regular expression

  cause This error occurs in Databricks Runtime 15.4 LTS+ when a regular expression provided in a `with_sql` command (or other Spark SQL regex functions) is considered invalid by Photon's updated regex engine.
  fix Review and correct the SQL regular expression pattern to adhere strictly to Apache Spark's regex syntax. Test patterns in a standard Spark SQL context if unsure.

• Performance degradation or OutOfMemoryError for operations on a very large number of tables.

  cause The default concurrency (10) for multi-table operations might not be optimal for extremely large lakehouses or specific workload patterns, leading to resource contention or inefficient execution.
  fix Adjust the concurrency level using the `with_concurrency()` method (e.g., `dx.from_tables(...).with_concurrency(20).with_sql(...).apply()`) to match your Databricks cluster's capacity and the nature of the operations.

Warnings

• breaking DiscoverX is a Databricks Labs project and is provided "AS-IS" without formal Service Level Agreements (SLAs). Issues should be filed as GitHub Issues and will be reviewed as time permits.
  Fix: Be aware of the experimental nature; do not rely on it for critical production workloads without internal support.
• deprecated The `scan` command has been deprecated. Users should migrate to `intro` or `scan (experimental)` for semantic classification and other scanning functionalities.
  Fix: Replace `.scan(...)` with `.intro()` for general overview or `.scan(experimental=True, ...)` for detailed scanning.
• gotcha When installing `dbl-discoverx` within a Databricks notebook using `%pip install`, it is often necessary to restart the Python kernel (`dbutils.library.restartPython()`) for the newly installed package to be properly loaded and available.
  Fix: Always follow `%pip install dbl-discoverx` with `dbutils.library.restartPython()` in Databricks notebooks.
• gotcha On Databricks Runtime 15.4 LTS and above, regular expression handling in Photon is updated to match Apache Spark behavior. Previously accepted invalid regex patterns in `with_sql` commands might now cause queries to fail.
  Fix: Ensure all regex patterns used in `with_sql` or similar operations are strictly valid according to Apache Spark's regular expression syntax.

Install

• pip install dbl-discoverx Standard pip install
• %pip install dbl-discoverx Databricks notebook

Imports

• DX

  from discoverx import DX

Quickstart

This quickstart demonstrates how to initialize DiscoverX, define a set of tables using a wildcard pattern, and then apply a SQL template (counting rows) concurrently across all matching tables in a Databricks environment.

from discoverx import DX

# Initialize DiscoverX. 'locale' can be set for region-specific rules.
dx = DX(locale="US")

# Define the tables to operate on using a wildcard pattern
# Example: all tables in 'my_catalog.my_schema'
from_tables = "my_catalog.my_schema.*"

# Example: Count rows in all selected tables and display the results
# The '{full_table_name}' placeholder is automatically replaced.
table_counts = dx.from_tables(from_tables).with_sql("SELECT COUNT(*) FROM {full_table_name}").apply()

# Display the resulting DataFrame
table_counts.display()