Databricks Labs - PySpark Synthetic Data Generator

Common errors

• ModuleNotFoundError: No module named 'jmespath'

  cause Some environments, particularly non-Databricks Spark setups like Google Colab, might not have `jmespath` pre-installed, leading to import errors when `dbldatagen` attempts to load.
  fix Install the `jmespath` package: `pip install jmespath`.

• Error: Attempting to add a column named id which conflicts with the internal seed column.

  cause The `id` column name is reserved internally by `dbldatagen` for its seeding mechanism. If you define a column with this name, it causes a conflict.
  fix Rename your column or, if you specifically need an 'id' column for your data, use the `seedColumnName` parameter in `DataGenerator` to assign a different internal seed column name (e.g., `DataGenerator(spark, ..., seedColumnName='_internal_seed_id')`).

• pyspark.sql.utils.AnalysisException: Cannot resolve 'element_at(`array_column`, 0)` due to data type mismatch.

  cause Older versions of PySpark or specific Databricks Runtimes might have incompatibilities with direct array indexing or `element_at` function usage. This was specifically addressed in `dbldatagen` v0.3.6.
  fix Ensure you are using `dbldatagen` version 0.3.6 or later, and a compatible PySpark version (>=3.2.1 recommended for 0.4.0+). If manually constructing Spark SQL expressions, consult PySpark documentation for array access compatible with your runtime.

Warnings

• breaking Version 0.4.0 increased the minimum `pyspark` version to 3.2.1 and requires Databricks runtime 10.4 LTS or later. Older PySpark versions or Databricks runtimes will not be compatible.
  Fix: Upgrade your PySpark installation (`pip install 'pyspark>=3.2.1'`) and ensure your Databricks Runtime is 10.4 LTS or newer.
• gotcha Spark SQL column names are case-insensitive. Defining new columns with the same name but different casing than existing ones may lead to unexpected behavior or errors in downstream operations.
  Fix: Ensure consistent casing for column names throughout your data generation specifications to avoid conflicts.
• gotcha When using `dbldatagen.constraints.UniqueCombinations` with streaming dataframes, deduplication is performed only within a batch. For full stream-wide deduplication, you must implement explicit watermarking and deduplication logic on the resultant DataFrame, which can be resource-intensive for high-volume streams.
  Fix: For stateful deduplication across an entire stream, apply watermarking and deduplication using Spark's native streaming APIs on the DataFrame produced by `build()`.
• gotcha The column name 'id' is reserved internally by `dbldatagen` as the seed column for data generation. If your generated data requires a column named 'id' with different semantics, it will conflict with this internal mechanism.
  Fix: Customize the internal seed column name by setting the `seedColumnName` attribute when creating the `DataGenerator` instance (e.g., `DataGenerator(..., seedColumnName="_internal_id")`).
• gotcha When running on Databricks Unity Catalog enabled environments with Runtimes prior to 13.2, `dbldatagen` requires 'Single User' or 'No Isolation Shared' access modes. 'Shared' access mode in these older runtimes lacks necessary features (e.g., 3rd party libraries, Python UDFs) for `dbldatagen` to function correctly. This limitation is resolved in Databricks Runtimes 13.2 and newer.
  Fix: Use Databricks Runtime 13.2 or later, or configure your cluster to use 'Single User' or 'No Isolation Shared' access modes if using older runtimes.

Install

• pip install dbldatagen Standard pip install
• %pip install dbldatagen Databricks Notebook

Imports

• DataGenerator
  wrong:

  import dbldatagen.DataGenerator

  correct:

  from dbldatagen import DataGenerator

  The primary class for defining custom data generation specifications. Prefer aliased import for brevity: import dbldatagen as dg.
• dg

  import dbldatagen as dg

  Standard aliased import for the dbldatagen library, commonly used with DataGenerator or Datasets classes.
• Datasets
  wrong:

  from dbldatagen.datasets import Datasets

  correct:

  import dbldatagen as dg
  df = dg.Datasets(spark, "basic/user").get().build()

  While technically possible, directly importing submodules like `Datasets` is less common. The recommended pattern is `dg.Datasets` for accessing standard datasets.

Quickstart

This quickstart demonstrates how to generate a synthetic dataset using `dbldatagen`'s `Datasets` class, which provides pre-configured data generation recipes. It initializes a SparkSession (if not already present), creates a 'basic/user' dataset with 1 million rows and 4 partitions, then displays its schema and the first few rows. This approach is recommended for quickly generating common synthetic data patterns.

from pyspark.sql import SparkSession
import dbldatagen as dg

# Initialize SparkSession (if not in Databricks already)
try:
    spark
except NameError:
    spark = SparkSession.builder.appName("dbldatagen_quickstart").getOrCreate()

# Generate a basic user dataset using standard datasets feature
# This creates 1 million rows and 4 partitions
print("Generating a basic user dataset...")
df = dg.Datasets(spark, "basic/user").get(rows=1_000_000, partitions=4).build()

# Display schema and a few rows
print("Schema:")
df.printSchema()

print("Sample data:")
df.show(5, truncate=False)

# Stop SparkSession if it was created here (optional)
# spark.stop()