Quinn PySpark Utilities

Warnings

• breaking Version 0.2.0 introduced significant breaking changes to the directory structure and import interfaces for PySpark extensions and functions.
  Fix: Users migrating from versions prior to 0.2.0 will need to update their import statements and potentially function calls to align with the new module structure. Refer to the GitHub releases for details.
• deprecated The `print_athena_create_table` functionality has been deprecated.
  Fix: Avoid using `print_athena_create_table`. Check the latest documentation for alternative methods to generate Athena table DDL or construct it manually.
• gotcha Using wildcard imports (`from quinn import *` for the main `quinn` module, or even `from quinn.extensions import *` if only specific functions are needed) can make it difficult to trace where functions originate, potentially leading to name collisions.
  Fix: For the main `quinn` library, prefer `import quinn` and then call functions as `quinn.function_name()`. For extensions, `from quinn.extensions import *` is often intended for the automatic patching, but be aware of the namespace pollution. If only specific extension functions are needed and not the automatic patching, consider importing them directly, though this might miss the intended auto-patching behavior.
• gotcha PySpark operations (including those using `quinn`) are lazily evaluated. Transformations build a logical plan and are only executed when an action (e.g., `show()`, `collect()`, `write()`) is called. This can be a common pitfall for Python developers used to immediate execution.
  Fix: Always remember that your DataFrame transformations won't run until an action is triggered. Use actions strategically for debugging (e.g., `df.show()`) and ensure your job design accounts for this lazy execution model.

Install

• pip install quinn Install with pip

Imports

• quinn

  import quinn

  Import the main quinn library for DataFrame transformations and helper functions.
• extensions

  from quinn.extensions import *

  The `extensions` module automatically patches methods like `create_df` onto the SparkSession and `Column` objects when imported with `*`. Using specific imports for individual extensions is less common as the primary intention is to enhance the PySpark environment broadly.
• F

  from pyspark.sql import functions as F

  This is a standard PySpark best practice and crucial for working with Column expressions, which quinn often extends or uses.

Quickstart

This quickstart demonstrates how to initialize a SparkSession, create a DataFrame using `quinn`'s extended `create_df` method, apply a common DataFrame transformation like `snake_case_columns`, and utilize a Column extension such as `isTruthy`.

from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, StringType, IntegerType
import quinn
from quinn.extensions import *

# Initialize SparkSession
spark = SparkSession.builder \
    .appName("QuinnQuickstart") \
    .master("local[*]") \
    .getOrCreate()

# NOTE: `spark.create_df` and Column methods like `isTruthy()` are automatically
# available after `from quinn.extensions import *`

# Create a DataFrame using quinn's extended create_df method
data = [
    ("Alice", 1, "USA"),
    ("Bob", 2, "Canada"),
    ("Charlie", 3, "Mexico")
]
schema_def = [
    ("firstName", "string", True),
    ("id", "integer", True),
    ("country", "string", True)
]
df = spark.create_df(data, schema_def)
print("Original DataFrame Schema:")
df.printSchema()
print("Original DataFrame Data:")
df.show()

# Apply a quinn DataFrame transformation: snake_case_columns
snake_cased_df = quinn.snake_case_columns(df)
print("\nDataFrame with snake_cased columns:")
snake_cased_df.printSchema()
snake_cased_df.show()

# Demonstrate a Column extension (e.g., isTruthy from quinn.extensions)
from pyspark.sql import functions as F
extended_df = df.withColumn("is_id_truthy", F.col("id").isTruthy())
print("\nDataFrame with 'is_id_truthy' column (using quinn extension):")
extended_df.show()

# Stop SparkSession
spark.stop()