PySpark

Warnings

• breaking PySpark 4.0 dropped support for Python 3.8. Ensure your Python environment is 3.10 or higher.
  Fix: Upgrade your Python environment to 3.10 or newer. PySpark 4.0+ supports Python 3.10, 3.11, and 3.12.
• breaking The minimum required Pandas version for PySpark 4.0+ was raised. If using the Pandas API on Spark, ensure your Pandas installation meets the new requirement (1.0.5 or higher).
  Fix: Upgrade Pandas in your environment: `pip install 'pandas>=1.0.5'`
• breaking In PySpark 4.1, the Pandas API on Spark operates under ANSI mode by default. This might change behavior for certain operations, especially concerning null handling and type conversions.
  Fix: Review your code for potential changes in behavior related to ANSI SQL mode if relying on Pandas API on Spark. Refer to Spark documentation for specifics on ANSI mode implications.
• gotcha PySpark operations are lazily evaluated. Transformations (e.g., `filter`, `select`) do not execute immediately; computation only triggers when an action (e.g., `show`, `count`, `collect`, `write`) is called.
  Fix: Understand the lazy evaluation model. Use `df.explain()` to see the execution plan without triggering computation, and be aware that actions force execution.
• gotcha Calling `.collect()` on a large DataFrame can pull all distributed data to the driver node, potentially causing OutOfMemory (OOM) errors and crashing the application.
  Fix: Avoid `.collect()` for large datasets. Use `df.show()`, `df.take(N)`, `df.limit(N).toPandas()`, or write to distributed storage for inspecting data or small samples.
• gotcha Apache Spark (and thus PySpark) requires a Java Development Kit (JDK) to be installed and the `JAVA_HOME` environment variable to be correctly set, otherwise PySpark applications will fail to launch.
  Fix: Install a compatible JDK (e.g., OpenJDK 8 or 11) and set the `JAVA_HOME` environment variable to point to your JDK installation directory.
• gotcha Ignoring partitioning strategies for DataFrames can lead to data skew and inefficient shuffles, significantly degrading performance for wide transformations like `groupBy` or `join`.
  Fix: Analyze your data's distribution and use `df.repartition(N, *columns)` or `df.coalesce(N)` before wide transformations to optimize partitioning, especially for high-cardinality columns.

Install

• pip install pyspark Base installation
• pip install pyspark[sql] pyspark[pandas_on_spark] pyspark[connect] With optional components (SQL, Pandas API, Connect)

Imports

• SparkSession

  from pyspark.sql import SparkSession

  The primary entry point for PySpark functionality.
• functions

  from pyspark.sql import functions as F

  Commonly imported for SQL-like functions such as col, sum, when, lit, etc.
• Row

  from pyspark.sql import Row

  Used for creating DataFrames from lists of Row objects.
• types

  from pyspark.sql.types import StructType, StructField, StringType, IntegerType

  Used for explicitly defining DataFrame schemas.

Quickstart

This quickstart demonstrates how to initialize a SparkSession, create a DataFrame from Python data, display its schema and content, perform a filtering transformation, and then a grouping and aggregation. It also highlights the importance of setting `JAVA_HOME`.

import os
from pyspark.sql import SparkSession
from pyspark.sql.functions import col

# PySpark requires JAVA_HOME to be set. Ensure it points to your JDK installation.
# For example: os.environ['JAVA_HOME'] = '/usr/lib/jvm/java-11-openjdk-amd64'

# Create a SparkSession - the entry point to Spark functionality
spark = SparkSession.builder \
    .appName("PySparkQuickstart") \
    .getOrCreate()

# Create a simple DataFrame
data = [("Alice", 1), ("Bob", 2), ("Charlie", 3), ("David", 1)]
columns = ["name", "value"]
df = spark.createDataFrame(data, columns)

# Show the DataFrame schema and data
df.printSchema()
df.show()

# Perform a simple transformation (filter) and action (show)
filtered_df = df.filter(col("value") > 1)
print("Filtered DataFrame:")
filtered_df.show()

# Group by 'value' and count occurrences
grouped_df = df.groupBy("value").count()
print("Grouped DataFrame:")
grouped_df.show()

# Stop the SparkSession
spark.stop()