PySpark Connect Client

4.1.1 · active · verified Wed Apr 15

The `pyspark-client` is the Python Spark Connect client for Apache Spark, providing a decoupled client-server architecture that enables remote connectivity to Spark clusters using the DataFrame API. It uses gRPC and Apache Arrow for efficient communication. The library is part of the broader Apache Spark project and is actively developed, with releases typically aligning with Apache Spark's minor and major version updates. The current version is 4.1.1, supporting Spark 4.1.1.

Warnings

gotcha Spark Connect operates on a decoupled client-server architecture. This means your client application does not run in the same JVM process as the Spark driver. Consequently, direct access to the underlying Java Virtual Machine (JVM) objects (e.g., `df._jdf`) via Py4J, common in traditional PySpark, is not possible.
Fix: Rewrite code to use standard PySpark DataFrame API methods. Avoid direct JVM object manipulation. Focus on the DataFrame API and logical plans.
gotcha The `pyspark-client` is a client library only. It does not include or automatically start a Spark cluster or Spark Connect server. You must have a Spark Connect server running and accessible (e.g., via `start-connect-server.sh` from a full Spark distribution) before your client application can connect.
Fix: Ensure a Spark Connect server is deployed and running, and configure your client's `SparkSession.builder.remote()` method with the correct connection string (e.g., `sc://localhost:15002`).
breaking Starting with Spark 4.0, ANSI SQL mode is enabled by default (`spark.sql.ansi.enabled` set to `true`). This changes how SQL operations handle invalid or undefined results. Operations that previously returned `NULL` (e.g., division by zero, invalid casts) will now throw runtime exceptions.
Fix: Review existing SQL queries and DataFrame operations that might rely on `NULL` handling. Implement explicit error handling or adjust queries to conform to ANSI SQL standards. The legacy behavior can be restored by setting `spark.sql.ansi.enabled` to `false`.
breaking PySpark 4.1 (and thus `pyspark-client` 4.1.1) drops support for Python 3.9. Additionally, minimum required versions for `pyarrow` and `pandas` have been raised to `pyarrow>=15.0.0` and `pandas>=2.2.0`.
Fix: Upgrade your Python environment to 3.10 or later. Update `pyarrow` and `pandas` to their respective minimum required versions or newer. If using `pyspark[connect]`, `pip` will generally handle these dependencies correctly.
gotcha In Spark 4.1, `DataFrame['name']` on the Spark Connect Python Client no longer eagerly validates the column name. This means misspelled or non-existent column names might not raise an error until later in execution.
Fix: Set the environment variable `PYSPARK_VALIDATE_COLUMN_NAME_LEGACY=1` to restore the legacy eager validation behavior if desired.

Install

pip install pyspark-client Basic Install
pip install pyspark[connect] Full PySpark with Connect

Imports

SparkSession
wrong:
```
from pyspark.sql.connect.session import SparkSession
```
correct:
```
from pyspark.sql import SparkSession
```
The `SparkSession` is imported from `pyspark.sql` and then configured for remote connection using `.remote()`.

Quickstart

This quickstart demonstrates how to establish a connection to a Spark Connect server and perform basic DataFrame operations. It assumes a Spark Connect server is already running and accessible at the specified URL (defaulting to `sc://localhost:15002`). The `SPARK_CONNECT_SERVER_URL` environment variable can be used to override the connection string.

import os
from pyspark.sql import SparkSession
from pyspark.sql.functions import lit

# Ensure a Spark Connect server is running, e.g., via ./sbin/start-connect-server.sh
# The default address is sc://localhost:15002

# Connect to the Spark Connect server
spark = SparkSession.builder.remote(os.environ.get('SPARK_CONNECT_SERVER_URL', 'sc://localhost:15002')).getOrCreate()

# Create a DataFrame
df = spark.range(10).withColumn("hello", lit("world"))

# Show the DataFrame
df.show()

# Perform a simple operation
result = df.filter(df.id > 5).count()
print(f"Count of rows with id > 5: {result}")

spark.stop()

view raw JSON →