Snowpark Connect

1.21.1 · active · verified Thu Apr 16

Snowpark Connect (current version 1.21.1) allows developers to run Snowpark Python code locally using a local Spark cluster, emulating Snowpark functionalities without requiring a direct Snowflake connection. This facilitates offline development, testing, and CI/CD pipelines. It receives updates typically aligned with Snowpark Python and underlying Spark/Snowflake connector releases, and is actively maintained by Snowflake Labs.

Common errors

```
Error: JAVA_HOME is not set
```
cause The `JAVA_HOME` environment variable, which points to your Java installation, is not configured.

fix Set the `JAVA_HOME` environment variable to the path of your Java JDK/JRE installation (e.g., `/usr/lib/jvm/java-11-openjdk-amd64`). Also ensure Java is in your system's PATH.
```
java.io.IOException: Cannot run program "java": error=2, No such file or directory
```
cause The 'java' executable is not found in your system's PATH, or `JAVA_HOME` is incorrectly set.

fix Verify Java is installed and its `bin` directory is included in your system's PATH, or that `JAVA_HOME` points directly to the Java installation directory, and the `bin` subdirectory contains `java`.
```
org.apache.spark.SparkException: Cannot find any SparkSession in the current JVM.
```
cause This error typically indicates that the Spark session was not properly initialized or has been stopped before being used, or there's an issue with the underlying Spark environment.

fix Ensure `connect_with_spark_session_builder()` and `getOrCreateSnowparkSession()` are called successfully before any Snowpark DataFrame operations. Check Spark configuration, especially `spark.jars.packages` for correctness.
```
ModuleNotFoundError: No module named 'pyspark'
```
cause Although `pyspark` might be installed, Python cannot find it, often due to environment path issues or `findspark` not being initialized.

fix If `pyspark` is installed, add `import findspark; findspark.init()` at the beginning of your script. Ensure `pyspark` is indeed installed with `pip show pyspark`.

Warnings

breaking Snowpark Connect has specific Python version requirements (currently >=3.10, <3.13). Using incompatible Python versions can lead to installation failures or runtime errors.
Fix: Ensure your Python environment meets the `requires_python` specification. Consider using `pyenv` or `conda` to manage Python versions.
gotcha A `java.io.IOException` or similar error indicating 'Cannot run program "java"' often means the `JAVA_HOME` environment variable is not set correctly, or Java is not installed or discoverable in your system's PATH.
Fix: Install a compatible Java Runtime Environment (JRE) or Java Development Kit (JDK) (e.g., OpenJDK 8 or 11) and set the `JAVA_HOME` environment variable to its installation directory.
gotcha Incorrect or outdated `spark.jars.packages` values in the `config` dictionary can lead to runtime errors when Snowpark Connect tries to load Spark-Snowflake connector JARs, preventing proper emulation.
Fix: Refer to the official Snowpark Connect documentation or GitHub README for the recommended `spark.jars.packages` values compatible with your desired Spark and Snowpark Python versions.
gotcha It's common to confuse imports: `snowpark_connect` provides the session *builder*, but core Snowpark objects like `Session`, `DataFrame`, and `functions` are imported directly from the `snowpark` library.
Fix: Always import `Session`, `DataFrame`, `functions` etc., from `snowpark` (e.g., `from snowpark.session import Session`), and `connect_with_spark_session_builder` from `snowpark_connect.session`.

Install

pip install snowpark-connect Install Snowpark Connect

Imports

connect_with_spark_session_builder

from snowpark_connect.session import connect_with_spark_session_builder

Session
wrong:
```
from snowpark_connect.session import Session
```
correct:
```
from snowpark.snowpark_session import Session
```
The actual Snowpark Session class is imported from `snowpark`, not `snowpark_connect`.
DataFrame
wrong:
```
from snowpark_connect.dataframe import DataFrame
```
correct:
```
from snowpark.dataframe import DataFrame
```
Snowpark DataFrame objects are part of the `snowpark` library, not `snowpark_connect`.

Quickstart

This quickstart demonstrates how to initialize a local Snowpark Connect session using `connect_with_spark_session_builder`, create a Snowpark session from it, and perform a basic DataFrame operation. It requires a Java Runtime Environment (JRE) to be installed and `JAVA_HOME` configured for Spark to run.

import os
from snowpark_connect.session import connect_with_spark_session_builder
from snowpark.types import StructType, StructField, StringType, IntegerType

# Create a local Spark session that emulates Snowpark behavior
# Ensure these JARs are compatible with your Spark and Snowflake versions.
spark_session = connect_with_spark_session_builder(
    app_name="SnowparkConnectLocalApp",
    config={
        "spark.jars.packages": "net.snowflake:snowflake-jdbc:3.13.29,net.snowflake:spark-snowflake_2.12:2.11.0-spark_3.4",
        "spark.jars.repositories": "https://repo1.maven.org/maven2"
    }
)

# Use the Spark session to create a Snowpark session
session = spark_session.getOrCreateSnowparkSession()

# Example: Create a Snowpark DataFrame and show its content
schema = StructType([
    StructField("name", StringType()),
    StructField("age", IntegerType())
])
data = [("Alice", 30), ("Bob", 25)]
df = session.create_dataframe(data, schema=schema)
df.show()

session.close()
spark_session.stop()

view raw JSON →