{"library":"pyspark-client","title":"PySpark Connect Client","description":"The `pyspark-client` is the Python Spark Connect client for Apache Spark, providing a decoupled client-server architecture that enables remote connectivity to Spark clusters using the DataFrame API. It uses gRPC and Apache Arrow for efficient communication. The library is part of the broader Apache Spark project and is actively developed, with releases typically aligning with Apache Spark's minor and major version updates. The current version is 4.1.1, supporting Spark 4.1.1.","language":"python","status":"active","last_verified":"Sun May 17","install":{"commands":["pip install pyspark-client","pip install pyspark[connect]"],"cli":null},"imports":["from pyspark.sql import SparkSession"],"auth":{"required":false,"env_vars":[]},"quickstart":{"code":"import os\nfrom pyspark.sql import SparkSession\nfrom pyspark.sql.functions import lit\n\n# Ensure a Spark Connect server is running, e.g., via ./sbin/start-connect-server.sh\n# The default address is sc://localhost:15002\n\n# Connect to the Spark Connect server\nspark = SparkSession.builder.remote(os.environ.get('SPARK_CONNECT_SERVER_URL', 'sc://localhost:15002')).getOrCreate()\n\n# Create a DataFrame\ndf = spark.range(10).withColumn(\"hello\", lit(\"world\"))\n\n# Show the DataFrame\ndf.show()\n\n# Perform a simple operation\nresult = df.filter(df.id > 5).count()\nprint(f\"Count of rows with id > 5: {result}\")\n\nspark.stop()","lang":"python","description":"This quickstart demonstrates how to establish a connection to a Spark Connect server and perform basic DataFrame operations. It assumes a Spark Connect server is already running and accessible at the specified URL (defaulting to `sc://localhost:15002`). The `SPARK_CONNECT_SERVER_URL` environment variable can be used to override the connection string.","tag":null,"tag_description":null,"last_tested":null,"results":[]},"compatibility":{"tag":null,"tag_description":null,"last_tested":"2026-05-17","installed_version":"4.0.2","pypi_latest":"4.1.1","is_stale":true,"summary":{"python_range":"3.10–3.9","success_rate":100,"avg_install_s":24.2,"avg_import_s":0.56,"wheel_type":"sdist"},"results":[{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":"pyspark-client","exit_code":0,"wheel_type":"sdist","failure_reason":null,"import_side_effects":"clean","install_time_s":null,"import_time_s":0.29,"mem_mb":10.6,"disk_size":"399.0M"},{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":"connect","exit_code":0,"wheel_type":"sdist","failure_reason":null,"import_side_effects":"clean","install_time_s":null,"import_time_s":0.71,"mem_mb":17.8,"disk_size":"1.8G"},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":"pyspark-client","exit_code":0,"wheel_type":"sdist","failure_reason":null,"import_side_effects":"clean","install_time_s":13.4,"import_time_s":0.2,"mem_mb":10.6,"disk_size":"374M"},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":"connect","exit_code":0,"wheel_type":"sdist","failure_reason":null,"import_side_effects":"clean","install_time_s":40,"import_time_s":0.55,"mem_mb":17.8,"disk_size":"1.9G"},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":"pyspark-client","exit_code":0,"wheel_type":"sdist","failure_reason":null,"import_side_effects":"clean","install_time_s":null,"import_time_s":0.5,"mem_mb":11.7,"disk_size":"418.6M"},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":"connect","exit_code":0,"wheel_type":"sdist","failure_reason":null,"import_side_effects":"clean","install_time_s":0.1,"import_time_s":1.28,"mem_mb":19.2,"disk_size":"1.9G"},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":"pyspark-client","exit_code":0,"wheel_type":"sdist","failure_reason":null,"import_side_effects":"clean","install_time_s":12.3,"import_time_s":0.43,"mem_mb":11.7,"disk_size":"393M"},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":"connect","exit_code":0,"wheel_type":"sdist","failure_reason":null,"import_side_effects":"clean","install_time_s":38.5,"import_time_s":0.86,"mem_mb":19.2,"disk_size":"1.9G"},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":"pyspark-client","exit_code":0,"wheel_type":"sdist","failure_reason":null,"import_side_effects":"clean","install_time_s":null,"import_time_s":0.39,"mem_mb":11.5,"disk_size":"400.6M"},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":"connect","exit_code":0,"wheel_type":"sdist","failure_reason":null,"import_side_effects":"clean","install_time_s":null,"import_time_s":0.85,"mem_mb":19.2,"disk_size":"1.8G"},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":"pyspark-client","exit_code":0,"wheel_type":"sdist","failure_reason":null,"import_side_effects":"clean","install_time_s":13.2,"import_time_s":0.42,"mem_mb":11.5,"disk_size":"375M"},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":"connect","exit_code":0,"wheel_type":"sdist","failure_reason":null,"import_side_effects":"clean","install_time_s":38.4,"import_time_s":0.9,"mem_mb":19.2,"disk_size":"1.9G"},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":"pyspark-client","exit_code":0,"wheel_type":"sdist","failure_reason":null,"import_side_effects":"clean","install_time_s":null,"import_time_s":0.38,"mem_mb":11.4,"disk_size":"399.2M"},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":"connect","exit_code":0,"wheel_type":"sdist","failure_reason":null,"import_side_effects":"clean","install_time_s":null,"import_time_s":0.66,"mem_mb":19.1,"disk_size":"1.8G"},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":"pyspark-client","exit_code":0,"wheel_type":"sdist","failure_reason":null,"import_side_effects":"clean","install_time_s":13.1,"import_time_s":0.39,"mem_mb":11.4,"disk_size":"373M"},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":"connect","exit_code":0,"wheel_type":"sdist","failure_reason":null,"import_side_effects":"clean","install_time_s":38.9,"import_time_s":0.76,"mem_mb":19.1,"disk_size":"1.9G"},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":"pyspark-client","exit_code":0,"wheel_type":"sdist","failure_reason":null,"import_side_effects":"clean","install_time_s":null,"import_time_s":0.28,"mem_mb":10.1,"disk_size":"370.3M"},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":"connect","exit_code":0,"wheel_type":"sdist","failure_reason":null,"import_side_effects":"clean","install_time_s":null,"import_time_s":0.58,"mem_mb":17.3,"disk_size":"1.7G"},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":"pyspark-client","exit_code":0,"wheel_type":"sdist","failure_reason":null,"import_side_effects":"clean","install_time_s":14.7,"import_time_s":0.25,"mem_mb":10.1,"disk_size":"345M"},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":"connect","exit_code":0,"wheel_type":"sdist","failure_reason":null,"import_side_effects":"clean","install_time_s":43.6,"import_time_s":0.51,"mem_mb":17.3,"disk_size":"1.8G"}]}}