{"library":"sparkmeasure","title":"SparkMeasure Python API","type":"library","description":"sparkMeasure is a Python API for the core Scala library, designed for performance troubleshooting of Apache Spark workloads. It simplifies the collection and analysis of Spark metrics, making it suitable for interactive analysis, testing, and production monitoring. The library focuses on easing metric collection and analysis for both developers and data engineers. Releases are frequent, typically on a quarterly to half-yearly cadence, with the current stable version being 0.27.0.","language":"python","status":"active","last_verified":"Fri May 15","install":{"commands":["pip install sparkmeasure"],"cli":null},"imports":["from sparkmeasure import StageMetrics","from sparkmeasure import TaskMetrics"],"auth":{"required":false,"env_vars":[]},"links":{"homepage":null,"github":"https://github.com/lucacanali/sparkMeasure","docs":"https://github.com/lucacanali/sparkMeasure","changelog":"https://github.com/lucacanali/sparkMeasure/releases","pypi":"https://pypi.org/project/sparkmeasure/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null},"quickstart":{"code":"from pyspark.sql import SparkSession\nfrom sparkmeasure import StageMetrics\n\n# Configure SparkSession to include the spark-measure JAR\nspark = (SparkSession.builder\n         .appName(\"SparkMeasure Quickstart\")\n         .master(\"local[*]\")\n         .config(\"spark.jars.packages\", \"ch.cern.sparkmeasure:spark-measure_2.13:0.27\")\n         .getOrCreate())\n\n# Initialize StageMetrics\nstagemetrics = StageMetrics(spark)\n\n# Run and measure a Spark job\nprint(\"Running a simple Spark SQL query and measuring performance...\")\nstagemetrics.runandmeasure(globals(), 'spark.sql(\"SELECT count(*) FROM range(1000) CROSS JOIN range(1000)\").show()')\n\nprint(\"\\nPrinting performance report:\")\nstagemetrics.print_report()\n\nspark.stop()","lang":"python","description":"This quickstart demonstrates how to initialize SparkMeasure and use the `runandmeasure` method to collect and report performance metrics for a Spark SQL query. It sets up a local SparkSession, loads the necessary SparkMeasure JAR, and then uses `StageMetrics` to instrument a simple operation.","tag":null,"tag_description":null,"last_tested":null,"results":[]},"compatibility":{"tag":null,"tag_description":null,"last_tested":"2026-05-15","installed_version":"0.27.0","pypi_latest":"0.27.0","is_stale":false,"summary":{"python_range":"3.10–3.9","success_rate":100,"avg_install_s":1.7,"avg_import_s":0,"wheel_type":"wheel"},"results":[{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":"sparkmeasure","exit_code":0,"wheel_type":"wheel","failure_reason":null,"import_side_effects":"clean","install_time_s":null,"import_time_s":0,"mem_mb":0,"disk_size":"17.8M"},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":"sparkmeasure","exit_code":0,"wheel_type":"wheel","failure_reason":null,"import_side_effects":"clean","install_time_s":1.9,"import_time_s":0,"mem_mb":0,"disk_size":"18M"},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":"sparkmeasure","exit_code":0,"wheel_type":"wheel","failure_reason":null,"import_side_effects":"clean","install_time_s":null,"import_time_s":0,"mem_mb":0,"disk_size":"19.7M"},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":"sparkmeasure","exit_code":0,"wheel_type":"wheel","failure_reason":null,"import_side_effects":"clean","install_time_s":1.6,"import_time_s":0,"mem_mb":0,"disk_size":"20M"},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":"sparkmeasure","exit_code":0,"wheel_type":"wheel","failure_reason":null,"import_side_effects":"clean","install_time_s":null,"import_time_s":0,"mem_mb":0,"disk_size":"11.6M"},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":"sparkmeasure","exit_code":0,"wheel_type":"wheel","failure_reason":null,"import_side_effects":"clean","install_time_s":1.6,"import_time_s":0,"mem_mb":0,"disk_size":"12M"},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":"sparkmeasure","exit_code":0,"wheel_type":"wheel","failure_reason":null,"import_side_effects":"clean","install_time_s":null,"import_time_s":0,"mem_mb":0.2,"disk_size":"11.3M"},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":"sparkmeasure","exit_code":0,"wheel_type":"wheel","failure_reason":null,"import_side_effects":"clean","install_time_s":1.4,"import_time_s":0,"mem_mb":0,"disk_size":"12M"},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":"sparkmeasure","exit_code":0,"wheel_type":"wheel","failure_reason":null,"import_side_effects":"clean","install_time_s":null,"import_time_s":0,"mem_mb":0,"disk_size":"17.3M"},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":"sparkmeasure","exit_code":0,"wheel_type":"wheel","failure_reason":null,"import_side_effects":"clean","install_time_s":1.8,"import_time_s":0,"mem_mb":0,"disk_size":"18M"}]}}