SparkMeasure Python API

0.27.0 · active · verified Sun Apr 12

sparkMeasure is a Python API for the core Scala library, designed for performance troubleshooting of Apache Spark workloads. It simplifies the collection and analysis of Spark metrics, making it suitable for interactive analysis, testing, and production monitoring. The library focuses on easing metric collection and analysis for both developers and data engineers. Releases are frequent, typically on a quarterly to half-yearly cadence, with the current stable version being 0.27.0.

Warnings

breaking The configuration properties for the Kafka sink have changed in version 0.27. Kafka Producer properties must now be passed via Spark config using the prefix `spark.sparkmeasure.kafka.<kafkaProperty>=<value>`.
Fix: Update Spark configuration to use the new prefix for Kafka Producer properties (e.g., `spark.sparkmeasure.kafka.bootstrap.servers`).
gotcha Spark Connect integration is partial. While Flight Recorder mode can capture application-wide metrics, per-client (Spark Connect client-side) metrics are not fully reported. Direct access to SparkContext and its listener interface is required for full integration.
Fix: Be aware of these limitations when using sparkMeasure with Spark Connect. Consider Flight Recorder mode for application-level metrics or alternative monitoring for client-specific details.
gotcha Metrics collected by sparkMeasure are buffered in the driver's memory. For very large workloads or extensive metric collection, this can become a bottleneck or lead to out-of-memory errors on the driver.
Fix: Monitor driver memory usage. For large-scale data, consider saving metrics to file or external sinks (Kafka, InfluxDB) using Flight Recorder mode, which directly writes collected metrics.
gotcha Collecting metrics using `TaskMetrics` (at the granularity of each task completion) incurs additional overhead compared to `StageMetrics` (aggregated by stage). Use `TaskMetrics` only when fine-grained data (e.g., for skew analysis) is strictly necessary.
Fix: Prefer `StageMetrics` for general performance overview and troubleshooting, and switch to `TaskMetrics` only when detailed task-level analysis is required.
gotcha sparkMeasure primarily collects metrics for *successfully* executed tasks. Resources consumed by failed tasks are generally not included in the reports.
Fix: When analyzing job failures, be aware that reported metrics might not fully account for all resource usage leading up to the failure. Complement with Spark UI event logs for full failure context.

Install

pip install sparkmeasure Install Python package
pyspark --packages ch.cern.sparkmeasure:spark-measure_2.13:0.27 Run PySpark with SparkMeasure JAR (Spark 3/4 with Scala 2.13)

Imports

StageMetrics
```
from sparkmeasure import StageMetrics
```
TaskMetrics
```
from sparkmeasure import TaskMetrics
```

Quickstart

This quickstart demonstrates how to initialize SparkMeasure and use the `runandmeasure` method to collect and report performance metrics for a Spark SQL query. It sets up a local SparkSession, loads the necessary SparkMeasure JAR, and then uses `StageMetrics` to instrument a simple operation.

from pyspark.sql import SparkSession
from sparkmeasure import StageMetrics

# Configure SparkSession to include the spark-measure JAR
spark = (SparkSession.builder
         .appName("SparkMeasure Quickstart")
         .master("local[*]")
         .config("spark.jars.packages", "ch.cern.sparkmeasure:spark-measure_2.13:0.27")
         .getOrCreate())

# Initialize StageMetrics
stagemetrics = StageMetrics(spark)

# Run and measure a Spark job
print("Running a simple Spark SQL query and measuring performance...")
stagemetrics.runandmeasure(globals(), 'spark.sql("SELECT count(*) FROM range(1000) CROSS JOIN range(1000)").show()')

print("\nPrinting performance report:")
stagemetrics.print_report()

spark.stop()

view raw JSON →