PySpark Bindings for H3

Common errors

• TypeError: Invalid argument, not a string or column: DataFrame[lat: double, lng: double] of type <class 'pyspark.sql.dataframe.DataFrame'>

  cause You are passing an entire DataFrame or a literal of the wrong type to an H3 function that expects a Spark Column expression.
  fix When calling H3 functions (e.g., `geo_to_h3`, `index_shape`), ensure you pass Spark Column objects (e.g., `F.col('column_name')`) or appropriate literals, not entire DataFrames or Python native types directly if the function expects a Column.

• Py4JJavaError: An error occurred while calling o.apache.spark.sql.functions.udf

  cause This generic PySpark error often indicates an issue in the underlying Java/Scala code when a UDF (User Defined Function) from h3-pyspark encounters invalid input or an unhandled condition.
  fix Check the full stack trace for more specific details. Common causes include invalid H3 resolutions (e.g., negative, too high), malformed H3 indices, or corrupted/unexpected data types in the input columns. Ensure input data conforms to H3 requirements and has no unexpected nulls or invalid values.

• AnalysisException: 'No such struct field <field_name> in <schema_string>'

  cause You are trying to access a column that does not exist in your Spark DataFrame, often due to a typo or incorrect case in the column name.
  fix Verify the exact names and casing of columns in your DataFrame (e.g., `df.printSchema()`) and ensure they match the column names used in `h3_pyspark` function calls (e.g., `h3_pyspark.geo_to_h3('latitude_column', 'longitude_column', 'res_column')`).

Warnings

• breaking The underlying `h3-py` library (which `h3-pyspark` wraps) introduced significant breaking changes in its 4.x versions, primarily around function naming conventions (e.g., `kRing` became `gridDisk`) and error handling.
  Fix: Refer to the `h3-py` migration guide for changes between H3 v3.x and v4.x. Adapt your code to the new function names and error handling. Verify the version of `h3-py` installed alongside `h3-pyspark` to ensure compatibility.
• gotcha Prior to version 1.2.4, `h3-pyspark` functions might not robustly handle null values in input columns to UDFs, potentially leading to errors or unexpected behavior.
  Fix: Upgrade to `h3-pyspark` version 1.2.4 or newer to benefit from improved null value handling. Ensure your input data is clean, or explicitly handle nulls (e.g., `na.drop()`, `fillna()`) before passing to H3 functions. [cite: `1.2.4` release notes]
• gotcha The `index_shape` function in versions prior to 1.2.3 had a known bug where it might miss H3 cells for long line segments, leading to incomplete or inaccurate spatial indexing for complex geometries.
  Fix: Upgrade to `h3-pyspark` version 1.2.3 or newer, which includes a fix for this bug and improved error handling for malformed geometries. [cite: `1.2.3` release notes]
• gotcha h3-pyspark assumes that geospatial geometries are represented as GeoJSON strings within a Spark DataFrame column, rather than other formats like WKT.
  Fix: Ensure that your geometry data is formatted as GeoJSON strings before passing it to functions like `h3_pyspark.index_shape`. Convert from other formats if necessary.

Install

• pip install h3-pyspark PyPI
• conda install -c conda-forge h3-pyspark Conda

Imports

• h3_pyspark

  import h3_pyspark

  The primary way to import the library and access its functions.
• geo_to_h3
  wrong:

  from h3_pyspark.functions import geo_to_h3

  correct:

  df = df.withColumn('h3_9', h3_pyspark.geo_to_h3('lat', 'lng', 'resolution'))

  Functions are typically accessed directly via the imported `h3_pyspark` module, not from a `functions` submodule.

Quickstart

This quickstart demonstrates how to initialize a SparkSession, create a DataFrame with geospatial coordinates, and use `h3_pyspark.geo_to_h3` to convert latitude and longitude to an H3 index. It also includes an example of `h3_pyspark.index_shape` for indexing GeoJSON polygons. Ensure `pyspark` is configured correctly for your environment.

from pyspark.sql import SparkSession, functions as F
import h3_pyspark
import os

# Initialize Spark Session (adjust master for your environment, e.g., 'local[*]'):
spark = SparkSession.builder.master(os.environ.get('SPARK_MASTER', 'local[*]')).appName("H3PySparkQuickstart").getOrCreate()

# Create a DataFrame with latitude, longitude, and desired H3 resolution
data = [{"lat": 37.769377, "lng": -122.388903, 'resolution': 9}]
df = spark.createDataFrame(data)

# Convert geographic coordinates to H3 index
df_with_h3 = df.withColumn('h3_index', h3_pyspark.geo_to_h3(F.col('lat'), F.col('lng'), F.col('resolution')))

df_with_h3.show()

# Example of an extension function: index_shape for GeoJSON polygons
geojson_polygon = "{\"type\":\"Polygon\",\"coordinates\":[[[-122.4,37.8],[-122.3,37.8],[-122.3,37.7],[-122.4,37.7],[-122.4,37.8]]]}"
polygon_df = spark.createDataFrame([{'id': 1, 'geometry': geojson_polygon, 'resolution': 9}])

polygon_h3_df = polygon_df.withColumn(
    'h3_cells',
    h3_pyspark.index_shape(F.col('geometry'), F.col('resolution'))
)

polygon_h3_df.show(truncate=False)

spark.stop()