ceja

0.4.0 · maintenance · verified Thu Apr 16

ceja is a Python library that provides PySpark implementations of string and phonetic matching algorithms. It enables users to apply functions like NYSIIS, Metaphone, Jaro-Winkler similarity, and Damerau-Levenshtein distance directly within PySpark DataFrames, leveraging Spark's distributed processing capabilities for large datasets. The library is currently at version 0.4.0, with its last release in February 2023, indicating a slow release cadence.

Common errors

```
ModuleNotFoundError: No module named 'pyspark'
```
cause The PySpark library is not installed or not correctly configured in your Python environment. ceja relies entirely on PySpark for its functionality.

fix Install PySpark using `pip install pyspark` and ensure your environment variables (like `SPARK_HOME`) are correctly set if running locally.
```
AttributeError: 'str' object has no attribute '_jc_op_eq'
```
cause You are attempting to pass a literal Python string to a ceja function which expects a Spark Column object.

fix Wrap your string literals with `sf.lit()` or ensure you are passing `sf.col("your_column")` to ceja functions.
```
TypeError: Column is not iterable
```
cause A ceja function received an incorrect type, likely a Spark Column object where a Python iterable or a different Column operation was expected, or vice-versa.

fix Verify the expected input types for the specific ceja function you are using. Ensure you are applying the function to the correct Spark DataFrame columns.

Warnings

gotcha ceja functions are designed for PySpark DataFrames. Attempting to use them directly on native Python strings or non-Spark data structures will result in runtime errors like 'AttributeError' or 'TypeError'.
Fix: Always pass Spark DataFrame columns (e.g., `sf.col("column_name")`) to ceja functions after importing `pyspark.sql.functions as sf`.
gotcha The library's last update was in February 2023. While generally stable, this slow release cadence might lead to compatibility issues with very recent versions of Python or PySpark, or a slower response to new feature requests/bug fixes.
Fix: Test thoroughly with your specific PySpark and Python environment. Consider pinning versions if stability is critical.
gotcha The project lacks extensive official documentation beyond the GitHub README and has an empty project description on PyPI, which can make advanced usage or troubleshooting more challenging.
Fix: Refer to the GitHub repository's README for available functions and basic usage patterns. Inspect the source code if deeper understanding is required.

Install

pip install ceja Install latest version

Imports

ceja
```
import ceja
```
All functions are exposed directly under the 'ceja' module namespace after a simple import.

Quickstart

This quickstart demonstrates how to initialize a SparkSession, create a DataFrame, and apply a ceja string matching function (damerau_levenshtein_distance) to columns within the DataFrame. Ensure findspark is installed and initialized if you're not running directly on a Spark cluster.

import findspark
findspark.init() # Initialize findspark if not running in a native Spark environment
from pyspark.sql import SparkSession
import pyspark.sql.functions as sf
import ceja

# Create a SparkSession
spark = SparkSession.builder.appName("CejaQuickstart").getOrCreate()

# Sample data
data = [ ("jellyfish", "smellyfish"), ("li", "lee"), ("luisa", "bruna"), (None, None) ]
df = spark.createDataFrame(data, ["word1", "word2"])

# Apply a ceja function (e.g., damerau_levenshtein_distance)
result_df = df.withColumn(
    "damerau_levenshtein_distance",
    ceja.damerau_levenshtein_distance(sf.col("word1"), sf.col("word2"))
)

result_df.show()

# Stop SparkSession
spark.stop()

view raw JSON →