Spark NLP

Warnings

• breaking Spark NLP has strict compatibility requirements with Apache Spark and Scala versions. Mismatches can lead to `ClassNotFoundException`, `NoSuchMethodError`, or other runtime errors.
  Fix: Ensure your `pyspark` and underlying Spark cluster's Scala version (e.g., 2.12 or 2.13) are compatible with the Spark NLP version. Use `sparknlp.start(spark_version='X.Y', scala_version='Z.W')` or consult the official Spark NLP compatibility matrix (https://nlp.johnsnowlabs.com/docs/en/install#compatibility-matrix) for precise version pairings. For example, Spark NLP 6.x generally works with Spark 3.x-4.x.
• gotcha Spark NLP operations, particularly model training or processing large documents/datasets, can be memory-intensive due to its reliance on the JVM. Default Spark/JVM memory settings may be insufficient, leading to `OutOfMemoryError`.
  Fix: Increase Spark driver and executor memory. You can set this when starting the session: `spark = sparknlp.start(spark_memory='16g', spark_driver_maxResultSize='8g')` or configure `spark.driver.memory`, `spark.executor.memory`, and `spark.driver.maxResultSize` directly in your Spark configuration.
• gotcha Choosing between `LightPipeline` and the full `Pipeline` for inference. `LightPipeline` is optimized for fast, single-record processing (Python-based), while `Pipeline` is for large-scale, distributed batch processing within Spark. Misusing them can lead to performance bottlenecks.
  Fix: Use `LightPipeline` for individual text strings or small lists (e.g., API endpoints, quick demos). Use the full `Pipeline` (fitting and transforming a Spark DataFrame) for production-scale batch processing, leveraging Spark's distribution capabilities.
• deprecated Manually configuring SparkSession to include Spark NLP JARs using `spark.jars.packages` is largely superseded by `sparknlp.start()`, which automates this process and handles versioning and compatibility.
  Fix: Prefer `sparknlp.start()` for initializing your SparkSession, as it simplifies dependency management and configuration. If manual configuration is necessary (e.g., specific cluster setups), ensure `spark.jars.packages` correctly specifies the Spark NLP version matching your environment.

Install

• pip install spark-nlp pyspark Install core Spark NLP and PySpark
• pip install spark-nlp[tensorflow] Install with TensorFlow dependencies (e.g., for some models)
• pip install spark-nlp[databricks] Install for Databricks environments

Imports

• sparknlp

  import sparknlp

• DocumentAssembler

  from sparknlp.base import DocumentAssembler

• Tokenizer

  from sparknlp.annotator import Tokenizer

• WordEmbeddingsModel

  from sparknlp.annotator import WordEmbeddingsModel

• Pipeline

  from pyspark.ml import Pipeline

• SparkSession

  from pyspark.sql import SparkSession

  While SparkSession is often instantiated using sparknlp.start(), the class itself is from pyspark.sql. When configuring manually, import from pyspark.sql.
• LightPipeline

  from sparknlp.base import LightPipeline

Quickstart

This quickstart demonstrates how to initialize Spark NLP, define a basic NLP pipeline with document assembly, tokenization, and pre-trained word embeddings, and process a Spark DataFrame. It also shows how to use `LightPipeline` for faster inference on single inputs. The `sparknlp.start()` function simplifies Spark session setup, ensuring correct JAR and configuration loading.

import sparknlp
from sparknlp.base import DocumentAssembler, Pipeline, LightPipeline
from sparknlp.annotator import Tokenizer, WordEmbeddingsModel

# 1. Initialize SparkSession with Spark NLP
# Automatically handles Spark NLP JAR dependencies and configures Spark
# Adjust spark_version, spark_memory, and scala_version as needed for your environment
spark = sparknlp.start(spark_version='3.4', spark_memory='16g', scala_version='2.12')

# 2. Define a simple Spark NLP pipeline
document_assembler = DocumentAssembler().setInputCol("text").setOutputCol("document")
tokenizer = Tokenizer().setInputCols(["document"]).setOutputCol("token")

# Load a pre-trained Word Embeddings model
# (glove_100d is a small, general-purpose model suitable for quickstarts)
word_embeddings = WordEmbeddingsModel.pretrained("glove_100d", "en")\
    .setInputCols(["document", "token"])\
    .setOutputCol("embeddings")

nlp_pipeline = Pipeline(stages=[
    document_assembler,
    tokenizer,
    word_embeddings
])

# 3. Create a Spark DataFrame and process it
data = spark.createDataFrame([["Spark NLP is a powerful library for natural language processing on Apache Spark."]]).toDF("text")

# Fit the pipeline to the data (this often involves downloading models if not cached)
pipeline_model = nlp_pipeline.fit(data)
result = pipeline_model.transform(data)

# 4. Show results
print("\nPipeline Result:")
result.select("token.result", "embeddings.result").show(truncate=False)

# Example of LightPipeline for single-record inference
light_pipeline = LightPipeline(pipeline_model)
light_result = light_pipeline.annotate("Spark NLP makes NLP scalable and easy.")
print("\nLightPipeline Result (tokens):", light_result['token'])

# Don't forget to stop the SparkSession when done
spark.stop()