GraphFrames: DataFrame-based Graphs

0.11.0 · active · verified Fri Apr 10

GraphFrames is a powerful library for graph processing built on Apache Spark DataFrames. It allows users to perform graph analytics, queries, and algorithms like PageRank, connected components, and shortest paths directly using Spark's DataFrame API. The project is actively maintained, with frequent releases. The official Python package on PyPI is `graphframes-py`, and its latest version is 0.11.0.

Warnings

breaking The official PyPI package name for GraphFrames changed from `graphframes` to `graphframes-py` starting with v0.9.0. The old `graphframes` package on PyPI is severely outdated (v0.6) and should not be used.
Fix: Always use `pip install graphframes-py`. Do not install `graphframes`.
breaking The Maven groupId for the GraphFrames JAR changed from `graphframes` to `io.graphframes` in v0.9.0. This affects how you specify the package with `--packages` in `spark-submit` or in `SparkSession` configurations.
Fix: Update your Spark configurations to use `org.graphframes:graphframes:...` instead of `graphframes:graphframes:...`.
gotcha GraphFrames is a Spark library and requires a running Spark cluster. Installing the Python package (`graphframes-py`) is not enough; you must also provide the GraphFrames JAR to your Spark runtime. This is typically done via `spark-submit --packages` or `SparkSession.builder.config("spark.jars.packages", ...)`.
Fix: Ensure your Spark environment is configured to include the GraphFrames JAR. Refer to the 'install' and 'quickstart' examples for correct setup.
gotcha GraphFrames has strict compatibility requirements with specific versions of Spark and Scala. Using an incompatible GraphFrames JAR version with your Spark distribution will lead to runtime errors.
Fix: Always check the official GraphFrames documentation for the correct JAR artifact name (e.g., `org.graphframes:graphframes:0.11.0-spark3.5-s_2.12`) that matches your Spark and Scala versions.
breaking Significant API updates occurred in v0.9.0, including changes to the Pregel API and internal implementations of algorithms like Connected Components (CC), Community Detection using Label Propagation (CDLP), and Shortest Paths (SP). Some GraphX-free implementations were introduced.
Fix: Review the official GraphFrames documentation for v0.9.0+ when upgrading from older versions, especially if using advanced algorithms or custom Pregel implementations.

Install

pip install graphframes-py Install Python package
spark-submit --packages org.graphframes:graphframes:0.11.0-spark3.5-s_2.12 your_app.py Run with Spark (example for Spark 3.5, Scala 2.12)

Imports

GraphFrame
```
from graphframes import GraphFrame
```
GraphFrames is an external library, not part of core PySpark.

Quickstart

This quickstart demonstrates how to initialize a SparkSession with the GraphFrames package, create vertex and edge DataFrames, construct a GraphFrame, and run a PageRank algorithm. Ensure the `--packages` option uses the correct GraphFrames version for your Spark and Scala environment.

from pyspark.sql import SparkSession
from pyspark.sql.functions import lit
from graphframes import GraphFrame

# Create a SparkSession with GraphFrames package
# IMPORTANT: Replace '0.11.0-spark3.5-s_2.12' with the version compatible
# with your Spark and Scala installation. Check GraphFrames docs for details.
spark = SparkSession.builder \
    .appName("GraphFrames Quickstart") \
    .config("spark.jars.packages", "org.graphframes:graphframes:0.11.0-spark3.5-s_2.12") \
    .getOrCreate()

# Create a Vertex DataFrame
v = spark.createDataFrame([
  ("a", "Alice", 34),
  ("b", "Bob", 36),
  ("c", "Charlie", 30),
  ("d", "David", 29),
  ("e", "Esther", 32),
  ("f", "Fanny", 36)
], ["id", "name", "age"])

# Create an Edge DataFrame
e = spark.createDataFrame([
  ("a", "b", "friend"),
  ("b", "c", "follow"),
  ("c", "b", "follow"),
  ("f", "c", "follow"),
  ("e", "f", "follow"),
  ("e", "d", "friend"),
  ("d", "a", "friend")
], ["src", "dst", "relationship"])

# Create a GraphFrame
g = GraphFrame(v, e)

# Run PageRank algorithm
results = g.pagerank(resetProbability=0.15, maxIter=5)
results.vertices.select("id", "pagerank").show()

spark.stop()

view raw JSON →