AWS Glue Local Development

Common errors

• ModuleNotFoundError: No module named 'awsglue.context'

  cause The Python environment does not have the `awsglue` module accessible on its `sys.path` or `awsglue3-local` failed to properly configure the environment.
  fix Ensure `awsglue3-local` is correctly installed. If running with PySpark, confirm that the Glue libraries (often `aws-glue-libs.jar`) are included in your Spark classpath. For `awsglue3-local`, this should ideally be handled, but manual intervention might be needed for complex setups.

• java.lang.ClassNotFoundException: org.apache.hadoop.fs.s3a.S3AFileSystem

  cause This typically indicates that the necessary Hadoop AWS S3 connector JARs are missing from your Spark classpath, which are required for interacting with S3 buckets from Spark/Glue.
  fix Ensure your local Spark environment (or the environment configured by `awsglue3-local`) includes the correct Hadoop-AWS JARs. For `pyspark` directly, this often involves `spark-submit --packages org.apache.hadoop:hadoop-aws:x.y.z ...`.

• Py4JJavaError: An error occurred while calling o72.getDynamicFrame.fromDF.

  cause This error often occurs when `DynamicFrame` operations are attempted without a fully initialized Glue context or when there are underlying Spark/JVM issues with the Glue extensions. It can also happen if the `awsglue` libraries are not properly linked.
  fix Verify that `GlueContext` is correctly initialized (`GlueContext(sc)`), and `job.init()` is called. Ensure your local environment is robust enough to handle the Glue-specific Spark extensions. Sometimes, restarting the Spark session helps.

Warnings

• gotcha The `awsglue` module is typically part of the AWS Glue runtime and is not fully pip-installable as a complete, standalone library providing all native Glue functionalities. `awsglue3-local` aims to provide the necessary environment and stub modules to allow standard Glue job scripts to run locally, but some features (e.g., direct S3/JDBC connectors without specific Hadoop/Spark configurations) might still require additional setup or behave differently.
  Fix: Be aware that local execution with `awsglue3-local` provides an approximation of the Glue runtime. For full fidelity, consider using official AWS Glue Docker images or `spark-submit` with `aws-glue-libs` on the classpath.
• breaking Local Glue development environments, including those set up with `awsglue3-local`, can exhibit behavioral differences compared to the actual AWS Glue cloud environment. These discrepancies can stem from differences in Spark configuration, underlying libraries, resource management, or specific Glue service integrations not fully replicated locally.
  Fix: Always perform final validation and testing on the actual AWS Glue environment. Ensure local testing focuses on logic and syntax, rather than subtle performance or integration nuances.
• gotcha When using `getResolvedOptions`, if job arguments are not provided (e.g., when running a script directly without emulating `spark-submit --conf 'spark.driver.args="--JOB_NAME myjob"'`), it will raise an error indicating required arguments are missing. This is a common pitfall in local development.
  Fix: For local testing, ensure you either pass dummy arguments (e.g., `sys.argv.extend(['--JOB_NAME', 'my_local_job'])`) or handle missing arguments gracefully in your script, e.g., by providing default values or checking for argument existence.

Install

• pip install awsglue3-local Install the package

Imports

• GlueContext
  wrong:

  from glue.context import GlueContext

  correct:

  from awsglue.context import GlueContext

  Standard import for GlueContext in a Glue environment.
• SparkSession

  from pyspark.sql import SparkSession

  Standard PySpark SparkSession import. awsglue3-local configures the environment.
• getResolvedOptions
  wrong:

  from glue.utils import getResolvedOptions

  correct:

  from awsglue.utils import getResolvedOptions

  Used to parse job arguments from the Glue environment.
• DynamicFrame

  from awsglue.dynamicframe import DynamicFrame

  Core Glue data structure for ETL operations.

Quickstart

A basic AWS Glue job script demonstrating the initialization of `GlueContext`, `SparkSession`, and parsing job arguments using `getResolvedOptions`. This code assumes `awsglue3-local` has correctly set up the environment for these imports to resolve. Note that full `DynamicFrame` functionality often requires additional Glue libraries and configurations, which `awsglue3-local` aims to facilitate but might require more complex setup for specific connectors.

import sys
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job

# This part mimics how Glue passes arguments
# In local development, you might set these via command line or hardcode them
# For quickstart, we use an empty dict if not provided.
args = getResolvedOptions(sys.argv, ['JOB_NAME'])

sc = SparkContext()
glucueContext = GlueContext(sc)
spark = glucueContext.spark_session
job = Job(glucueContext)
job.init(args['JOB_NAME'], args)

# Example: Create a simple Spark DataFrame
data = [("Alice", 1), ("Bob", 2)]
df = spark.createDataFrame(data, ["Name", "Id"])
df.show()

# Example: Use Glue DynamicFrame (requires more setup for actual data sources)
# try:
#     from awsglue.dynamicframe import DynamicFrame
#     # This part would typically involve reading from S3, JDBC, etc.
#     # For a truly local test, you might convert a Spark DataFrame to DynamicFrame
#     dynamic_frame = DynamicFrame.fromDF(df, glucueContext, "example_df")
#     dynamic_frame.printSchema()
# except ImportError:
#     print("awsglue.dynamicframe not fully functional in this minimal local setup without full Glue libs.")

print("Glue job finished locally.")

job.commit()