AWS Glue Development Library (awsglue-dev)

2021.12.30 · active · verified Wed Apr 15

awsglue-dev provides Python interfaces to the AWS Glue ETL library, primarily for local development, IDE auto-completion, and local script validation. It extends Apache Spark with additional data types and operations for ETL workflows. The package version `2021.12.30` is part of an ecosystem that facilitates authoring scripts for AWS Glue, a fully managed, serverless ETL service.

Warnings

breaking The `awsglue-dev` package primarily offers Python interfaces for local development (e.g., IDE auto-completion, static analysis). Actual AWS Glue ETL scripts built with these interfaces *must be executed within the AWS Glue service* or a compatible local Docker environment that includes the Glue Spark runtime JARs.
Fix: Do not expect scripts using `awsglue-dev` to run fully functional locally without a full AWS Glue runtime setup (e.g., official AWS Glue Docker images) or deployment to the AWS Glue service. Use it for development and testing logic, not full local execution.
gotcha For local development with `awsglue-dev`, the `pyspark` library is a mandatory peer dependency and must be installed separately. Without it, core components like `SparkContext` and `GlueContext` will not function.
Fix: Ensure `pip install pyspark` is run in your development environment alongside `pip install awsglue-dev`.
gotcha AWS Glue job scripts, by default, may not automatically partition output data when writing to target data sources. This can lead to poor performance on large datasets.
Fix: Explicitly implement partitioning in your ETL job scripts, especially when writing `DynamicFrame` or `DataFrame` outputs, to leverage performance optimizations like partition pruning. Refer to AWS Glue documentation for `repartition` or `write` options.
gotcha AWS Glue environments (which `awsglue-dev` mirrors) often ship with a set of pre-installed Python packages, some of which may contain known vulnerabilities or be outdated. Relying solely on these default versions can pose security risks.
Fix: For production Glue jobs, explicitly manage dependencies using `requirements.txt` and ensure packages are updated to secure versions. Regularly audit your dependencies using security scanning tools.
breaking Migrating AWS Glue jobs between major Glue versions (e.g., from Glue 2.0/3.0 to 4.0/5.0) can introduce breaking changes due to underlying Spark version upgrades, changes in supported Python versions, or deprecation of certain libraries/APIs. Scripts developed with `awsglue-dev` might need adjustments.
Fix: Consult the official AWS Glue migration guides for each specific version upgrade. Test your Glue scripts thoroughly against the target Glue version in a development environment before deploying to production. Pay attention to Spark and Python compatibility.

Install

pip install awsglue-dev Install core library
pip install pyspark Install required dependency for local execution

Imports

GlueContext
```
from awsglue.context import GlueContext
```

SparkContext

from pyspark.context import SparkContext

Job
```
from awsglue.job import Job
```

getResolvedOptions

from awsglue.utils import getResolvedOptions

DynamicFrame

from awsglue.dynamicframe import DynamicFrame

* (transforms)
```
from awsglue.transforms import *
```
Commonly, individual transform classes are imported directly or using `*` for convenience in Glue scripts, not the parent module.

Quickstart

This quickstart demonstrates the foundational boilerplate for an AWS Glue ETL script, initializing the SparkContext, GlueContext, and Job objects. While `awsglue-dev` provides the interfaces locally, actual ETL execution often requires a Glue environment (e.g., Docker container or AWS Glue service) to run successfully with real data sources. The `getResolvedOptions` function is used to handle job parameters, which are central to Glue job execution.

import sys
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from awsglue.utils import getResolvedOptions

# These parameters are typically passed by AWS Glue service
# For local development, you might set dummy values or omit if not testing getResolvedOptions
args = getResolvedOptions(sys.argv, ['JOB_NAME'])

sc = SparkContext()
gLueContext = GlueContext(sc)
spark = gLueContext.spark_session
job = Job(gLueContext)
job.init(args['JOB_NAME'], args)

# Your Glue ETL script logic would go here
# For example, to create a DynamicFrame:
# from awsglue.dynamicframe import DynamicFrame
# dynamic_frame = gLueContext.create_dynamic_frame.from_options(
#     connection_type='s3', 
#     connection_options={'paths': ['s3://your-bucket/your-data/'], 'recurse': True}, 
#     format='json'
# )

print(f"Initialized GlueContext and SparkSession for job: {args['JOB_NAME']}")
# Don't forget job.commit() in a real Glue job
# job.commit()

view raw JSON →