EMR Notebooks Magics

0.2.4 · active · verified Sat Apr 11

EMR Notebooks Magics is a Python library providing Jupyter magics specifically for Amazon EMR Notebooks. These magics enhance the interactive experience by allowing direct interaction with EMR cluster resources, such as mounting S3 workspaces and executing other notebooks. The library is currently at version 0.2.4 and is actively maintained by AWS Labs, with updates tied to EMR Notebooks and EMR Studio feature development.

Warnings

gotcha After installing `emr-notebooks-magics` (e.g., via `pip install`), you must restart the Jupyter kernel before the magics become available. Installing via bootstrap actions is not supported.
Fix: Restart the Jupyter kernel (`Kernel -> Restart Kernel`) after installation. Do not attempt to install it via EMR bootstrap actions as the notebook environment is installed later.
breaking The `%mount_workspace_dir` magic primarily functions with Python 3 kernels. Spark executors do not have access to the mounted directory when used with a Python 3 kernel.
Fix: Be aware of this limitation. If Spark executors need access to data, ensure data is directly accessible via S3 paths or other cluster-wide storage, rather than relying solely on a Python kernel's mounted workspace.
gotcha Enabling write access with `%mount_workspace_dir` is irreversible and applies changes directly to your S3 Workspace. By default, mounts are read-only.
Fix: Always enable S3 versioning on your S3 Workspace bucket as a precaution if you enable write access. Strongly consider keeping mounts read-only unless write access is absolutely necessary.
gotcha Using magics like `%execute_notebook` requires specific IAM permissions for the EMR-EC2 instance role. Lack of proper S3 read access for the EC2 instance profile will lead to failures.
Fix: Ensure that your EMR cluster's EC2 instance profile (e.g., `EMR_EC2_DefaultRole`) has `AmazonElasticMapReduceforEC2Role` managed policy or equivalent S3 read permissions. For `%execute_notebook`, additional permissions might be needed.
breaking EMR Serverless and Amazon EMR on EKS clusters do not support EMR Notebooks magics, including those from `emr-notebooks-magics`.
Fix: These magics are only compatible with traditional EMR clusters. For EMR Serverless, manage Spark configurations directly via the `SparkContext` object as a workaround for some functionalities.

Install

pip install emr-notebooks-magics Install latest version

Imports

%mount_workspace_dir
```
%mount_workspace_dir .
```
This library provides IPython magics, not traditional Python modules for direct import. Magics are used directly in Jupyter cells with a '%' prefix after installation and kernel restart.
%generate_s3_download_url
```
%generate_s3_download_url s3://my_bucket/path/to/object
```
Magics are invoked directly in Jupyter cells; they are not imported as standard Python functions.

Quickstart

This quickstart demonstrates the core functionalities of EMR Notebooks Magics: mounting a workspace directory, generating a presigned S3 download URL, and executing another notebook. These commands are run directly in a Jupyter cell within an EMR Notebook environment.

# After installing the package and restarting the kernel in an EMR Notebook.

# Mount the entire Workspace directory to the EMR cluster instance
%mount_workspace_dir .

# Generate a presigned URL for an S3 object
# Replace with your actual S3 bucket and object key
%generate_s3_download_url s3://my-example-bucket/path/to/my-file.csv

# Execute another notebook in the background
# Make sure 'another_notebook.ipynb' exists in your workspace
%execute_notebook another_notebook.ipynb

view raw JSON →