Hadoop YARN API Client

Warnings

• breaking Python 2.7 support was officially dropped in version 1.0.3. Code written for Python 2.7 will likely fail with syntax errors or missing features.
  Fix: Migrate your codebase to Python 3.6+ to use versions 1.0.3 and later. If Python 2.7 is strictly required, pin the library version to <1.0.3.
• breaking Version 1.0.0 introduced a major API cleanup. The `ResourceManager`, `ApplicationMaster`, `HistoryServer`, and `NodeManager` constructors no longer accept separate `address` and `port` parameters. Instead, they require complete endpoint URLs (e.g., `['http://localhost:8088']`). `ResourceManager` also now accepts a list of endpoints for HA support.
  Fix: Update constructor calls to pass full endpoint URLs as a list (even for a single endpoint). For example, `ResourceManager('localhost', 8088)` becomes `ResourceManager(['http://localhost:8088'])`.
• gotcha When using YARN in High Availability (HA) mode, ensure you provide a list of all active ResourceManager endpoints to the `ResourceManager` constructor. The client will attempt to connect to the active RM from the provided list.
  Fix: Pass a list of all ResourceManager URLs: `ResourceManager(['http://rm1.example.com:8088', 'http://rm2.example.com:8088'])`.
• gotcha The library can automatically discover Hadoop configuration by checking `YARN_CONF_DIR` or `HADOOP_CONF_DIR` environment variables. If these are set, explicit endpoints provided in the constructor might be overridden or interact unexpectedly with discovered configurations.
  Fix: Be mindful of these environment variables. If you wish to explicitly control endpoints, ensure these variables are not set or that your explicit endpoint configuration takes precedence as expected. Consult the official documentation for precedence rules.
• gotcha Older YARN deployments or certain API calls might return empty JSON responses, which could cause parsing errors in the client. Version 1.0.2 improved handling of such cases.
  Fix: Upgrade to version 1.0.2 or later to benefit from improved robustness when handling empty or malformed YARN responses. Implement robust error handling and check for empty content in your application logic.

Install

• pip install yarn-api-client Install stable version

Imports

• ResourceManager

  from yarn_api_client.resource_manager import ResourceManager

• ApplicationMaster

  from yarn_api_client.application_master import ApplicationMaster

• HistoryServer

  from yarn_api_client.history_server import HistoryServer

• NodeManager

  from yarn_api_client.node_manager import NodeManager

Quickstart

This quickstart demonstrates how to initialize the `ResourceManager` and fetch a list of applications currently running or finished on the YARN cluster. It highlights configuring YARN endpoints and includes basic error handling for connection issues.

import os
from yarn_api_client.resource_manager import ResourceManager

# Configure YARN ResourceManager endpoints
# These can also be discovered automatically if YARN_CONF_DIR or HADOOP_CONF_DIR
# environment variables are set and point to valid Hadoop configuration.
# For local testing, ensure a YARN ResourceManager is running or adjust endpoint.

# Using a dummy endpoint if not set, for demonstration purposes.
# In a real scenario, replace with your actual YARN ResourceManager URL(s).
# Example for HA: ['http://rm1.example.com:8088', 'http://rm2.example.com:8088']
rm_endpoints = os.environ.get('YARN_RM_ENDPOINTS', 'http://localhost:8088').split(',')

if not rm_endpoints or rm_endpoints == ['']:
    print("Warning: YARN_RM_ENDPOINTS environment variable not set. Using 'http://localhost:8088' as default.")
    rm_endpoints = ['http://localhost:8088']

print(f"Attempting to connect to YARN ResourceManager at: {rm_endpoints}")

try:
    resource_manager = ResourceManager(rm_endpoints)

    # Fetch cluster applications
    applications_response = resource_manager.cluster_applications()
    
    if applications_response.apps:
        print(f"Found {len(applications_response.apps)} applications.")
        for app in applications_response.apps[:3]: # Print first 3 apps
            print(f"  Application ID: {app.id}, Name: {app.name}, State: {app.state}")
    else:
        print("No applications found on the YARN cluster.")

except Exception as e:
    print(f"Error connecting to YARN ResourceManager or fetching applications: {e}")
    print("Please ensure a YARN ResourceManager is running and accessible at the configured endpoint(s).")
    print("You can set the YARN_RM_ENDPOINTS environment variable, e.g., export YARN_RM_ENDPOINTS='http://your-rm-host:8088'")