HdfsCLI: API and Command Line Interface for HDFS

Warnings

• breaking HdfsCLI version 2.x and above has dropped official support for Python 2.x. It is compatible with Python 3.7+.
  Fix: Upgrade your Python environment to 3.7 or newer. If you must use Python 2, you'll need to use an older version of the `hdfs` library (e.g., `hdfs<2.0.0`), but this is not recommended due to lack of maintenance.
• gotcha By default, `client.write()` will raise an `HdfsError` if trying to write to an existing path. To overwrite an existing file, you must explicitly set `overwrite=True`.
  Fix: When calling `client.write()`, include `overwrite=True` in the arguments if you intend to replace an existing file (e.g., `client.write(path, overwrite=True)`).
• gotcha Deleting a non-empty directory without `recursive=True` will raise an `HdfsError`. This is a safety mechanism.
  Fix: To delete a directory and its contents, use `client.delete(path, recursive=True)`. Consider `skip_trash=False` (requires Hadoop 2.9+) if you want files to go to trash instead of being permanently deleted.
• gotcha Using `Client.from_alias()` relies on a configuration file (default: `~/.hdfscli.cfg`) which defines cluster connection details. Without proper configuration, this method will fail.
  Fix: Ensure you have a `~/.hdfscli.cfg` file (or `HDFSCLI_CONFIG` environment variable pointing to one) with valid alias definitions, including `url` and optional `user` or `client` (e.g., `KerberosClient`).
• gotcha The `KerberosClient` requires the `hdfs[kerberos]` extra to be installed and proper Kerberos configuration on the client machine and HDFS cluster. Misconfiguration often leads to authentication errors.
  Fix: Install the kerberos extension (`pip install hdfs[kerberos]`). Ensure your `krb5.conf` is correctly configured and you have a valid Kerberos ticket. Refer to the HdfsCLI documentation for detailed Kerberos setup instructions.

Install

• pip install hdfs Core library
• pip install hdfs[avro,dataframe,kerberos] With optional extensions

Imports

• InsecureClient

  from hdfs.client import InsecureClient

  The default and simplest client for insecure HDFS clusters.
• Client

  from hdfs.client import Client

  The base client class, InsecureClient is a subclass. Often used via `Client.from_alias()`.
• TokenClient

  from hdfs.client import TokenClient

  Used for token-based authentication with HDFS.
• KerberosClient

  from hdfs.ext.kerberos import KerberosClient

  Used for Kerberos authenticated clusters, requires the 'kerberos' extension.

Quickstart

This quickstart demonstrates how to establish a connection to an HDFS Namenode using `InsecureClient`, write a simple file, list directory contents, read the file back, and then delete it. It uses environment variables for the Namenode URL and user for flexibility. Ensure your HDFS cluster is running and accessible at the specified URL.

import os
from hdfs.client import InsecureClient

HDFS_NAMENODE_URL = os.environ.get('HDFS_NAMENODE_URL', 'http://localhost:50070')
HDFS_USER = os.environ.get('HDFS_USER', 'guest') # Or a specific HDFS user

try:
    client = InsecureClient(HDFS_NAMENODE_URL, user=HDFS_USER)
    print(f"Connected to HDFS at {HDFS_NAMENODE_URL} as user {HDFS_USER}")

    # Example: Create a file
    hdfs_path = '/user/temp/my_test_file.txt'
    local_data = b'Hello, HdfsCLI world!'
    with client.write(hdfs_path, encoding='utf-8', overwrite=True) as writer:
        writer.write(local_data.decode('utf-8'))
    print(f"Successfully wrote to {hdfs_path}")

    # Example: List contents of a directory
    parent_dir = os.path.dirname(hdfs_path)
    if parent_dir == '': parent_dir = '/' # handle root edge case
    print(f"Contents of {parent_dir}:")
    for item in client.list(parent_dir):
        print(f"- {item}")
    
    # Example: Read the file back
    with client.read(hdfs_path, encoding='utf-8') as reader:
        read_data = reader.read()
    print(f"Read from {hdfs_path}: {read_data}")

    # Example: Delete the file
    client.delete(hdfs_path)
    print(f"Successfully deleted {hdfs_path}")

except Exception as e:
    print(f"An error occurred: {e}")
    print("Please ensure HDFS is running and HDFS_NAMENODE_URL/HDFS_USER are correctly configured.")