Azure Data Lake Store Filesystem Client Library

Warnings

• breaking This library is exclusively for **Azure Data Lake Storage Gen 1**. For Azure Data Lake Storage Gen 2, which is the current generation, you **must** use the `azure-storage-file-datalake` library. Using `azure-datalake-store` for Gen 2 will result in compatibility issues.
  Fix: For ADLS Gen 2, install `azure-storage-file-datalake` (`pip install azure-storage-file-datalake`) and migrate your code to use its APIs (e.g., `DataLakeServiceClient`).
• gotcha The official documentation states that this 'software is under active development and not yet recommended for general use'. This suggests it may not be suitable for critical production workloads or that its APIs could still undergo significant changes.
  Fix: Evaluate your use case carefully. For production systems, consider the stability implications or explore alternatives like ADLS Gen 2 with `azure-storage-file-datalake`.
• breaking Authentication mechanisms changed significantly from `0.x` to `1.0.x`. Older versions used ADAL and custom authentication. Version `1.0.0-alpha0` and `1.0.1` shifted to 'generic azure token credential for auth instead of custom lib.auth' and removed ADAL support, replacing it with MSAL internally within `lib.auth`.
  Fix: Update your authentication code. While `lib.auth` still exists, ensure your credentials (tenant ID, username, password) are compatible with the new underlying MSAL-based token acquisition or consider using `azure-identity` for direct token management.
• breaking In version `1.0.1`, the `concat` operation was removed from multi-part uploads, and large files are now uploaded in a single chunk. This changes the behavior for handling very large files.
  Fix: Review any code relying on multi-part upload concatenation. No direct fix is provided in the release notes, implying the library now handles large files differently internally, which might impact custom upload logic or performance expectations.
• deprecated All `0.0.x` versions were explicitly labeled as 'pre-release or preview version' with a warning that there 'will be fairly rapid development and bug fixing, which might result in breaking changes from release to release.' Upgrading directly from `0.0.x` to `1.0.x` will likely involve significant breaking changes.
  Fix: Avoid using `0.0.x` versions for new development. For existing `0.0.x` deployments, plan for a thorough migration and testing period when upgrading to `1.0.x` or later, as many aspects of the API and behavior may have changed.
• breaking Version `1.0.0-alpha0` and `1.0.1` removed support for older Python versions. The specific versions removed are not detailed in the release notes, but users on older Python environments should verify compatibility.
  Fix: Upgrade your Python environment to a currently supported version (e.g., Python 3.8+). Check the project's `setup.py` or latest documentation for explicit Python version requirements.

Install

• pip install azure-datalake-store Install stable version

Imports

• AzureDLFileSystem

  from azure.datalake.store import core
  adl_fs = core.AzureDLFileSystem(...)

  The common pattern shown in documentation is `import core` and then `core.AzureDLFileSystem`. Direct import from `azure.datalake.store.core` also works.
• lib.auth

  from azure.datalake.store import lib
  token = lib.auth(tenant_id, username, password)

  This method for obtaining a token is still documented, even though the internal authentication mechanism shifted to use generic Azure token credentials in v1.0.x. Consider using environment variables or `azure-identity` for modern auth.

Quickstart

This quickstart demonstrates how to authenticate and perform basic file operations (list, create directory, create file, write, read, delete) with Azure Data Lake Store Gen 1 using `AzureDLFileSystem`. It relies on environment variables (`AZURE_TENANT_ID`, `AZURE_USERNAME`, `AZURE_PASSWORD`, `AZURE_STORE_NAME`) for authentication, which is a common and recommended approach for service principals.

import os
from azure.datalake.store import core

# Set these environment variables for authentication
# Ensure AZURE_TENANT_ID, AZURE_USERNAME, AZURE_PASSWORD, AZURE_STORE_NAME are set
# For testing, use placeholder values if not connecting to a real ADLS Gen1

tenant_id = os.environ.get('AZURE_TENANT_ID', 'YOUR_TENANT_ID')
username = os.environ.get('AZURE_USERNAME', 'YOUR_USERNAME')
password = os.environ.get('AZURE_PASSWORD', 'YOUR_PASSWORD')
store_name = os.environ.get('AZURE_STORE_NAME', 'youradlstorename')

try:
    # Authenticate (lib.auth now uses generic Azure token credentials internally)
    token = core.lib.auth(tenant_id, username, password)
    
    # Initialize the Data Lake Store filesystem client
    adl = core.AzureDLFileSystem(store_name, token=token)
    
    # Example: List contents of the root directory
    print(f"Listing contents of / in {store_name}:")
    items = adl.ls('/', detail=True)
    if items:
        for item in items:
            print(item)
    else:
        print("Directory is empty or path does not exist.")

    # Example: Create a directory and a file
    test_dir = 'mytestdir'
    test_file = f'{test_dir}/testfile.txt'
    if not adl.exists(test_dir):
        adl.mkdir(test_dir)
        print(f"Created directory: {test_dir}")

    with adl.open(test_file, 'wb') as f:
        f.write(b"Hello from Azure Data Lake Store Gen1!")
    print(f"Created and wrote to file: {test_file}")

    # Example: Read the file
    with adl.open(test_file, 'rb') as f:
        content = f.read()
        print(f"Content of {test_file}: {content.decode('utf-8')}")

    # Example: Delete the file and directory
    adl.rm(test_file)
    print(f"Deleted file: {test_file}")
    adl.rmdir(test_dir)
    print(f"Deleted directory: {test_dir}")

except Exception as e:
    print(f"An error occurred: {e}")
    print("Please ensure environment variables for ADLS Gen1 authentication (AZURE_TENANT_ID, AZURE_USERNAME, AZURE_PASSWORD, AZURE_STORE_NAME) are correctly set, or replace placeholders.")