{"id":970,"library":"azure-datalake-store","title":"Azure Data Lake Store Filesystem Client Library","description":"The `azure-datalake-store` library provides a pure-Python interface for Azure Data Lake Storage Gen 1, offering Pythonic file-system and file objects with capabilities for high-performance uploading and downloading. It is currently at version 1.0.1, having recently transitioned from a series of `0.0.x` pre-releases to a `1.0.x` stable branch. The project is under active development, but the official documentation notes it is 'not yet recommended for general use'. This library specifically supports ADLS Gen 1; for ADLS Gen 2, users should refer to `azure-storage-file-datalake`.","status":"active","version":"1.0.1","language":"python","source_language":"en","source_url":"https://github.com/Azure/azure-data-lake-store-python","tags":["azure","data lake","storage","filesystem","adls gen1"],"install":[{"cmd":"pip install azure-datalake-store","lang":"bash","label":"Install stable version"}],"dependencies":[{"reason":"Recommended for modern Azure authentication, especially since v1.0.x moved to generic Azure token credentials. While 'lib.auth' still exists, `azure-identity` aligns with the broader Azure SDK authentication patterns.","package":"azure-identity","optional":false}],"imports":[{"note":"The common pattern shown in documentation is `import core` and then `core.AzureDLFileSystem`. Direct import from `azure.datalake.store.core` also works.","wrong":"from azure.datalake.store.core import AzureDLFileSystem # Direct import is fine, but common pattern is through 'core'","symbol":"AzureDLFileSystem","correct":"from azure.datalake.store import core\nadl_fs = core.AzureDLFileSystem(...)"},{"note":"This method for obtaining a token is still documented, even though the internal authentication mechanism shifted to use generic Azure token credentials in v1.0.x. Consider using environment variables or `azure-identity` for modern auth.","symbol":"lib.auth","correct":"from azure.datalake.store import lib\ntoken = lib.auth(tenant_id, username, password)"}],"quickstart":{"code":"import os\nfrom azure.datalake.store import core\n\n# Set these environment variables for authentication\n# Ensure AZURE_TENANT_ID, AZURE_USERNAME, AZURE_PASSWORD, AZURE_STORE_NAME are set\n# For testing, use placeholder values if not connecting to a real ADLS Gen1\n\ntenant_id = os.environ.get('AZURE_TENANT_ID', 'YOUR_TENANT_ID')\nusername = os.environ.get('AZURE_USERNAME', 'YOUR_USERNAME')\npassword = os.environ.get('AZURE_PASSWORD', 'YOUR_PASSWORD')\nstore_name = os.environ.get('AZURE_STORE_NAME', 'youradlstorename')\n\ntry:\n    # Authenticate (lib.auth now uses generic Azure token credentials internally)\n    token = core.lib.auth(tenant_id, username, password)\n    \n    # Initialize the Data Lake Store filesystem client\n    adl = core.AzureDLFileSystem(store_name, token=token)\n    \n    # Example: List contents of the root directory\n    print(f\"Listing contents of / in {store_name}:\")\n    items = adl.ls('/', detail=True)\n    if items:\n        for item in items:\n            print(item)\n    else:\n        print(\"Directory is empty or path does not exist.\")\n\n    # Example: Create a directory and a file\n    test_dir = 'mytestdir'\n    test_file = f'{test_dir}/testfile.txt'\n    if not adl.exists(test_dir):\n        adl.mkdir(test_dir)\n        print(f\"Created directory: {test_dir}\")\n\n    with adl.open(test_file, 'wb') as f:\n        f.write(b\"Hello from Azure Data Lake Store Gen1!\")\n    print(f\"Created and wrote to file: {test_file}\")\n\n    # Example: Read the file\n    with adl.open(test_file, 'rb') as f:\n        content = f.read()\n        print(f\"Content of {test_file}: {content.decode('utf-8')}\")\n\n    # Example: Delete the file and directory\n    adl.rm(test_file)\n    print(f\"Deleted file: {test_file}\")\n    adl.rmdir(test_dir)\n    print(f\"Deleted directory: {test_dir}\")\n\nexcept Exception as e:\n    print(f\"An error occurred: {e}\")\n    print(\"Please ensure environment variables for ADLS Gen1 authentication (AZURE_TENANT_ID, AZURE_USERNAME, AZURE_PASSWORD, AZURE_STORE_NAME) are correctly set, or replace placeholders.\")\n\n","lang":"python","description":"This quickstart demonstrates how to authenticate and perform basic file operations (list, create directory, create file, write, read, delete) with Azure Data Lake Store Gen 1 using `AzureDLFileSystem`. It relies on environment variables (`AZURE_TENANT_ID`, `AZURE_USERNAME`, `AZURE_PASSWORD`, `AZURE_STORE_NAME`) for authentication, which is a common and recommended approach for service principals."},"warnings":[{"fix":"For ADLS Gen 2, install `azure-storage-file-datalake` (`pip install azure-storage-file-datalake`) and migrate your code to use its APIs (e.g., `DataLakeServiceClient`).","message":"This library is exclusively for **Azure Data Lake Storage Gen 1**. For Azure Data Lake Storage Gen 2, which is the current generation, you **must** use the `azure-storage-file-datalake` library. Using `azure-datalake-store` for Gen 2 will result in compatibility issues.","severity":"breaking","affected_versions":"All versions"},{"fix":"Evaluate your use case carefully. For production systems, consider the stability implications or explore alternatives like ADLS Gen 2 with `azure-storage-file-datalake`.","message":"The official documentation states that this 'software is under active development and not yet recommended for general use'. This suggests it may not be suitable for critical production workloads or that its APIs could still undergo significant changes.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Update your authentication code. While `lib.auth` still exists, ensure your credentials (tenant ID, username, password) are compatible with the new underlying MSAL-based token acquisition or consider using `azure-identity` for direct token management.","message":"Authentication mechanisms changed significantly from `0.x` to `1.0.x`. Older versions used ADAL and custom authentication. Version `1.0.0-alpha0` and `1.0.1` shifted to 'generic azure token credential for auth instead of custom lib.auth' and removed ADAL support, replacing it with MSAL internally within `lib.auth`.","severity":"breaking","affected_versions":"0.0.x to 1.0.x and later"},{"fix":"Review any code relying on multi-part upload concatenation. No direct fix is provided in the release notes, implying the library now handles large files differently internally, which might impact custom upload logic or performance expectations.","message":"In version `1.0.1`, the `concat` operation was removed from multi-part uploads, and large files are now uploaded in a single chunk. This changes the behavior for handling very large files.","severity":"breaking","affected_versions":"1.0.1 and later"},{"fix":"Avoid using `0.0.x` versions for new development. For existing `0.0.x` deployments, plan for a thorough migration and testing period when upgrading to `1.0.x` or later, as many aspects of the API and behavior may have changed.","message":"All `0.0.x` versions were explicitly labeled as 'pre-release or preview version' with a warning that there 'will be fairly rapid development and bug fixing, which might result in breaking changes from release to release.' Upgrading directly from `0.0.x` to `1.0.x` will likely involve significant breaking changes.","severity":"deprecated","affected_versions":"0.0.x"},{"fix":"Upgrade your Python environment to a currently supported version (e.g., Python 3.8+). Check the project's `setup.py` or latest documentation for explicit Python version requirements.","message":"Version `1.0.0-alpha0` and `1.0.1` removed support for older Python versions. The specific versions removed are not detailed in the release notes, but users on older Python environments should verify compatibility.","severity":"breaking","affected_versions":"1.0.0-alpha0 and later"}],"env_vars":null,"last_verified":"2026-05-12T21:56:55.139Z","next_check":"2026-06-27T00:00:00.000Z","problems":[{"fix":"Install the package using 'pip install azure-datalake-store'.","cause":"The 'azure-datalake-store' package is not installed or not accessible in the current environment.","error":"ModuleNotFoundError: No module named 'azure.datalake.store'"},{"fix":"Ensure that the 'azure' package is installed by running 'pip install azure'.","cause":"The 'azure' namespace package is missing, possibly due to incomplete installation of Azure-related packages.","error":"ModuleNotFoundError: No module named 'azure'"},{"fix":"Verify the installation of 'azure-datalake-store' and reinstall it if necessary using 'pip install --upgrade azure-datalake-store'.","cause":"The 'core' module is not found within the 'azure.datalake.store' package, possibly due to an outdated or incorrect installation.","error":"ImportError: cannot import name 'core' from 'azure.datalake.store'"},{"fix":"Check the authentication credentials and ensure the account has the necessary permissions to access the specified directory.","cause":"An error occurred while attempting to list the contents of a directory in Azure Data Lake Store, possibly due to incorrect credentials or insufficient permissions.","error":"DatalakeRESTException: Data-lake REST exception: LISTSTATUS"},{"fix":"Ensure the `azure-datalake-store` package is correctly installed using `pip install azure-datalake-store` and that your Python environment (e.g., virtual environment) is activated and configured correctly. For some older setups, it might also be `pip install azure` if other Azure SDK components are missing, but for this specific library, `azure-datalake-store` is the direct dependency.","cause":"The Python environment cannot find the 'azure.datalake' module, often due to incorrect installation, an inactive virtual environment, or an outdated package structure.","error":"ModuleNotFoundError: No module named 'azure.datalake'"}],"ecosystem":"pypi","meta_description":null,"install_score":100,"install_tag":"verified","quickstart_score":null,"quickstart_tag":null,"pypi_latest":"1.0.1","cli_name":"","install_checks":{"last_tested":"2026-05-12","tag":"verified","tag_description":"installs cleanly on critical runtimes, fast import, recently tested","results":[{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":null,"import_time_s":0.69,"mem_mb":11.1,"disk_size":"23.0M"},{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.66,"mem_mb":11,"disk_size":"23.0M"},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":2.4,"import_time_s":0.48,"mem_mb":11.1,"disk_size":"23M"},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.46,"mem_mb":11,"disk_size":"23M"},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":null,"import_time_s":0.8,"mem_mb":12.1,"disk_size":"25.4M"},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.86,"mem_mb":12.1,"disk_size":"25.4M"},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":2.5,"import_time_s":0.72,"mem_mb":12.1,"disk_size":"26M"},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.65,"mem_mb":12.1,"disk_size":"26M"},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":null,"import_time_s":0.74,"mem_mb":12.8,"disk_size":"17.2M"},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.76,"mem_mb":12.7,"disk_size":"17.1M"},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":2.2,"import_time_s":0.75,"mem_mb":12.8,"disk_size":"18M"},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.76,"mem_mb":12.7,"disk_size":"18M"},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":null,"import_time_s":0.78,"mem_mb":13.1,"disk_size":"16.9M"},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.74,"mem_mb":13.1,"disk_size":"16.8M"},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":2.2,"import_time_s":0.72,"mem_mb":13.1,"disk_size":"17M"},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.76,"mem_mb":13.1,"disk_size":"17M"},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":null,"import_time_s":0.58,"mem_mb":10.7,"disk_size":"23.0M"},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.57,"mem_mb":10.8,"disk_size":"23.0M"},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":2.9,"import_time_s":0.57,"mem_mb":10.7,"disk_size":"23M"},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.48,"mem_mb":10.8,"disk_size":"24M"}]},"quickstart_checks":{"last_tested":"2026-04-24","tag":null,"tag_description":null,"results":[{"runtime":"python:3.10-alpine","exit_code":0},{"runtime":"python:3.10-slim","exit_code":0},{"runtime":"python:3.11-alpine","exit_code":0},{"runtime":"python:3.11-slim","exit_code":0},{"runtime":"python:3.12-alpine","exit_code":0},{"runtime":"python:3.12-slim","exit_code":0},{"runtime":"python:3.13-alpine","exit_code":0},{"runtime":"python:3.13-slim","exit_code":0},{"runtime":"python:3.9-alpine","exit_code":0},{"runtime":"python:3.9-slim","exit_code":0}]}}