{"id":7968,"library":"azureml-fsspec","title":"AzureML fsspec Protocol Handler","description":"The `azureml-fsspec` library enables the `fsspec` (Filesystem Spec) library to interact with Azure Machine Learning datastores. It registers 'azureml://' and 'adl://' protocols, allowing users to access, read, and write files within Azure ML datastores using the familiar `fsspec` API. The current version is 1.3.1, and it's released as part of the broader Azure ML SDK ecosystem, typically following its release cadence.","status":"active","version":"1.3.1","language":"en","source_language":"en","source_url":"https://github.com/Azure/azureml-open-source-libraries/tree/main/azureml-fsspec","tags":["azure","azureml","fsspec","filesystem","cloud storage"],"install":[{"cmd":"pip install azureml-fsspec","lang":"bash","label":"Install latest version"}],"dependencies":[{"reason":"Core dependency for filesystem abstraction, requires version >=2023.10.0.","package":"fsspec"},{"reason":"Underlying Azure Machine Learning SDK for datastore authentication and management, requires version ~=1.51.0.","package":"azureml-core"},{"reason":"Microsoft Authentication Library for Python, used for Azure AD authentication, requires >=1.20.0,<2.0.0.","package":"msal"}],"imports":[{"symbol":"AzureMLFileSystem","correct":"from azureml.fsspec import AzureMLFileSystem"},{"note":"Importing the package registers the 'azureml' and 'adl' protocols with fsspec automatically.","symbol":"Protocol Registration","correct":"import azureml.fsspec"}],"quickstart":{"code":"import fsspec\nimport azureml.fsspec # This import registers the \"azureml\" and \"adl\" protocols\nimport os\n\nprint(\"azureml-fsspec imported, protocols registered.\")\n\n# Environment variables for AzureML workspace details. \n# In a real scenario, these must be set or passed via the fsspec.filesystem() call or URI query params.\nsubscription_id = os.environ.get(\"AZUREML_SUBSCRIPTION_ID\", \"\")\nresource_group = os.environ.get(\"AZUREML_RESOURCE_GROUP\", \"\")\nworkspace_name = os.environ.get(\"AZUREML_WORKSPACE_NAME\", \"\")\ndatastore_name = os.environ.get(\"AZUREML_DATASTORE_NAME\", \"workspaceblobstore\") # Common default datastore name\n\nif all([subscription_id, resource_group, workspace_name]):\n    print(f\"\\nAttempting to get AzureML filesystem instance for datastore: {datastore_name}...\")\n    try:\n        # Get a filesystem instance using specific workspace details and datastore.\n        # This approach explicitly provides connection details.\n        fs = fsspec.filesystem(\"azureml\",\n                                subscription_id=subscription_id,\n                                resource_group=resource_group,\n                                workspace_name=workspace_name,\n                                datastore_name=datastore_name)\n        print(f\"Successfully initialized AzureML filesystem object: {type(fs)}\")\n\n        # Example: Try to list the root of the specified datastore. \n        # This will only succeed with valid credentials and permissions.\n        # path_to_list = f\"azureml://datastores/{datastore_name}/paths/\"\n        # print(f\"Attempting to list contents of: {path_to_list}\")\n        # contents = fs.ls(\"/\") # Listing root of the specific datastore instance\n        # print(f\"First 5 items from {datastore_name} root: {contents[:5]}...\") \n        # print(f\"Total items found: {len(contents)}\")\n\n    except Exception as e:\n        print(f\"Could not initialize AzureML filesystem (this is common without a full setup): {e}\")\n        print(\"Please ensure AZUREML_SUBSCRIPTION_ID, AZUREML_RESOURCE_GROUP, AZUREML_WORKSPACE_NAME, and AZUREML_DATASTORE_NAME are correctly set as environment variables or passed as arguments.\")\nelse:\n    print(\"\\nSkipping AzureML filesystem initialization and IO: AzureML environment variables not fully set.\")\n    print(\"Set AZUREML_SUBSCRIPTION_ID, AZUREML_RESOURCE_GROUP, AZUREML_WORKSPACE_NAME, and AZUREML_DATASTORE_NAME for a live example.\")\n\nprint(\"\\nDemonstrating direct import of AzureMLFileSystem class:\")\nfrom azureml.fsspec import AzureMLFileSystem\nprint(f\"AzureMLFileSystem class found: {AzureMLFileSystem}\")","lang":"python","description":"This quickstart demonstrates how to import `azureml-fsspec` to register its protocols with `fsspec` and how to obtain an `AzureMLFileSystem` instance. It highlights the need for Azure ML workspace details (subscription ID, resource group, workspace name, datastore name), typically provided via environment variables. Actual I/O operations (like listing files) are commented out, as they require a live Azure ML setup with valid credentials and permissions to succeed without errors."},"warnings":[{"fix":"Ensure your environment is correctly authenticated with Azure. Use `az login` for Azure CLI, or set appropriate environment variables for service principal or managed identity authentication before running your Python code.","message":"AzureML authentication is required and can be complex. This library relies on the underlying AzureML SDK's authentication mechanisms, which can include environment variables (e.g., AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, AZURE_TENANT_ID), Azure CLI login, or managed identity.","severity":"gotcha","affected_versions":">=1.0.0"},{"fix":"Always include `import azureml.fsspec` at the beginning of your script when intending to use `fsspec` with Azure ML datastores. This import performs the necessary protocol registration.","message":"Forgetting to import `azureml.fsspec` will prevent the 'azureml' and 'adl' protocols from being registered with `fsspec`, leading to 'No such protocol' errors.","severity":"gotcha","affected_versions":">=1.0.0"},{"fix":"Ensure your URI follows the pattern `azureml://datastores/<datastore_name>/paths/<path_to_file>` and that workspace details are provided either in the URI (as query parameters), as `fsspec.filesystem` arguments, or via environment variables. `adl://` is specific to ADLS Gen2 datastores.","message":"Incorrect or incomplete Azure ML URI format can lead to errors. The `azureml://` protocol expects specific parameters (subscription ID, resource group, workspace name, datastore name, and path). These can be in the URI itself or passed as keyword arguments to `fsspec.filesystem()`.","severity":"gotcha","affected_versions":">=1.0.0"},{"fix":"Check `pip list` and ensure `fsspec>=2023.10.0` and `azureml-core~=1.51.0` (or compatible versions) are installed. Upgrade with `pip install --upgrade fsspec azureml-core` if needed.","message":"Requires specific minimum versions of `fsspec` and `azureml-core`. Older versions of these dependencies might not provide the necessary APIs or features.","severity":"gotcha","affected_versions":"<1.3.0"}],"env_vars":null,"last_verified":"2026-04-16T00:00:00.000Z","next_check":"2026-07-15T00:00:00.000Z","problems":[{"fix":"Ensure `azureml-fsspec` is installed (`pip install azureml-fsspec`) and that `import azureml.fsspec` is present in your code before attempting to use an `azureml://` or `adl://` URI with `fsspec`.","cause":"The `azureml` protocol was not registered with `fsspec`.","error":"fsspec.exceptions.FSSpecError: No such protocol: azureml"},{"fix":"Verify your Azure authentication setup. This could involve running `az login`, setting environment variables for service principal details (e.g., `AZURE_CLIENT_ID`, `AZURE_CLIENT_SECRET`, `AZURE_TENANT_ID`), or ensuring your managed identity has the correct permissions.","cause":"The AzureML SDK could not authenticate with your Azure subscription or workspace.","error":"AuthenticationException: Authentication failed. Please check your credentials."},{"fix":"Double-check the datastore name for typos. Ensure the `subscription_id`, `resource_group`, and `workspace_name` parameters or environment variables point to the correct Azure ML workspace where the datastore is registered.","cause":"The datastore name specified in the URI or `fsspec.filesystem` call does not exist in the targeted Azure ML workspace, or the workspace details are incorrect.","error":"azureml.exceptions._azureml_exception.DatastoreNotFoundException: Datastore with name '<your_datastore_name>' not found."},{"fix":"Review the correct URI format for `azureml-fsspec`. It typically follows `azureml://datastores/<datastore_name>/paths/<path_to_file>`. Ensure all necessary parts like datastore name and path are correctly specified.","cause":"The provided `azureml://` or `adl://` URI does not conform to the expected structure or is missing required components.","error":"ValueError: Invalid URI format: 'azureml://...' (or similar URI parsing error)"}]}