{"id":1392,"library":"azure-synapse-spark","title":"Azure Synapse Spark Client Library","description":"The Azure Synapse Spark Client Library for Python provides capabilities to interact with Azure Synapse Analytics Spark pools. It allows for submitting Spark batch jobs, managing Spark sessions, and interacting with Livy endpoints programmatically. As of version 0.7.0, it is part of the larger Azure SDK for Python ecosystem and typically sees updates aligned with new Synapse service features or general Azure SDK releases.","status":"active","version":"0.7.0","language":"en","source_language":"en","source_url":"https://github.com/Azure/azure-sdk-for-python","tags":["azure","synapse","spark","cloud","data analytics"],"install":[{"cmd":"pip install azure-synapse-spark azure-identity","lang":"bash","label":"Install core library and Azure Identity for authentication"}],"dependencies":[{"reason":"Provides shared primitives, HTTP pipeline, and error handling for Azure SDK client libraries.","package":"azure-core"},{"reason":"Used for serialization/deserialization of REST API models.","package":"msrest"},{"reason":"Required for authentication using DefaultAzureCredential and other credential types.","package":"azure-identity","optional":true}],"imports":[{"note":"This is the primary client for interacting with Synapse Spark.","symbol":"SparkClient","correct":"from azure.synapse.spark import SparkClient"},{"note":"Recommended credential for authenticating with Azure services.","symbol":"DefaultAzureCredential","correct":"from azure.identity import DefaultAzureCredential"}],"quickstart":{"code":"import os\nfrom azure.identity import DefaultAzureCredential\nfrom azure.synapse.spark import SparkClient\n\n# Replace with your Synapse workspace name and a Spark pool name\nsynapse_workspace_name = os.environ.get(\"SYNAPSE_WORKSPACE_NAME\", \"your_synapse_workspace_name\")\nspark_pool_name = os.environ.get(\"SYNAPSE_SPARK_POOL_NAME\", \"your_spark_pool_name\")\nendpoint = f\"https://{synapse_workspace_name}.dev.azuresynapse.net\"\n\nif synapse_workspace_name == \"your_synapse_workspace_name\" or spark_pool_name == \"your_spark_pool_name\":\n    print(\"Please set SYNAPSE_WORKSPACE_NAME and SYNAPSE_SPARK_POOL_NAME environment variables \",\n          \"or replace the placeholder values in the code.\")\nelse:\n    try:\n        # Obtain a credential from Azure Identity. Ensure you're logged in via Azure CLI/VS Code, or env vars are set.\n        credential = DefaultAzureCredential()\n\n        # Create a SparkClient\n        spark_client = SparkClient(endpoint=endpoint, credential=credential)\n\n        # List Spark batch jobs in a specific pool (example operation)\n        print(f\"Listing Spark batch jobs for Spark Pool '{spark_pool_name}' in workspace '{synapse_workspace_name}'...\")\n        batch_jobs_collection = spark_client.spark_batch.get_spark_batch_jobs(\n            workspace_name=synapse_workspace_name, \n            spark_pool_name=spark_pool_name\n        )\n\n        print(f\"Found {len(batch_jobs_collection.value)} Spark batch jobs:\")\n        for job in batch_jobs_collection.value:\n            print(f\"  - Job ID: {job.id}, Name: {job.name}, State: {job.state}\")\n\n    except Exception as e:\n        print(f\"Error interacting with Azure Synapse Spark: {e}\")\n        print(\"Ensure your Azure credentials are set up and you have permissions to the Synapse workspace and Spark pool.\")","lang":"python","description":"This quickstart demonstrates how to authenticate with Azure Synapse and list Spark batch jobs within a specified Spark pool. You'll need to set `SYNAPSE_WORKSPACE_NAME` and `SYNAPSE_SPARK_POOL_NAME` environment variables (or hardcode them) and ensure your Azure credentials are configured (e.g., via `az login`)."},"warnings":[{"fix":"Consult the official Azure SDK for Python changelog for `azure-synapse-spark` before upgrading to new preview versions.","message":"As a library in an early preview version (0.x.x), `azure-synapse-spark` may introduce breaking changes in minor version updates. Always review release notes when upgrading.","severity":"breaking","affected_versions":"0.x.x"},{"fix":"Always check the method signatures and examples. Pass `workspace_name` and `spark_pool_name` explicitly where required by the method, typically derived from environment variables or configuration.","message":"Many client methods, like `get_spark_batch_jobs`, require explicit `workspace_name` and `spark_pool_name` arguments, even if the workspace name is implicitly part of the client's `endpoint` URL. Ensure these parameters are consistently provided.","severity":"gotcha","affected_versions":"0.7.0 and likely later"},{"fix":"Ensure `azure-identity` is installed (`pip install azure-identity`) and that your environment is correctly configured for `DefaultAzureCredential` to find credentials (e.g., `az login` or appropriate environment variables).","message":"Authentication with Azure services using `DefaultAzureCredential` relies on specific environment variables (e.g., `AZURE_CLIENT_ID`, `AZURE_CLIENT_SECRET`, `AZURE_TENANT_ID`), being logged in via Azure CLI (`az login`), or other Azure Identity sources. Without proper setup, authentication will fail.","severity":"gotcha","affected_versions":"All versions using `azure-identity`"}],"env_vars":null,"last_verified":"2026-04-09T00:00:00.000Z","next_check":"2026-07-08T00:00:00.000Z"}