{"id":475,"library":"databricks-sql-connector","title":"Databricks SQL Connector for Python","description":"The Databricks SQL Connector for Python is a Python library that enables running SQL commands on Databricks clusters and SQL warehouses. It is a Thrift-based client, conforms to the Python DB API 2.0 specification, and uses Apache Arrow for efficient data exchange. The library is actively maintained with frequent releases, often multiple times a month.","status":"active","version":"4.2.5","language":"python","source_language":"en","source_url":"https://github.com/databricks/databricks-sql-python","tags":["databricks","sql","database","connector","db-api","pyarrow"],"install":[{"cmd":"pip install databricks-sql-connector","lang":"bash","label":"Core library"},{"cmd":"pip install databricks-sql-connector[pyarrow]","lang":"bash","label":"With PyArrow for Arrow functionality"}],"dependencies":[{"reason":"Runtime environment","package":"python","version":">=3.8.0, <4.0.0"},{"reason":"Optional, for CloudFetch and Apache Arrow-based data transfer features. Not installed by default in v4.0.0+.","package":"pyarrow","optional":true},{"reason":"Optional, required for OAuth Machine-to-Machine (M2M) and User-to-Machine (U2M) authentication.","package":"databricks-sdk","optional":true}],"imports":[{"symbol":"sql","correct":"from databricks import sql"}],"quickstart":{"code":"import os\nfrom databricks import sql\n\n# Ensure these environment variables are set:\n# DATABRICKS_SERVER_HOSTNAME, DATABRICKS_HTTP_PATH, DATABRICKS_TOKEN\n\nhost = os.environ.get('DATABRICKS_SERVER_HOSTNAME', 'your_server_hostname.databricks.com')\nhttp_path = os.environ.get('DATABRICKS_HTTP_PATH', '/sql/1.0/endpoints/your_sql_warehouse_id')\naccess_token = os.environ.get('DATABRICKS_TOKEN', 'dapiXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX')\n\nif not all([host, http_path, access_token]):\n    print(\"Please set DATABRICKS_SERVER_HOSTNAME, DATABRICKS_HTTP_PATH, and DATABRICKS_TOKEN environment variables.\")\nelse:\n    try:\n        with sql.connect(\n            server_hostname=host,\n            http_path=http_path,\n            access_token=access_token\n        ) as connection:\n            with connection.cursor() as cursor:\n                cursor.execute(\"SELECT 1 as id, 'hello' as message\")\n                result = cursor.fetchall()\n                for row in result:\n                    print(row)\n    except Exception as e:\n        print(f\"An error occurred: {e}\")","lang":"python","description":"This quickstart demonstrates how to establish a connection to a Databricks SQL warehouse using a Personal Access Token (PAT) and execute a simple query. Ensure that `DATABRICKS_SERVER_HOSTNAME`, `DATABRICKS_HTTP_PATH`, and `DATABRICKS_TOKEN` environment variables are set with your Databricks connection details."},"warnings":[{"fix":"Install with `pip install databricks-sql-connector[pyarrow]` or `pip install pyarrow` separately.","message":"PyArrow is no longer a default dependency since version 4.0.0. Users must explicitly install `databricks-sql-connector[pyarrow]` or `pip install pyarrow` to enable Arrow-based features like CloudFetch and `fetchmany_arrow`. Failing to do so may impact performance for large datasets.","severity":"breaking","affected_versions":">=4.0.0"},{"fix":"Install the separate SQLAlchemy dialect: `pip install databricks-sqlalchemy`.","message":"The SQLAlchemy dialect for Databricks was split into a separate `databricks-sqlalchemy` package in version 4.0.0. Users leveraging SQLAlchemy must now explicitly install `databricks-sqlalchemy` alongside the core connector.","severity":"breaking","affected_versions":">=4.0.0"},{"fix":"Optionally, use the `databricks-sqlalchemy` engine for `pandas.read_sql` to suppress the warning or if broader SQLAlchemy compatibility is needed.","message":"When using `pandas.read_sql` directly with a `databricks-sql-connector` connection object (v3.0.0+), a `UserWarning` regarding DB API 2.0 support might appear. While this warning can generally be ignored due to PyArrow's efficiency, using `databricks-sqlalchemy` as the engine provides a warning-free experience and broader compatibility.","severity":"gotcha","affected_versions":">=3.0.0"},{"fix":"Consider using alternative methods for large inserts, or a dedicated fix package like `pandas-tosql-dbx-fix` which compiles the SQL query before sending it.","message":"The `pandas.DataFrame.to_sql()` method effectively broke for inserts exceeding 255 values (not rows) into Delta tables with `databricks-sql-connector` versions 3.0.0 and above, due to the introduction of native parameters and a server-side limitation.","severity":"breaking","affected_versions":">=3.0.0"},{"fix":"Ensure `autocommit` is managed correctly by the connector's transaction methods. For explicit control, set `autocommit=False` during connection establishment or directly on the connection object, then use `commit()`/`rollback()`. Avoid direct setting if it's causing the `CONFIG_NOT_AVAILABLE` error.","message":"Version 4.2.0 introduced changes to `autocommit` properties on the connection object. Directly setting `connection.autocommit = False` via API calls (e.g., in an Airflow hook) can now lead to `TransactionError: [CONFIG_NOT_AVAILABLE] Configuration AUTOCOMMIT is not available. SQLSTATE: 42K0I`. Multi-statement transaction control should be managed through `connection.autocommit = False` then `connection.commit()` and `connection.rollback()`.","severity":"breaking","affected_versions":">=4.2.0"},{"fix":"Ensure your environment uses Python 3.9 or newer when working with `databricks-sql-connector` versions 4.x.x.","message":"While PyPI metadata states Python >=3.8.0 is supported, the official GitHub README for the latest versions (4.x.x) recommends Python 3.9 or above for full compatibility and access to the latest features and stability.","severity":"gotcha","affected_versions":">=4.0.0"},{"fix":"Ensure C build tools (e.g., `gcc`, `build-base` for Alpine) are installed in your environment before installing `databricks-sql-connector`.","message":"Installation fails with `error: command 'gcc' failed` when building the `lz4` dependency (e.g., in minimal environments like Alpine Linux). This occurs because `lz4` requires C compilation, and build tools are not present.","severity":"breaking","affected_versions":">=4.0.0"}],"env_vars":null,"last_verified":"2026-05-12T14:08:02.698Z","next_check":"2026-06-26T00:00:00.000Z","problems":[{"fix":"Ensure `databricks-sql-connector` is installed using `pip install databricks-sql-connector`. The correct import for the SQL connector is `from databricks.sql import connect`.","cause":"This error occurs when you try to import `databricks.sql` but the `databricks-sql-connector` package is not installed, or you are attempting to use an outdated import path or a different Databricks-related package like `databricks-connect` which has a different import structure.","error":"ModuleNotFoundError: No module named 'databricks'"},{"fix":"Double-check that `server_hostname`, `http_path`, and `access_token` are correct and have the necessary permissions. Ensure there are no typos, leading/trailing spaces, or incorrect environment variable configurations. Also, confirm network connectivity to the Databricks workspace.","cause":"This is a generic connection error, often indicating issues with the provided `server_hostname`, `http_path`, or an invalid or expired `access_token` (Personal Access Token).","error":"databricks.sql.exc.RequestError: Error during request to server"},{"fix":"Verify that all connection parameters are explicitly set and are not `None`. For example, if using environment variables, ensure they are defined in your execution environment: `os.getenv(\"DATABRICKS_SERVER_HOSTNAME\")` must return a string, not `None`.","cause":"This error typically occurs when one of the required connection parameters (like `server_hostname`, `http_path`, or `access_token`) is passed as `None` to the `sql.connect()` function, often because an environment variable was not set or retrieved correctly.","error":"AttributeError: 'NoneType' object has no attribute 'startswith'"},{"fix":"Install the missing `packaging` library: `pip install packaging`.","cause":"This error means that a required dependency, the `packaging` module, is missing from your Python environment, which `databricks-sql-connector` relies on.","error":"ModuleNotFoundError: No module named 'packaging'"},{"fix":"Pin the `databricks-sql-connector` version to an earlier compatible version (e.g., `pip install databricks-sql-connector==4.0.5`) or update the dependent library (like `dbt-databricks`) to a version that is compatible with your current `databricks-sql-connector` version.","cause":"This `AttributeError` indicates an incompatibility, typically with older versions of libraries that depend on `databricks-sql-connector` (e.g., `dbt-databricks`). These dependent libraries might be trying to access the `active_op_handle` attribute on the `Cursor` object, which was present in earlier versions of `databricks-sql-connector` but has since been removed or changed.","error":"AttributeError: 'Cursor' object has no attribute 'active_op_handle'"}],"ecosystem":"pypi","meta_description":null,"install_score":50,"install_tag":"draft","quickstart_score":0,"quickstart_tag":"stale","pypi_latest":null,"install_checks":{"last_tested":"2026-05-12","tag":"draft","tag_description":"notable install failures or slow imports","results":[{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":"default","exit_code":1,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":"pyarrow","exit_code":1,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.02,"mem_mb":1.6,"disk_size":"177M"},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":"pyarrow","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.02,"mem_mb":1.6,"disk_size":"324M"},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":"default","exit_code":1,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":"pyarrow","exit_code":1,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.06,"mem_mb":1.9,"disk_size":"199M"},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":"pyarrow","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.05,"mem_mb":1.9,"disk_size":"348M"},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":"default","exit_code":1,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":"pyarrow","exit_code":1,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.04,"mem_mb":1.5,"disk_size":"182M"},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":"pyarrow","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.04,"mem_mb":1.5,"disk_size":"330M"},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":"default","exit_code":1,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":"pyarrow","exit_code":1,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.05,"mem_mb":1.7,"disk_size":"181M"},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":"pyarrow","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.04,"mem_mb":1.7,"disk_size":"329M"},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":"default","exit_code":1,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":"pyarrow","exit_code":1,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.03,"mem_mb":1.6,"disk_size":"186M"},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":"pyarrow","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.03,"mem_mb":1.6,"disk_size":"325M"}]},"quickstart_checks":{"last_tested":"2026-04-23","tag":"stale","tag_description":"widespread failures or data too old to trust","results":[{"runtime":"python:3.10-alpine","exit_code":1},{"runtime":"python:3.10-slim","exit_code":-1},{"runtime":"python:3.11-alpine","exit_code":1},{"runtime":"python:3.11-slim","exit_code":-1},{"runtime":"python:3.12-alpine","exit_code":1},{"runtime":"python:3.12-slim","exit_code":-1},{"runtime":"python:3.13-alpine","exit_code":1},{"runtime":"python:3.13-slim","exit_code":-1},{"runtime":"python:3.9-alpine","exit_code":1},{"runtime":"python:3.9-slim","exit_code":-1}]}}