{"id":8073,"library":"dbt-fabricspark","title":"dbt-fabricspark","description":"dbt-fabricspark is a Microsoft Fabric Spark adapter plugin for dbt (data build tool), enabling data analysts and engineers to transform data within Microsoft Fabric Lakehouses. It connects to Fabric Lakehouses via Livy endpoints, supports both schema-enabled and non-schema configurations, and includes Livy session management. The library is actively maintained by Microsoft and its current version is 1.9.5, typically aligning with dbt-core release cycles.","status":"active","version":"1.9.5","language":"en","source_language":"en","source_url":"https://github.com/microsoft/dbt-fabricspark","tags":["dbt","microsoft fabric","apache spark","data transformation","etl","lakehouse"],"install":[{"cmd":"pip install dbt-core dbt-fabricspark","lang":"bash","label":"Install dbt-core and dbt-fabricspark"},{"cmd":"az login","lang":"bash","label":"Authenticate with Azure CLI"}],"dependencies":[{"reason":"Core dbt functionality","package":"dbt-core","optional":false},{"reason":"dbt adapter base classes","package":"dbt-adapters","optional":false},{"reason":"dbt common utilities","package":"dbt-common","optional":false},{"reason":"Authentication via Azure CLI for local development","package":"azure-cli","optional":false},{"reason":"Azure SDK dependency","package":"azure-core","optional":false},{"reason":"Azure SDK dependency for authentication","package":"azure-identity","optional":false},{"reason":"HTTP client for API interactions","package":"requests","optional":false}],"imports":[],"quickstart":{"code":"# 1. Ensure dbt-core and dbt-fabricspark are installed and Azure CLI is logged in.\n# pip install dbt-core dbt-fabricspark\n# az login\n\n# 2. Configure your ~/.dbt/profiles.yml file (replace placeholders):\n# fabricspark-dev:\n#   target: dev\n#   outputs:\n#     dev:\n#       type: fabricspark\n#       method: livy\n#       authentication: CLI\n#       endpoint: https://api.fabric.microsoft.com/v1\n#       workspaceid: <your-workspace-guid>\n#       lakehouseid: <your-lakehouse-guid>\n#       lakehouse: <your-lakehouse-name>\n#       schema: <your-schema-name> # Optional, defaults to target schema, or lakehouse name for non-schema lakehouses\n#       threads: 1\n#       connect_retries: 2\n#       connect_timeout: 10\n#       retry_all: true # Recommended for production\n\n# 3. Create a dbt project (if you don't have one):\n# dbt init my_fabric_project\n\n# 4. In your dbt project's dbt_project.yml, set the profile:\n# name: 'my_fabric_project'\n# profile: 'fabricspark-dev'\n\n# 5. Create a sample model (e.g., models/my_first_model.sql):\n# -- models/my_first_model.sql\n# {{ config(materialized='table') }}\n# SELECT\n#   1 as id,\n#   'dbt fabricspark test' as message\n\n# 6. Run your dbt models:\n# dbt run\n\n# This command will connect to your Fabric Lakehouse via Spark Livy and execute the SQL.\nprint(\"dbt-fabricspark quickstart involves configuring profiles.yml and running dbt CLI commands.\")\nprint(\"Authentication relies on an active Azure CLI login session.\")","lang":"python","description":"The quickstart involves setting up your `profiles.yml` with the `fabricspark` adapter type, specifying your Fabric workspace, lakehouse, and authentication method (typically Azure CLI). After configuring, standard `dbt` CLI commands like `dbt run` will execute transformations on your Microsoft Fabric Lakehouse. Ensure you are logged in via `az login` for CLI authentication."},"warnings":[{"fix":"Ensure you install both `dbt-core` and `dbt-fabricspark` explicitly: `pip install dbt-core dbt-fabricspark`.","message":"Breaking change in dbt-fabricspark v1.8: Prior to version 1.8, installing `dbt-fabricspark` automatically installed `dbt-core` and its dependencies. From version 1.8 onwards, `dbt-core` must be installed separately alongside the adapter.","severity":"breaking","affected_versions":">=1.8.0"},{"fix":"Always use the `lakehouse` and `schema` fields in `profiles.yml` for targeting. Avoid setting a `database` field directly.","message":"Do not use 'database' configuration: Microsoft Fabric Spark uses the terms 'schema' and 'database' interchangeably, but dbt treats 'database' as a higher level. When using `dbt-fabricspark`, you should never use or set `database` as a node configuration or in your `profiles.yml` target profile.","severity":"gotcha","affected_versions":"All versions"},{"fix":"For models where updates or overwrites are needed, explicitly set `incremental_strategy` to `insert_overwrite` (with `partition_by`) or `merge` (with `unique_key` and Delta file format) in your model configurations. For example: `{{ config(materialized='incremental', incremental_strategy='merge', unique_key='id') }}`.","message":"Incremental strategy default ('append') can cause duplicates: The default `incremental_strategy` for dbt-fabricspark is `append`. This strategy inserts new records without updating or overwriting existing data, which can lead to duplicate records in your models if not managed carefully.","severity":"gotcha","affected_versions":"All versions"},{"fix":"In your `profiles.yml`, under your target, add `reuse_session: true` and optionally `session_id_file: ./livy-session-id.txt` to enable session reuse for faster iterative development. Example: `reuse_session: true, session_id_file: '/tmp/my-dbt-session.txt'`","message":"Livy session management for local development: In 'Fabric mode' (`livy_mode: fabric`, default), enabling `reuse_session: true` can persist the Livy session ID to a local file, allowing dbt to reuse an existing Spark session on subsequent runs instead of creating a new one. This is key for faster development workflows but requires proper configuration of `session_id_file`.","severity":"gotcha","affected_versions":"All versions"}],"env_vars":null,"last_verified":"2026-04-16T00:00:00.000Z","next_check":"2026-07-15T00:00:00.000Z","problems":[{"fix":"Run `az login` in your terminal to refresh your Azure CLI session. For service principal authentication, ensure your client ID, tenant ID, and client secret/certificate are valid and correctly configured in `profiles.yml` or environment variables.","cause":"The Azure CLI token used for authentication has expired, or the service principal credentials are incorrect/expired.","error":"Authentication has expired. Please log in again. OR HTTP 401 Unauthorized errors."},{"fix":"Inspect the macro or variable in the specified file. Check its return type and how it's being accessed. Ensure all package macros are available by running `dbt deps`.","cause":"This typically means a Jinja macro or variable reference is incorrect or returns an unexpected `None` or string type when an object/dictionary is expected. This can happen if a macro doesn't return a value or a variable is referenced improperly (e.g., `{{ var.value }}` instead of `{{ var }}`).","error":"Compilation Error: 'None' has no attribute 'table' OR 'str object' has no attribute 'COLUMN_NAME'"},{"fix":"Review the compiled SQL in `target/compiled/<project_name>/models/...` and `target/run/<project_name>/models/...`. Run the compiled SQL directly in a Spark SQL client to debug syntax. Use Spark-compatible SQL and ensure data types are correctly handled (e.g., using `CAST` functions).","cause":"dbt's compile phase processes Jinja but doesn't fully validate underlying SQL syntax or data type compatibility until runtime. This error indicates an issue with the generated SQL that Spark cannot execute. Common causes include typos, incorrect SQL functions for Spark, or incompatible data types in operations.","error":"Database Error: SQL compilation error: syntax error line X at position Y unexpected 'keyword' OR Invalid data type cast."},{"fix":"For incremental models, verify the `materialized='incremental'` configuration. Ensure `incremental_strategy` is set correctly (`merge` or `insert_overwrite`) and `unique_key` and `partition_by` (if applicable) are defined in your model config. Validate the `is_incremental()` logic in your SQL for proper conditional execution.","cause":"This usually occurs with incremental models where the `incremental_strategy` or `unique_key` is misconfigured, leading dbt to attempt a full table creation instead of an incremental update or merge, or when `is_incremental()` logic is missing or flawed.","error":"Error creating relation 'my_table'. Table 'my_table' already exists. OR dbt is trying to create a table that already exists."}]}