{"id":4497,"library":"dbt-athena","title":"dbt-athena","description":"dbt-athena is an adapter plugin for dbt (data build tool) that enables data analysts and engineers to transform data in Amazon Athena, a serverless query service for S3 data, using SQL. It is currently at version 1.10.0 and is actively maintained by dbt Labs. The library generally follows semantic versioning with frequent patch releases, minor versions adding backward-compatible features, and major versions potentially introducing breaking changes.","status":"active","version":"1.10.0","language":"en","source_language":"en","source_url":"https://github.com/dbt-labs/dbt-adapters/tree/main/dbt-athena","tags":["dbt","athena","aws","data-transformation","elt","data-lake","sql","data-modeling"],"install":[{"cmd":"pip install dbt-core dbt-athena","lang":"bash","label":"Install dbt-athena and dbt-core"}],"dependencies":[{"reason":"Required as of dbt-core v1.8; adapters no longer bundle dbt-core.","package":"dbt-core","optional":false},{"reason":"Underlying AWS SDK used for interacting with Athena and S3.","package":"boto3","optional":false}],"imports":[{"note":"dbt adapters primarily extend dbt CLI functionality and provide connection configuration, rather than direct Python APIs for common use cases.","symbol":"dbt-athena","correct":"dbt-athena is used via the dbt CLI and profiles.yml configuration; no direct Python imports for end-user functionality are typically needed."}],"quickstart":{"code":"# 1. Install dbt-athena and dbt-core\npip install dbt-core dbt-athena\n\n# 2. Initialize a new dbt project (follow prompts, select 'athena' as database type)\ndbt init my_athena_project\n\n# 3. Configure your profiles.yml (e.g., ~/.dbt/profiles.yml or project_root/profiles.yml)\n# Example profiles.yml content:\n# my_athena_project:\n#   target: dev\n#   outputs:\n#     dev:\n#       type: athena\n#       s3_staging_dir: s3://your-athena-query-results-bucket/dbt-staging/\n#       region_name: us-east-1\n#       database: your_athena_database\n#       schema: dbt_schema\n#       threads: 4\n#       aws_profile_name: default # Or use aws_access_key_id and aws_secret_access_key\n\n# 4. Create an S3 bucket and Athena database as prerequisites\n# (AWS CLI/Console steps, not Python code):\n#   aws s3 mb s3://your-athena-query-results-bucket\n#   aws athena create-data-catalog --name your_athena_database --type LAMBDA --parameters \"catalog-id\"=\"your_glue_catalog_id\" # Or simply use an existing database\n\n# 5. Create a sample dbt model (e.g., my_athena_project/models/my_first_model.sql)\n# -- my_athena_project/models/my_first_model.sql\n# select 1 as id, 'hello' as message\n\n# 6. Test the connection and run your dbt project\ndbt debug --target dev\ndbt run --target dev\n","lang":"bash","description":"This quickstart outlines the steps to set up a dbt project with the dbt-athena adapter. It involves installing the necessary packages, initializing a dbt project, configuring your AWS Athena connection details in `profiles.yml`, and then running `dbt` commands to test and execute your models. Ensure you have an S3 bucket for query results and an Athena database set up in AWS."},"warnings":[{"fix":"Explicitly install `dbt-core` alongside `dbt-athena`: `pip install dbt-core dbt-athena`.","message":"As of dbt-core version 1.8, dbt-athena (and other adapters) no longer automatically install `dbt-core`. You must install `dbt-core` separately and ensure compatibility with your adapter version.","severity":"breaking","affected_versions":"dbt-athena >= 1.8.0"},{"fix":"Prefer `pip install dbt-athena` for new installations and when upgrading.","message":"The `dbt-athena-community` package is now a wrapper around `dbt-athena`. While still functional for backward compatibility, new projects and migrations should directly use `dbt-athena`.","severity":"gotcha","affected_versions":"All versions where `dbt-athena` exists alongside `dbt-athena-community`."},{"fix":"Ensure `s3_staging_dir` and `s3_data_dir` configurations are carefully managed to prevent unintended data loss. Use distinct locations for different models or projects if data needs to be preserved outside of dbt's management.","message":"If a dbt model has the same name as an existing table in the AWS Glue catalog, or if a model is configured to use the same S3 location as an existing table, the adapter *deletes* files in that table's S3 location before recreating the table. This is to avoid conflicts.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Set `num_retries` to `N+1` to achieve `N` retries (i.e., `num_retries: 2` for one retry).","message":"The `num_retries` parameter in `profiles.yml` (e.g., for query retries) is often misunderstood. Setting `num_retries: N` results in `N-1` actual retries after the initial attempt. For one retry, you should set `num_retries: 2`.","severity":"gotcha","affected_versions":"All versions"},{"fix":"For tables exceeding this limit during writes, consider batching operations or utilizing Athena Spark for Python models, which can handle larger partition counts. This limit does not apply to Athena Spark.","message":"Athena SQL (used by dbt-athena for most operations) imposes a limit of 100 partitions per write operation. This can cause issues with full table rebuilds or large incremental merges on highly partitioned tables.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Standardize all dbt model, schema, and database names to lowercase in your project and configurations.","message":"Table, schema, and database names should be lowercase when using dbt-athena to avoid potential conflicts and issues with Athena's case-insensitivity behavior.","severity":"gotcha","affected_versions":"All versions"}],"env_vars":null,"last_verified":"2026-04-12T00:00:00.000Z","next_check":"2026-07-11T00:00:00.000Z"}