{"id":4496,"library":"dbt-athena-community","title":"dbt-athena-community","description":"dbt-athena-community is a community-maintained dbt adapter that enables dbt to connect and transform data in AWS Athena. It allows users to leverage dbt's data transformation capabilities by querying data directly from S3 using Athena's serverless engine. Currently at version 1.10.0, its release cycle generally aligns with major `dbt-core` versions, ensuring compatibility and leveraging new dbt features. It is a popular alternative to the official `dbt-athena` adapter.","status":"active","version":"1.10.0","language":"en","source_language":"en","source_url":"https://github.com/dbt-athena/dbt-athena-community","tags":["dbt","data-warehouse","etl","athena","aws","data-transformation"],"install":[{"cmd":"pip install dbt-athena-community","lang":"bash","label":"Install `dbt-athena-community`"}],"dependencies":[{"reason":"Core dbt functionality","package":"dbt-core","optional":false},{"reason":"Python DB API 2.0 client for Amazon Athena","package":"pyathena","optional":false},{"reason":"AWS SDK for Python","package":"boto3","optional":false},{"reason":"Underlying AWS service interactions for boto3","package":"botocore","optional":false}],"imports":[{"note":"dbt adapters are primarily configured via YAML files (e.g., `profiles.yml`, `dbt_project.yml`) and used via the dbt CLI, not direct Python imports of adapter-specific classes for end-users.","symbol":"dbt CLI","correct":"Interact via dbt CLI commands like 'dbt run' after configuring profiles.yml"}],"quickstart":{"code":"import os\nimport subprocess\nimport yaml\nfrom pathlib import Path\nimport shutil\n\n# Setup a temporary dbt profiles directory and project\ntemp_dbt_dir = Path(\"./temp_dbt_profiles\")\ntemp_dbt_dir.mkdir(exist_ok=True)\nprofiles_path = temp_dbt_dir / \"profiles.yml\"\n\n# Use environment variables for sensitive data or set placeholders\naws_access_key_id = os.environ.get(\"AWS_ACCESS_KEY_ID\", \"YOUR_ACCESS_KEY\") # For IAM user\naws_secret_access_key = os.environ.get(\"AWS_SECRET_ACCESS_KEY\", \"YOUR_SECRET_KEY\") # For IAM user\naws_session_token = os.environ.get(\"AWS_SESSION_TOKEN\", \"\") # For temporary credentials\ns3_staging_dir = os.environ.get(\"DBT_ATHENA_S3_STAGING_DIR\", \"s3://your-dbt-athena-bucket/staging/\")\nathena_workgroup = os.environ.get(\"DBT_ATHENA_WORKGROUP\", \"primary\")\nathena_database = os.environ.get(\"DBT_ATHENA_DATABASE\", \"dbt_athena_db\")\naws_region = os.environ.get(\"AWS_REGION\", \"us-east-1\")\n\nprofiles_content = {\n    \"my_athena_project\": { # This name must match 'profile' in dbt_project.yml\n        \"target\": \"dev\",\n        \"outputs\": {\n            \"dev\": {\n                \"type\": \"athena\",\n                \"s3_staging_dir\": s3_staging_dir,\n                \"database\": athena_database,\n                \"schema\": \"dbt_schema\",\n                \"region_name\": aws_region,\n                \"work_group\": athena_workgroup,\n                # Authentication: Use one of the following methods\n                \"aws_profile_name\": \"default\", # Uses ~/.aws/credentials profile\n                # OR directly provide credentials (less secure for production)\n                # \"aws_access_key_id\": aws_access_key_id,\n                # \"aws_secret_access_key\": aws_secret_access_key,\n                # \"aws_session_token\": aws_session_token, # Optional\n                # Other common optional settings\n                \"poll_interval\": 5, # Seconds between status checks\n                \"num_retries\": 10,\n                \"threads\": 4\n            }\n        }\n    }\n}\n\nwith open(profiles_path, \"w\") as f:\n    yaml.dump(profiles_content, f, default_flow_style=False)\n\nprint(f\"Profiles file created at: {profiles_path}\")\n\n# Create a minimal dbt project structure\nproject_dir = Path(\"./temp_dbt_project\")\nproject_dir.mkdir(exist_ok=True)\n(project_dir / \"models\").mkdir(exist_ok=True)\n\ndbt_project_yml_content = f\"\"\"\nname: 'my_athena_project'\nversion: '1.0.0'\nconfig-version: 2\n\nprofile: 'my_athena_project'\n\nmodel-paths: [\"models\"]\nanalysis-paths: [\"analyses\"]\ntest-paths: [\"tests\"]\nseed-paths: [\"seeds\"]\nmacro-paths: [\"macros\"]\nsnapshot-paths: [\"snapshots\"]\n\ntarget-path: \"target\"\nclean-targets:\n  - \"target\"\n  - \"dbt_packages\"\n  - \"logs\"\n\nmodels:\n  my_athena_project:\n    +materialized: view\n\"\"\"\nwith open(project_dir / \"dbt_project.yml\", \"w\") as f:\n    f.write(dbt_project_yml_content)\n\n# Create a sample model\nmodel_sql_content = \"\"\"\n-- models/my_first_model.sql\nSELECT 1 AS id, 'hello from dbt-athena' AS message\n\"\"\"\nwith open(project_dir / \"models\" / \"my_first_model.sql\", \"w\") as f:\n    f.write(model_sql_content)\n\nprint(f\"dbt project created at: {project_dir}\")\n\n# Attempt to run dbt (this requires dbt-core to be installed in the environment)\ntry:\n    print(\"\\nAttempting to run dbt...\")\n    # Set DBT_PROFILES_DIR for the subprocess to use our temp profiles.yml\n    env_vars = os.environ.copy()\n    env_vars[\"DBT_PROFILES_DIR\"] = str(temp_dbt_dir.resolve())\n    \n    result = subprocess.run(\n        [\"dbt\", \"run\", \"--project-dir\", str(project_dir.resolve())],\n        check=True,\n        capture_output=True,\n        text=True,\n        env=env_vars\n    )\n    print(\"dbt run successful!\")\n    print(result.stdout)\nexcept subprocess.CalledProcessError as e:\n    print(f\"dbt run failed with exit code {e.returncode}: {e}\")\n    print(\"Stdout:\", e.stdout)\n    print(\"Stderr:\", e.stderr)\nexcept FileNotFoundError:\n    print(\"Error: 'dbt' command not found. Ensure dbt-core is installed (pip install dbt-core).\")\nfinally:\n    # Clean up temporary files/directories\n    if temp_dbt_dir.exists():\n        shutil.rmtree(temp_dbt_dir)\n        print(f\"Cleaned up {temp_dbt_dir}\")\n    if project_dir.exists():\n        shutil.rmtree(project_dir)\n        print(f\"Cleaned up {project_dir}\")","lang":"python","description":"This quickstart demonstrates how to configure `dbt-athena-community` using a `profiles.yml` file and run a sample dbt model. It creates temporary `profiles.yml` and `dbt_project.yml` files and then executes a `dbt run` command via `subprocess`. You need to replace placeholder AWS credentials/S3 path or ensure your environment has `~/.aws/credentials` configured. Ensure `dbt-core` is installed in your environment for the `dbt` CLI command to be found."},"warnings":[{"fix":"Always ensure `dbt-athena-community` and `dbt-core` major versions match (e.g., `dbt-core~=1.10.0` requires `dbt-athena-community~=1.10.0`). Check the adapter's `pyproject.toml` or documentation for exact compatibility.","message":"Adapter version must match `dbt-core` major version. `dbt-athena-community` versions are tightly coupled with `dbt-core`. A common footgun is upgrading `dbt-core` without upgrading the adapter.","severity":"breaking","affected_versions":"All versions"},{"fix":"Ensure the `s3_staging_dir` is correctly configured in your `profiles.yml` and that the AWS IAM role/credentials used have write permissions to this S3 location. Forgetting this or misconfiguring permissions is a very common setup error.","message":"The `s3_staging_dir` in `profiles.yml` is a mandatory configuration. All queries executed by dbt-athena-community require this S3 path.","severity":"gotcha","affected_versions":"All versions"},{"fix":"While `view` materialization is common, using `table` materialization without proper partitioning, bucketing, or `iceberg` table strategies can lead to high Athena costs and slow build times. Consider incremental models and `iceberg` tables for production use cases with large data volumes.","message":"Performance and cost considerations with `table` materialization on large datasets.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Users sometimes confuse `dbt-athena-community` with the official `dbt-athena` adapter maintained by dbt Labs (found in `dbt-adapters` repo). While `dbt-athena-community` is widely used and often preferred for its features, ensure you're installing and referencing the correct package. This registry entry specifically pertains to `dbt-athena-community`.","message":"Confusion between `dbt-athena-community` and the official `dbt-athena` adapter.","severity":"gotcha","affected_versions":"All versions"},{"fix":"`dbt-athena-community` depends on specific versions of `boto3` and `pyathena`. Conflicts with other libraries in the same Python environment requiring different versions of these can lead to runtime errors. It's often best practice to use a dedicated virtual environment for dbt projects to isolate dependencies.","message":"Potential `boto3` and `pyathena` version conflicts with other libraries.","severity":"gotcha","affected_versions":"All versions"}],"env_vars":null,"last_verified":"2026-04-12T00:00:00.000Z","next_check":"2026-07-11T00:00:00.000Z"}