{"id":792,"library":"pandas-gbq","title":"pandas-gbq","description":"pandas-gbq is a Python library that provides a convenient interface to connect pandas DataFrames with Google BigQuery. It simplifies reading data from BigQuery into a pandas.DataFrame and writing DataFrames to BigQuery tables. The current version is 0.34.1, released on 2026-03-26, and the library maintains a regular release cadence, typically with monthly or bi-monthly updates for new features and bug fixes.","status":"active","version":"0.34.1","language":"python","source_language":"en","source_url":"https://github.com/googleapis/google-cloud-python/tree/main/packages/pandas-gbq","tags":["bigquery","pandas","data-science","google-cloud","etl","data-warehouse"],"install":[{"cmd":"pip install pandas-gbq","lang":"bash","label":"Install latest version"}],"dependencies":[{"reason":"Core DataFrame manipulation library.","package":"pandas"},{"reason":"Google Cloud client library for BigQuery API interactions.","package":"google-cloud-bigquery"},{"reason":"Authentication and authorization for Google's APIs.","package":"google-auth"},{"reason":"Helpers for user-based authentication to Google's API.","package":"pydata-google-auth"},{"reason":"Used for efficient data formatting and transfer, especially with the BigQuery Storage API.","package":"pyarrow","optional":true},{"reason":"Client library for the BigQuery Storage API, enabling faster large data downloads.","package":"google-cloud-bigquery-storage","optional":true},{"reason":"Provides progress bars for data uploads/downloads.","package":"tqdm","optional":true}],"imports":[{"symbol":"read_gbq","correct":"import pandas_gbq\ndf = pandas_gbq.read_gbq(query, project_id=project_id)"},{"symbol":"to_gbq","correct":"import pandas_gbq\npandas_gbq.to_gbq(df, destination_table, project_id=project_id)"},{"note":"The `pandas.io.gbq` module was deprecated in pandas 3.0. Users should directly import and use `pandas_gbq.read_gbq` and `pandas_gbq.to_gbq` from the `pandas-gbq` library.","wrong":"import pandas as pd\npd.io.gbq.read_gbq(...)","symbol":"pd.io.gbq","correct":"import pandas_gbq"}],"quickstart":{"code":"import os\nimport pandas as pd\nimport pandas_gbq\n\n# Set your Google Cloud Project ID\n# It's recommended to set this as an environment variable or via credentials\nproject_id = os.environ.get('GOOGLE_CLOUD_PROJECT', 'your-gcp-project-id')\n\n# --- Reading data from BigQuery ---\n# Example query from a public dataset\nsql_query = \"\"\"\n    SELECT country_name, alpha_2_code\n    FROM `bigquery-public-data.utility_us.country_code_iso`\n    WHERE alpha_2_code LIKE 'U%'\n    LIMIT 5\n\"\"\"\n\ntry:\n    df_read = pandas_gbq.read_gbq(sql_query, project_id=project_id)\n    print(\"\\n--- Data read from BigQuery ---\")\n    print(df_read)\nexcept Exception as e:\n    print(f\"Error reading from BigQuery: {e}\")\n    print(\"Please ensure GOOGLE_CLOUD_PROJECT is set and you have authenticated (e.g., `gcloud auth application-default login`).\")\n\n# --- Writing data to BigQuery ---\n# Create a sample DataFrame to upload\ndata = {\n    'col1': [1, 2, 3],\n    'col2': ['A', 'B', 'C'],\n    'timestamp_col': pd.to_datetime(['2026-01-01', '2026-01-02', '2026-01-03'])\n}\ndf_write = pd.DataFrame(data)\n\n# Define destination table (dataset.tablename)\ndestination_table = 'my_test_dataset.my_test_table'\n\n# To avoid errors, you might want to replace the table if it exists for testing\n# In production, consider 'append' or 'fail' with proper checks\ntry:\n    pandas_gbq.to_gbq(\n        df_write,\n        destination_table,\n        project_id=project_id,\n        if_exists='replace' # Options: 'fail', 'replace', 'append'\n    )\n    print(f\"\\n--- DataFrame successfully written to {destination_table} in project {project_id} ---\")\nexcept Exception as e:\n    print(f\"Error writing to BigQuery: {e}\")\n    print(\"Ensure 'my_test_dataset' exists in BigQuery or remove 'my_test_dataset.' from 'destination_table' to allow automatic dataset creation if permitted.\")\n","lang":"python","description":"This quickstart demonstrates how to read data from a public BigQuery dataset into a pandas DataFrame and write a pandas DataFrame to a new BigQuery table. It assumes you have a Google Cloud project set up and have authenticated (e.g., using `gcloud auth application-default login`). The `project_id` is retrieved from the `GOOGLE_CLOUD_PROJECT` environment variable for robustness."},"warnings":[{"fix":"Upgrade to Python 3.9+ and ensure all project dependencies are compatible.","message":"Python 2 support was officially dropped as of January 1, 2020. Versions released after this date require Python 3.9 or higher.","severity":"breaking","affected_versions":"0.20.0 and later"},{"fix":"Set the `GOOGLE_CLOUD_PROJECT` environment variable. Authenticate using `gcloud auth application-default login`, provide a service account JSON file via the `credentials` parameter, or set `pandas_gbq.context.credentials` and `pandas_gbq.context.project` explicitly.","message":"Authentication is critical. Without proper credentials or a `project_id`, `pandas-gbq` will raise errors (e.g., `ValueError: Could not determine project ID`). Common authentication methods include Application Default Credentials (ADC), service account keys, or user-based OAuth.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Review and update BigQuery table schemas if necessary. For `datetime` columns, consider making them timezone-aware (`pd.to_datetime(..., utc=True)`) if `TIMESTAMP` is desired, or explicitly define `table_schema` in `to_gbq`.","message":"The `to_gbq` function has breaking changes in how it infers BigQuery data types for certain pandas dtypes. Naive (timezone-unaware) datetime columns are now loaded as BigQuery `DATETIME` instead of `TIMESTAMP`. Object columns containing boolean or dictionary values are loaded as `BOOLEAN` or `STRUCT` respectively, instead of `STRING`. `UInt8` columns are now `INT64`.","severity":"breaking","affected_versions":"0.34.0 and later"},{"fix":"Explicitly set `if_exists='replace'` to overwrite the table, or `if_exists='append'` to add data to an existing table. Always handle this parameter carefully to prevent unintended data loss or duplication.","message":"When using `to_gbq`, the default `if_exists` parameter is 'fail', meaning the operation will fail if the destination table already exists.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Ensure your environment allows for the local webserver flow (e.g., a browser can open `localhost:808X`). If working in a headless environment, consider using service account authentication.","message":"The `auth_local_webserver` parameter's default behavior changed from `False` to `True` in `pandas-gbq` version 1.5.0. This is due to Google deprecating the 'out-of-band' (copy-paste) authentication flow.","severity":"deprecated","affected_versions":"1.5.0 and later"}],"env_vars":null,"last_verified":"2026-05-12T19:13:12.303Z","next_check":"2026-06-27T00:00:00.000Z","problems":[{"fix":"Install the library using pip: `pip install pandas-gbq`","cause":"The `pandas-gbq` library has not been installed or is not accessible in the current Python environment.","error":"ModuleNotFoundError: No module named 'pandas_gbq'"},{"fix":"Provide credentials explicitly, for example, using a service account key file: `from google.oauth2 import service_account; credentials = service_account.Credentials.from_service_account_file('path/to/key.json'); df = pandas_gbq.read_gbq(sql, project_id='your-project-id', credentials=credentials)` or set the `GOOGLE_APPLICATION_CREDENTIALS` environment variable to the path of your service account key file.","cause":"The `pandas-gbq` library could not find valid Google Cloud credentials in the environment to authenticate with BigQuery.","error":"google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or provide credentials explicitly."},{"fix":"Ensure your DataFrame's column names, their order, and data types precisely match the BigQuery table's schema. You can also explicitly define the `table_schema` argument in `pandas_gbq.to_gbq()` to specify the BigQuery schema or use `if_exists='replace'` if you intend to overwrite the table and its schema.","cause":"The DataFrame being written to BigQuery has a schema (column names, order, or data types) that does not match the schema of the existing destination table.","error":"pandas_gbq.gbq.InvalidSchema: Please verify that the structure and data types in the DataFrame match the schema of the destination table."},{"fix":"Verify that the `project_id`, dataset, and table names are spelled correctly and exist in your Google Cloud Project. Additionally, ensure the authenticated user or service account has `bigquery.tables.get` and `bigquery.dataViewer` (or equivalent) permissions for the resource.","cause":"The specified BigQuery project, dataset, or table does not exist, or the user lacks the necessary permissions to access it.","error":"google.api_core.exceptions.NotFound: 404 Not Found: Table project_id:dataset.table_name not found."}],"ecosystem":"pypi","meta_description":null,"install_score":95,"install_tag":"verified","quickstart_score":60,"quickstart_tag":"reviewed","pypi_latest":"0.35.0","cli_name":null,"install_checks":{"last_tested":"2026-05-12","tag":"verified","tag_description":"installs cleanly on critical runtimes, fast import, recently tested","results":[{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":" $EXIT -eq 0 ","exit_code":0,"wheel_type":"sdist","failure_reason":null,"install_time_s":null,"import_time_s":3.16,"mem_mb":51.3,"disk_size":"396.7M"},{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":2.57,"mem_mb":50.9,"disk_size":"390.1M"},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":" $EXIT -eq 0 ","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":15,"import_time_s":2.04,"mem_mb":50,"disk_size":"362M"},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":1.86,"mem_mb":49.6,"disk_size":"357M"},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":" $EXIT -eq 0 ","exit_code":0,"wheel_type":"sdist","failure_reason":null,"install_time_s":null,"import_time_s":3.71,"mem_mb":57.7,"disk_size":"424.0M"},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":4.5,"mem_mb":57.4,"disk_size":"417.3M"},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":" $EXIT -eq 0 ","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":14.4,"import_time_s":2.91,"mem_mb":56.5,"disk_size":"389M"},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":2.9,"mem_mb":56.2,"disk_size":"383M"},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":" $EXIT -eq 0 ","exit_code":0,"wheel_type":"sdist","failure_reason":null,"install_time_s":null,"import_time_s":3.78,"mem_mb":56.5,"disk_size":"416.8M"},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":4.6,"mem_mb":56.3,"disk_size":"410.1M"},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":" $EXIT -eq 0 ","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":14.7,"import_time_s":3.35,"mem_mb":55.3,"disk_size":"382M"},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":3.72,"mem_mb":55.1,"disk_size":"376M"},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":" $EXIT -eq 0 ","exit_code":0,"wheel_type":"sdist","failure_reason":null,"install_time_s":null,"import_time_s":3.52,"mem_mb":56.6,"disk_size":"415.9M"},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":4.29,"mem_mb":56.3,"disk_size":"409.1M"},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":" $EXIT -eq 0 ","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":13.9,"import_time_s":3.02,"mem_mb":55.3,"disk_size":"381M"},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":4.31,"mem_mb":55.1,"disk_size":"375M"},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":" $EXIT -eq 0 ","exit_code":0,"wheel_type":"sdist","failure_reason":null,"install_time_s":null,"import_time_s":2.56,"mem_mb":51.7,"disk_size":"386.3M"},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":2.43,"mem_mb":51.5,"disk_size":"385.0M"},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":" $EXIT -eq 0 ","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":17.9,"import_time_s":2.41,"mem_mb":50.4,"disk_size":"361M"},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":2.11,"mem_mb":50.3,"disk_size":"359M"}]},"quickstart_checks":{"last_tested":"2026-04-24","tag":"reviewed","tag_description":"minor failures on some runtimes or slightly older test data","results":[{"runtime":"python:3.10-alpine","exit_code":0},{"runtime":"python:3.10-slim","exit_code":0},{"runtime":"python:3.11-alpine","exit_code":0},{"runtime":"python:3.11-slim","exit_code":0},{"runtime":"python:3.12-alpine","exit_code":0},{"runtime":"python:3.12-slim","exit_code":-1},{"runtime":"python:3.13-alpine","exit_code":0},{"runtime":"python:3.13-slim","exit_code":0},{"runtime":"python:3.9-alpine","exit_code":0},{"runtime":"python:3.9-slim","exit_code":0}]}}