BigQuery DataFrames (bigframes)

Warnings

• breaking In BigQuery DataFrames v2.0+, the default for `allow_large_results` changed from `True` to `False` for methods that return results to the client (e.g., `peek()`, `to_pandas()`, `to_pandas_batches()`). This can lead to 'BigQuery has a maximum response size limit' errors for large results.
  Fix: Set `bpd.options.bigquery.allow_large_results = True` or pass `allow_large_results=True` directly to the method, e.g., `df.to_pandas(allow_large_results=True)`.
• gotcha BigQuery DataFrames uses lazy evaluation. Operations are generally not executed immediately but are instead translated into BigQuery SQL and run only when results are explicitly requested (e.g., by calling `head()`, `to_pandas()`, `to_arrow()`, `plot()`, or printing the DataFrame/Series).
  Fix: Understand that BigQuery DataFrames operations build a query plan. Use methods like `head()`, `to_pandas()`, or printing the object to trigger execution and retrieve results.
• gotcha Converting large BigQuery DataFrames to pandas DataFrames using `to_pandas()` can lead to out-of-memory errors on the client side, as it pulls all data into local memory. This negates the scalability benefits of BigQuery DataFrames.
  Fix: Avoid converting large DataFrames to pandas locally. Perform aggregations, filtering, and transformations using BigQuery DataFrames APIs first. Only use `to_pandas()` on small, already reduced datasets, or when absolutely necessary.
• gotcha BigQuery DataFrames stores temporary data (e.g., intermediate results) in BigQuery tables within your specified project. These tables persist for seven days by default in `_anonymous_` datasets, incurring storage costs.
  Fix: Be aware of potential BigQuery storage costs. For long-running or frequently used temporary results, consider managing them explicitly. You can close sessions using `bpd.close_session()` to potentially clean up temporary resources faster, though tables persist for 7 days.
• gotcha When using `read_gbq()`, if your BigQuery dataset is not located in the default 'US' multi-region, you must explicitly set the location using `bpd.options.bigquery.location` or a `NotFound` exception will occur.
  Fix: Set `bpd.options.bigquery.location = "YOUR_REGION"` (e.g., "EU", "asia-east1") before calling `read_gbq()` if your data resides outside the 'US' multi-region.

Install

• pip install bigframes Install latest version

Imports

• bigframes.pandas

  import bigframes.pandas as bpd

  The primary entry point for the pandas-like API.
• bigframes.ml

  import bigframes.ml

  Used for scikit-learn-like machine learning APIs.
• bigframes.bigquery

  import bigframes.bigquery

  Provides access to BigQuery SQL functions without pandas equivalents.

Quickstart

This quickstart demonstrates how to initialize BigQuery DataFrames with your GCP project ID and load data from a public BigQuery table. It then performs a basic operation (`head()`) to trigger query execution and display results. Ensure you have authenticated to Google Cloud and enabled the BigQuery API for your project.

import bigframes.pandas as bpd
import os

# Set your GCP Project ID. Ensure the BigQuery API is enabled for this project.
# For local development, authenticate using `gcloud auth application-default login`.
PROJECT_ID = os.environ.get('GCP_PROJECT_ID', 'your-gcp-project-id') 

bpd.options.bigquery.project = PROJECT_ID
# bpd.options.bigquery.location = "US" # Uncomment and set if your dataset is not in US multi-region
# bpd.options.bigquery.ordering_mode = "partial" # Recommended for performance

# Load a public BigQuery dataset into a BigQuery DataFrame
df = bpd.read_gbq("bigquery-public-data.ml_datasets.penguins")

# Perform a simple operation and display the head (triggers computation)
print(df.head())