TensorFlow I/O GCS Filesystem

Warnings

• breaking TensorFlow I/O (and its sub-packages like GCS Filesystem) are often tightly coupled to specific TensorFlow versions. Mismatches can lead to runtime errors or build issues.
  Fix: Always install `tensorflow-io-gcs-filesystem` with a `tensorflow` version that is explicitly supported by the `tensorflow-io` release notes. For example, v0.37.0 supports TensorFlow 2.16. Check the release notes or the official TensorFlow I/O documentation for compatibility matrices.
• gotcha This package (`tensorflow-io-gcs-filesystem`) provides *only* GCS filesystem support. It does not include other features found in the full `tensorflow-io` meta-package (e.g., other file formats like Avro, or other filesystems like S3).
  Fix: If you need broader TensorFlow I/O functionality beyond GCS filesystem operations, install the full `tensorflow-io` package (`pip install tensorflow-io`). If only GCS is needed, `tensorflow-io-gcs-filesystem` is sufficient and lighter.
• gotcha Proper Google Cloud Storage authentication is crucial. Operations will fail with permission errors if authentication is not correctly configured in the environment where your TensorFlow code runs.
  Fix: Ensure your environment has access to GCS. Common methods include `gcloud auth application-default login`, setting the `GOOGLE_APPLICATION_CREDENTIALS` environment variable to a service account key file, or running on a GCE/GKE instance with appropriate service account scopes.
• gotcha The GCS filesystem handler is registered by importing `tensorflow_io_gcs_filesystem`. If this import is omitted, `tf.io.gfile` will not recognize `gs://` paths.
  Fix: Always include `import tensorflow_io_gcs_filesystem as _` at the beginning of your script or module where you intend to use `tf.io.gfile` with GCS paths.
• gotcha The library requires Python versions >=3.7 and <3.13.
  Fix: Ensure your Python environment is within the supported range. Check the `requires_python` field on PyPI for the exact specification.

Install

• pip install tensorflow-io-gcs-filesystem Install GCS Filesystem

Imports

• GCS Filesystem Registration

  import tensorflow_io_gcs_filesystem as _

  The `as _` convention is commonly used to indicate that the module is imported for its side effects (registering the GCS filesystem handler) rather than for direct symbol usage.

Quickstart

This quickstart demonstrates how to use `tensorflow-io-gcs-filesystem` to enable standard `tf.io.gfile` operations (write, read, list, delete) on Google Cloud Storage paths (starting with `gs://`). It requires prior GCS authentication and a GCS bucket.

import tensorflow as tf
import tensorflow_io_gcs_filesystem as _  # Registers the GCS filesystem handler
import os

# Ensure you have GCS authentication set up, e.g.,
# by running `gcloud auth application-default login`
# or setting the GOOGLE_APPLICATION_CREDENTIALS environment variable.
# For testing, you might use a public bucket or ensure permissions are granted.

bucket_name = os.environ.get('GCS_TEST_BUCKET', 'your-gcs-bucket-name') # Replace with your bucket
file_name = "registry_test_file.txt"
file_path = f"gs://{bucket_name}/{file_name}"

print(f"Attempting to interact with GCS path: {file_path}")

try:
    # 1. Write a file to GCS
    with tf.io.gfile.GFile(file_path, 'w') as f:
        f.write("Hello from TensorFlow I/O GCS Filesystem registry check!")
    print(f"Successfully wrote to {file_path}")

    # 2. Read the file from GCS
    with tf.io.gfile.GFile(file_path, 'r') as f:
        content = f.read()
    print(f"Successfully read from {file_path}. Content: '{content}'")

    # 3. List files in the directory (or bucket prefix)
    parent_path = f"gs://{bucket_name}"
    print(f"Listing contents of {parent_path}:")
    for item in tf.io.gfile.listdir(parent_path):
        print(f"  - {item}")

    # 4. Check if the file exists
    exists = tf.io.gfile.exists(file_path)
    print(f"File {file_path} exists: {exists}")

    # 5. Delete the file from GCS
    tf.io.gfile.remove(file_path)
    print(f"Successfully deleted {file_path}")

    # Verify deletion
    exists_after_delete = tf.io.gfile.exists(file_path)
    print(f"File {file_path} exists after delete: {exists_after_delete}")

except tf.errors.FailedPreconditionError as e:
    print(f"Error: Failed to connect to GCS. Please ensure you have authenticated and your bucket '{bucket_name}' exists and is accessible. Error details: {e}")
    print("HINT: Try running `gcloud auth application-default login` or setting `GOOGLE_APPLICATION_CREDENTIALS`.")
except tf.errors.NotFoundError as e:
    print(f"Error: GCS path not found. Please check bucket name and permissions. Error details: {e}")
except Exception as e:
    print(f"An unexpected error occurred: {e}")