Google Cloud Storage Emulator

Common errors

• requests.exceptions.ConnectionError: HTTPConnectionPool(host='0.0.0.0', port=9023): Max retries exceeded with url: /download/storage/v1/b/...

  cause The emulator might be returning an internal `0.0.0.0` IP for callbacks (e.g., during resumable uploads or downloads) which is not reachable by the client, especially when running the emulator in Docker.
  fix Ensure `STORAGE_EMULATOR_HOST` is set to an address resolvable by the client (e.g., `http://localhost:9023` or `http://host.docker.internal:9023` for Docker). Check GitHub issues for specific host configuration workarounds, such as explicitly binding the emulator to a specific IP or hostname.

• BlobNotFoundError: 404 Not Found

  cause This can occur if the blob was uploaded but not properly stored by the emulator, or if the client's configuration for the emulator is incorrect, causing it to look in the wrong place (e.g., a different emulator instance or the real GCS).
  fix Verify that `STORAGE_EMULATOR_HOST` is correctly set and that the emulator is running and accessible on the specified host and port. Check emulator logs for upload errors. Confirm that the bucket and blob names used in the client match those in the emulator.

• Custom metadata is not stored when a new object is uploaded.

  cause The emulator's implementation for custom metadata storage might be incomplete or not fully compliant with the Google Cloud Storage API.
  fix This is a known limitation or open issue in the emulator. You may need to adapt your tests to not rely on custom metadata or monitor the project's GitHub issues for updates on this feature.

Warnings

• breaking The `gcp-storage-emulator` only supports a limited subset of the full Google Cloud Storage API. Advanced features, specific HTTP headers, or less common API methods might not be fully emulated, leading to unexpected behavior or errors.
  Fix: Consult the project's GitHub README and open issues for supported features. For critical missing features, consider contributing or using a more comprehensive solution like MinIO configured for GCS interoperability if S3 API compatibility is acceptable.
• gotcha When running the emulator in Docker, especially if the client is outside the Docker network, resumable uploads or blob downloads might fail with connection errors (e.g., `host='0.0.0.0'`) or 404s due to incorrect hostname resolution in callback URLs.
  Fix: Ensure the `STORAGE_EMULATOR_HOST` environment variable points to a resolvable IP or hostname (e.g., `http://localhost:9023` or `http://host.docker.internal:9023` if using Docker Compose) that the client can reach. Check GitHub issues for specific workarounds related to `0.0.0.0` hostnames in callback URLs.
• gotcha By default, the emulator persists data to a local `.cloudstorage` directory relative to the current working directory. This can lead to unexpected state between test runs or local development sessions.
  Fix: For ephemeral testing, use the `--in-memory` CLI flag or `in_memory=True` when calling `create_server()`. For controlled persistence, specify `STORAGE_BASE` and `STORAGE_DIR` environment variables or use the `gcp-storage-emulator wipe` command.
• gotcha The `google-cloud-storage` client library automatically detects the emulator through the `STORAGE_EMULATOR_HOST` environment variable. If this variable is unset or points to an incorrect address, the client will attempt to connect to the production Google Cloud Storage API, potentially causing authentication errors or unexpected cloud costs.
  Fix: Always set `STORAGE_EMULATOR_HOST` to the emulator's address (e.g., `http://localhost:9023`) before initializing `google.cloud.storage.Client`. Use `os.environ.setdefault()` for flexible configuration.

Install

• pip install gcp-storage-emulator Install with pip

Imports

• create_server
  wrong:

  from gcloud_storage_emulator.server import create_server

  correct:

  from gcp_storage_emulator.server import create_server

  The package name for the server module is `gcp_storage_emulator`, not `gcloud_storage_emulator` (an older naming convention).
• storage.Client

  from google.cloud import storage

  This is the standard import for the official Google Cloud Storage client library, which you configure to point to the emulator.

Quickstart

This quickstart demonstrates how to programmatically start the `gcp-storage-emulator`, configure the `google-cloud-storage` client to interact with it, and perform basic operations like creating a bucket, uploading a blob, and downloading its content. The emulator is configured for in-memory storage for ephemeral testing.

import os
from google.cloud import storage, exceptions
from gcp_storage_emulator.server import create_server

HOST = "localhost"
PORT = 9023
BUCKET_NAME = "my-test-bucket"
FILE_NAME = "test-blob.txt"
CONTENT = b"Hello, GCS Emulator!"

# 1. Start the emulator server
# The default_bucket parameter creates the bucket automatically upon server start.
server = create_server(HOST, PORT, in_memory=True, default_bucket=BUCKET_NAME)
server.start()

try:
    # 2. Configure the Google Cloud Storage client to use the emulator
    # Use os.environ.setdefault to allow external configuration.
    os.environ.setdefault("STORAGE_EMULATOR_HOST", f"http://{HOST}:{PORT}")
    
    # 3. Create a client. Project ID is arbitrary for the emulator.
    client = storage.Client(project="test-project")
    
    # 4. Get the bucket (created by default_bucket in server or create manually)
    bucket = client.bucket(BUCKET_NAME)
    
    # Ensure the bucket exists (useful if not using default_bucket param)
    try:
        bucket.create()
        print(f"Bucket '{BUCKET_NAME}' created.")
    except exceptions.Conflict: # Bucket already exists
        print(f"Bucket '{BUCKET_NAME}' already exists.")
        pass
    
    # 5. Upload a blob
    blob = bucket.blob(FILE_NAME)
    blob.upload_from_string(CONTENT)
    print(f"Uploaded '{FILE_NAME}' with content: {CONTENT.decode()}.")
    
    # 6. Download the blob
    downloaded_content = blob.download_as_bytes()
    print(f"Downloaded '{FILE_NAME}' with content: {downloaded_content.decode()}.")

    # 7. List blobs in the bucket
    print(f"Blobs in '{BUCKET_NAME}':")
    for listed_blob in bucket.list_blobs():
        print(f" - {listed_blob.name}")

finally:
    # 8. Stop the emulator server
    server.stop()
    print("GCP Storage Emulator stopped.")
    # Clean up the environment variable if needed, though usually not critical after stop
    if "STORAGE_EMULATOR_HOST" in os.environ:
        del os.environ["STORAGE_EMULATOR_HOST"]