{"id":9002,"library":"gcp-storage-emulator","title":"Google Cloud Storage Emulator","description":"The `gcp-storage-emulator` is a Python library that provides a stub emulator for the Google Cloud Storage API, enabling local development and testing without requiring a connection to the actual Google Cloud Storage service. It's actively maintained, with frequent releases, and is currently at version 2024.8.3. This emulator is particularly useful for accelerating development cycles and running integration tests locally.","status":"active","version":"2024.8.3","language":"en","source_language":"en","source_url":"https://github.com/oittaa/gcp-storage-emulator","tags":["GCP","Google Cloud Storage","emulator","testing","local development","cloud"],"install":[{"cmd":"pip install gcp-storage-emulator","lang":"bash","label":"Install with pip"}],"dependencies":[{"reason":"Required for the client library to interact with the emulator. The emulator is designed to work with Google's official client libraries.","package":"google-cloud-storage","optional":false}],"imports":[{"note":"The package name for the server module is `gcp_storage_emulator`, not `gcloud_storage_emulator` (an older naming convention).","wrong":"from gcloud_storage_emulator.server import create_server","symbol":"create_server","correct":"from gcp_storage_emulator.server import create_server"},{"note":"This is the standard import for the official Google Cloud Storage client library, which you configure to point to the emulator.","symbol":"storage.Client","correct":"from google.cloud import storage"}],"quickstart":{"code":"import os\nfrom google.cloud import storage, exceptions\nfrom gcp_storage_emulator.server import create_server\n\nHOST = \"localhost\"\nPORT = 9023\nBUCKET_NAME = \"my-test-bucket\"\nFILE_NAME = \"test-blob.txt\"\nCONTENT = b\"Hello, GCS Emulator!\"\n\n# 1. Start the emulator server\n# The default_bucket parameter creates the bucket automatically upon server start.\nserver = create_server(HOST, PORT, in_memory=True, default_bucket=BUCKET_NAME)\nserver.start()\n\ntry:\n    # 2. Configure the Google Cloud Storage client to use the emulator\n    # Use os.environ.setdefault to allow external configuration.\n    os.environ.setdefault(\"STORAGE_EMULATOR_HOST\", f\"http://{HOST}:{PORT}\")\n    \n    # 3. Create a client. Project ID is arbitrary for the emulator.\n    client = storage.Client(project=\"test-project\")\n    \n    # 4. Get the bucket (created by default_bucket in server or create manually)\n    bucket = client.bucket(BUCKET_NAME)\n    \n    # Ensure the bucket exists (useful if not using default_bucket param)\n    try:\n        bucket.create()\n        print(f\"Bucket '{BUCKET_NAME}' created.\")\n    except exceptions.Conflict: # Bucket already exists\n        print(f\"Bucket '{BUCKET_NAME}' already exists.\")\n        pass\n    \n    # 5. Upload a blob\n    blob = bucket.blob(FILE_NAME)\n    blob.upload_from_string(CONTENT)\n    print(f\"Uploaded '{FILE_NAME}' with content: {CONTENT.decode()}.\")\n    \n    # 6. Download the blob\n    downloaded_content = blob.download_as_bytes()\n    print(f\"Downloaded '{FILE_NAME}' with content: {downloaded_content.decode()}.\")\n\n    # 7. List blobs in the bucket\n    print(f\"Blobs in '{BUCKET_NAME}':\")\n    for listed_blob in bucket.list_blobs():\n        print(f\" - {listed_blob.name}\")\n\nfinally:\n    # 8. Stop the emulator server\n    server.stop()\n    print(\"GCP Storage Emulator stopped.\")\n    # Clean up the environment variable if needed, though usually not critical after stop\n    if \"STORAGE_EMULATOR_HOST\" in os.environ:\n        del os.environ[\"STORAGE_EMULATOR_HOST\"]","lang":"python","description":"This quickstart demonstrates how to programmatically start the `gcp-storage-emulator`, configure the `google-cloud-storage` client to interact with it, and perform basic operations like creating a bucket, uploading a blob, and downloading its content. The emulator is configured for in-memory storage for ephemeral testing."},"warnings":[{"fix":"Consult the project's GitHub README and open issues for supported features. For critical missing features, consider contributing or using a more comprehensive solution like MinIO configured for GCS interoperability if S3 API compatibility is acceptable.","message":"The `gcp-storage-emulator` only supports a limited subset of the full Google Cloud Storage API. Advanced features, specific HTTP headers, or less common API methods might not be fully emulated, leading to unexpected behavior or errors.","severity":"breaking","affected_versions":"<=2024.8.3"},{"fix":"Ensure the `STORAGE_EMULATOR_HOST` environment variable points to a resolvable IP or hostname (e.g., `http://localhost:9023` or `http://host.docker.internal:9023` if using Docker Compose) that the client can reach. Check GitHub issues for specific workarounds related to `0.0.0.0` hostnames in callback URLs.","message":"When running the emulator in Docker, especially if the client is outside the Docker network, resumable uploads or blob downloads might fail with connection errors (e.g., `host='0.0.0.0'`) or 404s due to incorrect hostname resolution in callback URLs.","severity":"gotcha","affected_versions":"<=2024.8.3"},{"fix":"For ephemeral testing, use the `--in-memory` CLI flag or `in_memory=True` when calling `create_server()`. For controlled persistence, specify `STORAGE_BASE` and `STORAGE_DIR` environment variables or use the `gcp-storage-emulator wipe` command.","message":"By default, the emulator persists data to a local `.cloudstorage` directory relative to the current working directory. This can lead to unexpected state between test runs or local development sessions.","severity":"gotcha","affected_versions":"<=2024.8.3"},{"fix":"Always set `STORAGE_EMULATOR_HOST` to the emulator's address (e.g., `http://localhost:9023`) before initializing `google.cloud.storage.Client`. Use `os.environ.setdefault()` for flexible configuration.","message":"The `google-cloud-storage` client library automatically detects the emulator through the `STORAGE_EMULATOR_HOST` environment variable. If this variable is unset or points to an incorrect address, the client will attempt to connect to the production Google Cloud Storage API, potentially causing authentication errors or unexpected cloud costs.","severity":"gotcha","affected_versions":"<=2024.8.3"}],"env_vars":null,"last_verified":"2026-04-16T00:00:00.000Z","next_check":"2026-07-15T00:00:00.000Z","problems":[{"fix":"Ensure `STORAGE_EMULATOR_HOST` is set to an address resolvable by the client (e.g., `http://localhost:9023` or `http://host.docker.internal:9023` for Docker). Check GitHub issues for specific host configuration workarounds, such as explicitly binding the emulator to a specific IP or hostname.","cause":"The emulator might be returning an internal `0.0.0.0` IP for callbacks (e.g., during resumable uploads or downloads) which is not reachable by the client, especially when running the emulator in Docker.","error":"requests.exceptions.ConnectionError: HTTPConnectionPool(host='0.0.0.0', port=9023): Max retries exceeded with url: /download/storage/v1/b/..."},{"fix":"Verify that `STORAGE_EMULATOR_HOST` is correctly set and that the emulator is running and accessible on the specified host and port. Check emulator logs for upload errors. Confirm that the bucket and blob names used in the client match those in the emulator.","cause":"This can occur if the blob was uploaded but not properly stored by the emulator, or if the client's configuration for the emulator is incorrect, causing it to look in the wrong place (e.g., a different emulator instance or the real GCS).","error":"BlobNotFoundError: 404 Not Found"},{"fix":"This is a known limitation or open issue in the emulator. You may need to adapt your tests to not rely on custom metadata or monitor the project's GitHub issues for updates on this feature.","cause":"The emulator's implementation for custom metadata storage might be incomplete or not fully compliant with the Google Cloud Storage API.","error":"Custom metadata is not stored when a new object is uploaded."}]}