{"id":2308,"library":"tensorstore","title":"TensorStore","description":"TensorStore is an open-source C++ and Python library (version 0.1.82) designed for efficiently reading and writing large, multi-dimensional arrays. It provides a uniform API for various array formats (like Zarr, N5, Neuroglancer precomputed) and storage systems (local filesystems, Google Cloud Storage, Amazon S3-compatible object stores, HTTP servers, and in-memory storage). It features an asynchronous API, read/writeback caching, transactions with strong ACID guarantees, and optimistic concurrency for safe multi-process/machine access. The project maintains an active release cadence with frequent updates.","status":"active","version":"0.1.82","language":"en","source_language":"en","source_url":"https://github.com/google/tensorstore","tags":["array","scientific-computing","data-storage","multi-dimensional","asynchronous","HPC"],"install":[{"cmd":"pip install tensorstore","lang":"bash","label":"PyPI"},{"cmd":"conda install tensorstore -c conda-forge","lang":"bash","label":"Conda-Forge"}],"dependencies":[{"reason":"Requires Python 3.11 or later.","package":"python","optional":false},{"reason":"Typically used for array manipulation with TensorStore objects.","package":"numpy","optional":true}],"imports":[{"symbol":"tensorstore","correct":"import tensorstore as ts"}],"quickstart":{"code":"import tensorstore as ts\nimport numpy as np\nimport os\n\n# Define a temporary directory for local storage\noutput_dir = 'tmp_tensorstore_dataset'\n\n# Create a new N5 dataset on the local filesystem\ndataset = ts.open({\n    'driver': 'n5',\n    'kvstore': {\n        'driver': 'file',\n        'path': output_dir,\n    },\n    'metadata': {\n        'compression': {'type': 'gzip'},\n        'dataType': 'uint32',\n        'dimensions': [1000, 20000],\n        'blockSize': [100, 100],\n    },\n    'create': True,\n    'delete_existing': True,\n}).result()\n\n# Asynchronously write to a sub-region\nwrite_future = dataset[80:82, 99:102].write(np.array([[1, 2, 3], [4, 5, 6]], dtype=np.uint32))\n\n# Wait for the write to complete\nwrite_future.result()\n\n# Read back a larger region\nread_data = dataset[80:83, 99:102].read().result()\n\nprint(f\"Data written to {output_dir}\")\nprint(f\"Read data:\\n{read_data}\")\n\n# Clean up the temporary directory\nimport shutil\nshutil.rmtree(output_dir)\nprint(f\"Cleaned up {output_dir}\")","lang":"python","description":"This example demonstrates creating a new N5 dataset on the local filesystem, writing a small NumPy array to a sub-region, and then reading back a larger region. It highlights the asynchronous nature of TensorStore operations, requiring `.result()` or `await` to wait for completion."},"warnings":[{"fix":"Upgrade Python to version 3.11 or newer.","message":"TensorStore requires Python 3.11 or later. Users attempting to install or run it with older Python versions (e.g., 3.10 or below) will encounter installation failures or runtime errors.","severity":"breaking","affected_versions":"<0.1.82 (Python < 3.11)"},{"fix":"Always append `.result()` to asynchronous calls (e.g., `ts.open(...).result()`, `dataset.write(...).result()`) or use `await` within `async` functions.","message":"TensorStore's Python API is heavily asynchronous. Most I/O operations return `tensorstore.Future` objects. Failing to explicitly call `.result()` on these futures (or `await` them in an `async` context) will result in operations not completing or data not being available as expected.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Ensure your environment is authenticated with the respective cloud provider (e.g., `gcloud auth application-default login` for GCS, AWS credentials for S3).","message":"When using cloud storage drivers (e.g., Google Cloud Storage, Amazon S3), proper authentication credentials must be configured in your environment. Operations will fail with permission errors if credentials are missing or incorrect.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Prefer pre-built wheels (`pip install tensorstore`). If building from source on Windows, configure Bazel to use a shorter output base path (e.g., `TENSORSTORE_BAZEL_STARTUP_OPTIONS=\"--output_base=C:\\\\Out\"`).","message":"Building TensorStore from source (instead of using pre-built PyPI wheels) requires specific C++ compilers (e.g., GCC 10+, Clang 8+, MSVC 2022+) and the Bazel build system. On Windows, long path names can lead to compilation errors like `fatal error C1083: Cannot open include file`.","severity":"gotcha","affected_versions":"All versions (when building from source)"}],"env_vars":null,"last_verified":"2026-04-09T00:00:00.000Z","next_check":"2026-07-08T00:00:00.000Z"}