obstore: Python Object Storage Interface

Warnings

• breaking Support for Python 3.9 was deprecated in `obstore` version 0.9.0. Users on Python 3.9 should upgrade their Python environment.
  Fix: Upgrade Python to 3.10 or newer.
• breaking In older versions (prior to 0.7.0), `S3Store.from_session()` and `S3Store._from_native()` were removed. Users should transition to using credential providers for S3 authentication.
  Fix: Refactor S3 store construction to use explicit credential providers or environment variable configuration instead of `from_session`.
• breaking Path encoding behavior changed in version 0.8.2 to prevent unintentional double-encoding. Users must now ensure that paths provided to `obstore` are valid and correctly encoded.
  Fix: Review all path inputs to `obstore` functions and ensure they are properly formed (e.g., not already percent-encoded if they contain special characters).
• breaking In version 0.5.0, the `container` parameter for `AzureStore`'s constructor was renamed to `container_name` and became a keyword-only argument. Using `container` will raise an error.
  Fix: Update `AzureStore` constructor calls to use `container_name='my-container'` instead of `container='my-container'`.
• gotcha Unlike many object storage libraries, `obstore`'s core operations (`put`, `get`, `list`, `delete`, `copy`, etc.) are top-level functions (e.g., `obs.put(store, ...)`) rather than methods on the store object (e.g., `store.put(...)`).
  Fix: Always call object storage operations as `obstore.<function_name>(store_instance, ...)`.
• gotcha When listing objects with `obstore.list(return_arrow=True)`, the optional `arro3-core` dependency is required. Without it, attempting to use this feature will fail.
  Fix: If using `return_arrow=True`, ensure `pip install obstore[arrow]` or `pip install arro3-core` is run.

Install

• pip install obstore Install stable version

Imports

• obstore

  import obstore as obs

• MemoryStore

  from obstore.store import MemoryStore

  Store classes like MemoryStore, S3Store, GCSStore are located in the `obstore.store` submodule, not directly under `obstore`.
• put

  obs.put(store, 'path', b'data')

  All operations (put, get, list, delete, copy, etc.) are top-level functions exported by the `obstore` module, not methods of the store object itself.

Quickstart

This quickstart demonstrates how to initialize an in-memory object store and perform basic synchronous `put` and `get` operations. Note that operations like `put` and `get` are top-level functions within the `obstore` module, taking the store instance as their first argument. Asynchronous counterparts (e.g., `put_async`, `get_async`) are also available.

import obstore as obs
from obstore.store import MemoryStore

# Initialize an in-memory store for demonstration
store = MemoryStore()

# Define a file path and content
file_path = "my_document.txt"
file_content = b"Hello, obstore world!"

# Put the object into the store
obs.put(store, file_path, file_content)
print(f"Object '{file_path}' put into store.")

# Get the object from the store
response = obs.get(store, file_path)
retrieved_content = response.bytes()
print(f"Retrieved content: {retrieved_content.decode()}")

assert retrieved_content == file_content
print("Content matches!")

# Asynchronous operations are also available (requires an async context)
# async def main():
#    await obs.put_async(store, 'async_file.txt', b'async data')
#    res_async = await obs.get_async(store, 'async_file.txt')
#    print(f"Async retrieved: {res_async.bytes().decode()}")
# import asyncio
# asyncio.run(main())