{"id":10008,"library":"opendal","title":"OpenDAL Python Binding","description":"OpenDAL provides a unified data access layer, allowing Python applications to interact with various storage services (e.g., S3, Azure Blob, GCS, local filesystem) through a single API. It's a binding to the Apache OpenDAL Rust core, currently at version 0.46.0 of the Python package. The Python package typically follows the Rust core, though with some release lag, and provides both asynchronous and synchronous interfaces.","status":"active","version":"0.46.0","language":"en","source_language":"en","source_url":"https://github.com/apache/opendal","tags":["storage","cloud","data access","object storage","filesystem","binding","asyncio"],"install":[{"cmd":"pip install opendal","lang":"bash","label":"Install OpenDAL"}],"dependencies":[],"imports":[{"symbol":"Operator","correct":"from opendal import Operator"},{"symbol":"BlockingOperator","correct":"from opendal import BlockingOperator"},{"symbol":"Error","correct":"from opendal import Error"}],"quickstart":{"code":"import opendal\nimport asyncio\nimport os\n\nasync def main():\n    # Configure OpenDAL for S3. Replace with your actual credentials or use a different scheme.\n    # Using os.environ.get for security and flexibility.\n    config = {\n        \"scheme\": \"s3\",\n        \"bucket\": os.environ.get(\"OPENDAL_S3_BUCKET\", \"your-s3-bucket\"),\n        \"region\": os.environ.get(\"OPENDAL_S3_REGION\", \"us-east-1\"),\n        \"access_key_id\": os.environ.get(\"OPENDAL_S3_ACCESS_KEY_ID\", \"\"),\n        \"secret_access_key\": os.environ.get(\"OPENDAL_S3_SECRET_ACCESS_KEY\", \"\"),\n    }\n    \n    try:\n        # Initialize the asynchronous Operator\n        op = opendal.Operator(config)\n        key = \"hello_opendal.txt\"\n        content_to_write = b\"Hello from OpenDAL Python!\"\n        \n        # Write data asynchronously\n        await op.write(key, content_to_write)\n        print(f\"Successfully wrote '{content_to_write.decode()}' to {key}\")\n        \n        # Read data asynchronously\n        read_content = await op.read(key)\n        print(f\"Successfully read '{read_content.decode()}' from {key}\")\n        \n        # Get metadata asynchronously\n        metadata = await op.stat(key)\n        print(f\"Metadata for {key}: size={metadata.content_length}, last_modified={metadata.last_modified}\")\n        \n        # Clean up\n        await op.delete(key)\n        print(f\"Successfully deleted {key}\")\n\n    except opendal.Error as e:\n        print(f\"OpenDAL Error: {e}\")\n    except Exception as e:\n        print(f\"An unexpected error occurred: {e}\")\n\nif __name__ == \"__main__\":\n    # Ensure the environment variables are set for S3 or use a simpler scheme like 'memory'.\n    # Example for 'memory' scheme (no credentials needed, replace config):\n    # op = opendal.Operator(\"memory\")\n    asyncio.run(main())","lang":"python","description":"This quickstart demonstrates how to initialize an OpenDAL S3 operator and perform basic asynchronous operations: write, read, get metadata, and delete. Ensure your S3 bucket, region, and credentials are set as environment variables or provided directly in the configuration. For a simpler start, replace the config with `op = opendal.Operator(\"memory\")`."},"warnings":[{"fix":"Always `await` methods of `opendal.Operator` within an `asyncio` event loop. For synchronous code, use `opendal.BlockingOperator` and its methods directly without `await`.","message":"OpenDAL Python provides two distinct entry points: `opendal.Operator` for asynchronous (async/await) operations and `opendal.BlockingOperator` for synchronous use. Mixing these can lead to `RuntimeError` or `TypeError` if an async method is called synchronously, or vice-versa.","severity":"gotcha","affected_versions":">=0.1.0"},{"fix":"Refer to the official OpenDAL documentation for the exact configuration parameters required for your specific storage service. Ensure all mandatory parameters are provided with correct values.","message":"Incorrect or incomplete configuration for a chosen storage scheme (e.g., S3, Azure Blob, GCS) will result in `opendal.Error: ServiceError { kind: InvalidConfig, ... }`. Each service requires specific parameters (e.g., bucket, region, credentials).","severity":"gotcha","affected_versions":">=0.1.0"},{"fix":"If upgrading to a future OpenDAL Python binding, review code that calls `op.write()`. It will receive a `Metadata` object, allowing access to properties like `content_length` or `etag` immediately after writing.","message":"In Rust core v0.52.0 and later (which will affect future Python binding versions greater than 0.46.0), `write` operations will return `Metadata` instead of `None`. Code currently assuming `op.write()` returns `None` will need to be updated.","severity":"breaking","affected_versions":"Future Python versions >0.46.0 (corresponding to Rust core >=0.52.0)"},{"fix":"When upgrading to a future OpenDAL Python binding, ensure your code handles `jiff.Timestamp` objects for metadata timestamp fields. You may need to `pip install jiff` and adapt your datetime parsing/formatting logic.","message":"In Rust core v0.55.0 and later (which will affect future Python binding versions greater than 0.46.0), timestamp fields in `Metadata` (e.g., `last_modified`) will use `jiff.Timestamp` objects. Current versions might return a different type (e.g., `datetime.datetime`).","severity":"breaking","affected_versions":"Future Python versions >0.46.0 (corresponding to Rust core >=0.55.0)"}],"env_vars":null,"last_verified":"2026-04-17T00:00:00.000Z","next_check":"2026-07-16T00:00:00.000Z","problems":[{"fix":"Ensure all calls to `opendal.Operator` methods are preceded by `await` and executed within an `asyncio.run()` block or an already running event loop.","cause":"Attempting to call an asynchronous method (e.g., `op.read()`, `op.write()`) of `opendal.Operator` directly without `await` keyword or outside an `asyncio` event loop.","error":"RuntimeError: await wasn't called"},{"fix":"Double-check the OpenDAL documentation for the specific storage service you are trying to connect to. Verify all required configuration keys (e.g., `bucket`, `region`, `access_key_id`) are present and their values are correct.","cause":"The dictionary passed to `opendal.Operator()` for configuration is missing required parameters or contains incorrect values for the specified `scheme`.","error":"opendal.Error: ServiceError { kind: InvalidConfig, ... }"},{"fix":"Use `opendal.Operator` with `async with` if an asynchronous context manager is desired. For `opendal.BlockingOperator`, use it directly without `async with`.","cause":"Attempting to use `async with` context manager with `opendal.BlockingOperator`. The `BlockingOperator` is designed for synchronous use and does not implement the asynchronous context manager protocol.","error":"AttributeError: 'BlockingOperator' object has no attribute '__aenter__'"},{"fix":"Recognize that `op.write()` returns `None` in `opendal` v0.46.0. If you need to verify content or access metadata, perform a subsequent `op.read()` or `op.stat()` operation. Be aware that future versions will return `Metadata`.","cause":"In `opendal` Python binding version 0.46.0, `op.write()` returns `None`. Your code is attempting to access attributes or methods (like `decode()`) on the `None` return value, assuming it received data or metadata.","error":"AttributeError: 'NoneType' object has no attribute 'decode' (after a write operation)"}]}