Universal Pathlib

raw JSON →
0.3.10 verified Tue May 12 auth: no python install: verified quickstart: stale

Universal Pathlib is a Python library that extends the `pathlib.Path` interface to provide a unified, Pythonic way of interacting with a wide variety of backend filesystems, including local, cloud storage (S3, GCS, Azure), HTTP(S), and more, by leveraging `fsspec` backends. It is currently at version 0.3.10 and maintains a regular release cadence with frequent updates and bug fixes.

pip install universal-pathlib
error ModuleNotFoundError: No module named 'upath'
cause The 'universal-pathlib' library has not been installed, or there's a typo in the import statement.
fix
pip install universal-pathlib
error ModuleNotFoundError: No module named 'universal_pathlib'
cause The user is attempting to import using the package installation name 'universal_pathlib' instead of the aliased module name 'upath'.
fix
from upath import UPath
error ImportError: cannot import name 'Path' from 'upath'
cause The user is trying to import a class named 'Path' from the 'upath' module, but the correct class name is 'UPath'.
fix
from upath import UPath
error fsspec.exceptions.NoBackendError: No backend found for 's3'
cause The required fsspec backend package for the specified protocol (e.g., 's3fs' for S3, 'gcsfs' for GCS) is not installed.
fix
pip install s3fs
breaking The `_protocol_dispatch=False` argument for extending the UPath API has been deprecated. Using it may lead to incorrect behavior or be removed in future versions.
fix Migrate to subclassing `upath.extensions.ProxyUPath` for extending the API. Consult the official documentation for `ProxyUPath` usage examples.
gotcha Attempting to instantiate a specific `UPath` subclass (e.g., `S3Path`) with a path string for a different protocol (e.g., `http://` or `file://`) will now raise a `TypeError` due to protocol incompatibility, rather than silently failing or using an incorrect backend.
fix Ensure the protocol in the path string matches the `UPath` subclass being instantiated, or use the base `UPath` class which dispatches to the correct subclass based on the protocol.
gotcha While `UPath` can attempt to handle any `fsspec`-compatible protocol, explicit implementations are tested and guaranteed. If no specialized `UPath` implementation exists for a given protocol, `UPath` will return a generic instance, but its behavior might not be fully reliable or tested.
fix Prioritize using protocols with explicitly supported `UPath` implementations listed in the documentation. If using an untested protocol, thoroughly test its behavior and consider contributing an implementation if issues arise.
breaking Operations on S3Path (and potentially other cloud storage paths) require proper AWS credentials to be configured in the environment (e.g., via environment variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN, or AWS config files).
fix Ensure AWS credentials are correctly configured for the environment where the Python script is being run. Refer to the AWS documentation on configuring credentials for details (e.g., using environment variables, IAM roles, or the AWS CLI config file).
breaking When interacting with cloud storage services like S3, proper authentication credentials must be configured in the environment (e.g., AWS environment variables, credential files, or IAM roles). Without credentials, operations will fail with a `NoCredentialsError`.
fix Ensure that the execution environment has the necessary AWS credentials configured according to AWS best practices (e.g., AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY environment variables, or an ~/.aws/credentials file).
pip install universal-pathlib fsspec[s3,gcs]
python os / libc status wheel install import disk
3.10 alpine (musl) wheel - 0.23s 20.2M
3.10 alpine (musl) sdist - 0.20s 118.1M
3.10 alpine (musl) - - 0.24s 20.2M
3.10 alpine (musl) - - 0.22s 117.0M
3.10 slim (glibc) wheel 1.8s 0.17s 21M
3.10 slim (glibc) wheel 13.2s 0.18s 118M
3.10 slim (glibc) - - 0.16s 21M
3.10 slim (glibc) - - 0.17s 117M
3.11 alpine (musl) wheel - 0.30s 22.6M
3.11 alpine (musl) sdist - 0.36s 125.8M
3.11 alpine (musl) - - 0.34s 22.6M
3.11 alpine (musl) - - 0.34s 124.8M
3.11 slim (glibc) wheel 1.9s 0.27s 23M
3.11 slim (glibc) wheel 11.8s 0.29s 126M
3.11 slim (glibc) - - 0.25s 23M
3.11 slim (glibc) - - 0.26s 125M
3.12 alpine (musl) wheel - 0.49s 14.3M
3.12 alpine (musl) sdist - 0.57s 116.8M
3.12 alpine (musl) - - 0.80s 14.3M
3.12 alpine (musl) - - 0.54s 115.7M
3.12 slim (glibc) wheel 1.7s 0.46s 15M
3.12 slim (glibc) wheel 9.0s 0.53s 116M
3.12 slim (glibc) - - 0.49s 15M
3.12 slim (glibc) - - 0.50s 116M
3.13 alpine (musl) wheel - 0.50s 14.1M
3.13 alpine (musl) sdist - 0.52s 116.4M
3.13 alpine (musl) - - 0.51s 14.0M
3.13 alpine (musl) - - 0.55s 115.1M
3.13 slim (glibc) wheel 1.8s 0.45s 15M
3.13 slim (glibc) wheel 8.9s 0.45s 116M
3.13 slim (glibc) - - 0.47s 14M
3.13 slim (glibc) - - 0.49s 115M
3.9 alpine (musl) wheel - 0.17s 19.7M
3.9 alpine (musl) sdist - 0.19s 95.7M
3.9 alpine (musl) - - 0.19s 19.7M
3.9 alpine (musl) - - 0.21s 94.6M
3.9 slim (glibc) wheel 2.2s 0.15s 20M
3.9 slim (glibc) wheel 13.7s 0.18s 99M
3.9 slim (glibc) - - 0.18s 20M
3.9 slim (glibc) - - 0.18s 98M

This quickstart demonstrates how to use `UPath` for both a remote S3 bucket and a local file. It shows common operations like writing text, reading text, checking existence, and iterating directory contents. For S3, ensure `s3fs` is installed and AWS credentials are configured (e.g., via environment variables or `~/.aws/credentials`). For actual usage, replace `your-test-s3-bucket` with an accessible S3 bucket.

import os
from upath import UPath

# Example for S3 (requires 's3fs' to be installed and AWS credentials configured)
# You can use environment variables for credentials or pass storage_options
S3_BUCKET = os.environ.get('S3_TEST_BUCKET', 'your-test-s3-bucket')
S3_KEY = 'example.txt'

# Create a UPath object for an S3 location
s3path = UPath(f"s3://{S3_BUCKET}") / S3_KEY

# Write content to the S3 file
try:
    s3path.write_text('Hello from Universal Pathlib!')
    print(f"Successfully wrote to {s3path}.")

    # Read content from the S3 file
    content = s3path.read_text()
    print(f"Content of {s3path}: '{content}'.")

    # Check if the path exists
    print(f"Does {s3path} exist? {s3path.exists()}.")

    # List contents of the parent directory (if bucket exists and is not empty)
    print(f"Listing contents of {s3path.parent}:")
    for p in s3path.parent.iterdir():
        print(f"  - {p}")

finally:
    # Clean up: delete the created file
    if s3path.exists():
        s3path.unlink()
        print(f"Cleaned up: deleted {s3path}.")

# Example for a local file (no additional fsspec backend needed)
local_path = UPath("local_example.txt")
local_path.write_text("This is a local file.")
print(f"Local file content: {local_path.read_text()}")
local_path.unlink()