iopath

0.1.10 · active · verified Sat Apr 11

iopath is a Python library from Facebook Research that provides an abstraction layer for I/O operations, allowing a unified API to interact with local filesystems and various remote storage services like S3 and HDFS. It simplifies reading from and writing to different storage backends. The current version is 0.1.10, and releases are infrequent, often tied to major projects from Facebook Research.

Warnings

gotcha When using `pathmgr.get_local_path(uri)`, if the URI points to a remote file, `iopath` will download it to a temporary local path. This operation can be slow for large files and consumes local disk space, potentially leading to unexpected performance bottlenecks or out-of-disk issues if not managed carefully.
Fix: Be mindful of calling `get_local_path` only when necessary. For direct streaming, prefer `pathmgr.open(uri, 'rb')`. Consider using `tempfile.TemporaryDirectory` for managing local copies if explicit caching is desired.
gotcha While `iopath` provides a unified API, support for remote file systems like S3 or HDFS requires installing optional dependencies (e.g., `iopath[s3]`) and ensuring the respective `PathHandler` is registered. Although `PathManager` often auto-registers common handlers, explicit registration (e.g., `pathmgr.register_handler(S3PathHandler())`) can prevent unexpected 'scheme not supported' errors.
Fix: Install `iopath` with the necessary extras (e.g., `pip install iopath[s3]`). If issues persist, explicitly import and register the handler for your specific remote storage scheme: `from iopath.common.s3 import S3PathHandler; pathmgr.register_handler(S3PathHandler())`.
gotcha The core `PathManager` class is not directly exposed at the top-level `iopath` package. A common mistake is attempting `from iopath import PathManager` which will fail. It must be imported from `iopath.common.file_io`.
Fix: Always use `from iopath.common.file_io import PathManager` for importing the main PathManager class.

Install

pip install iopath Install base library
pip install iopath[s3] Install with S3 support

Imports

PathManager
```
from iopath.common.file_io import PathManager
```
PathManager is not directly exposed at the top-level `iopath` package, requiring import from its specific submodule.
S3PathHandler
```
from iopath.common.s3 import S3PathHandler
```
Used for explicitly registering S3 support, though `PathManager` often registers default handlers upon first S3 access.

Quickstart

This quickstart demonstrates how to use `PathManager` for basic file operations (write, read, check existence, remove) on a local filesystem. `PathManager` can transparently handle different schemes (e.g., `s3://`, `hdfs://`) once appropriate handlers are registered or if the optional dependencies are installed for common schemes like S3.

import os
from iopath.common.file_io import PathManager

# Initialize PathManager
pathmgr = PathManager()

# Define a local test file path
test_file_name = "iopath_test_file.txt"

# Ensure clean slate if previous run failed
if pathmgr.exists(test_file_name):
    pathmgr.rm(test_file_name)

print(f"Checking if '{test_file_name}' exists: {pathmgr.exists(test_file_name)}")

# Write content to the file using PathManager
content_to_write = "Hello from iopath! This is an I/O abstraction library."
with pathmgr.open(test_file_name, "w") as f:
    f.write(content_to_write)

print(f"After writing, does '{test_file_name}' exist? {pathmgr.exists(test_file_name)}")

# Read content from the file using PathManager
read_content = ""
with pathmgr.open(test_file_name, "r") as f:
    read_content = f.read()

print(f"Content read from '{test_file_name}':\n'{read_content}'")

# Clean up the test file
pathmgr.rm(test_file_name)
print(f"After removal, does '{test_file_name}' exist? {pathmgr.exists(test_file_name)}")

view raw JSON →