morefs: A Collection of fsspec-based Filesystems

0.2.2 · active · verified Sun Apr 12

morefs is a Python library offering a collection of self-contained filesystems built on top of `fsspec`. It currently provides `AsyncLocalFileSystem`, `DictFileSystem`, and `OverlayFileSystem` to extend `fsspec`'s capabilities for asynchronous local operations, in-memory dictionary-backed storage, and layered filesystem views, respectively. The library is actively maintained, with the current version being 0.2.2, and releases occur as needed to address compatibility or introduce new features.

Warnings

breaking Versions of morefs 0.2.1 and later require `fsspec>=2024.5.0` due to API changes in `fsspec`. Using an older version of `fsspec` may lead to runtime errors or unexpected behavior.
Fix: Ensure `fsspec` is updated to version `2024.5.0` or newer: `pip install --upgrade fsspec`.
gotcha The `AsyncLocalFileSystem` underwent a significant internal refactor in version 0.1.0 to wrap `fsspec.LocalFileSystem`. While the public API aimed for compatibility, users relying on specific prior internal behaviors or direct imports from the pre-0.1.0 `asyncfs` module (now `asyn_local`) might experience subtle changes or broken imports.
Fix: Verify that existing code using `AsyncLocalFileSystem` works as expected after upgrading to 0.1.0 or later. Update any direct `from morefs.asyncfs import ...` imports to `from morefs.asyn_local import ...`.
gotcha As `morefs` is currently in its 0.x.x version series, its API is still considered somewhat experimental and may undergo non-backward-compatible changes in minor releases. Users should pin exact versions in production environments.
Fix: Pin `morefs` to an exact version in your `requirements.txt` or `pyproject.toml` (e.g., `morefs==0.2.2`) and thoroughly review changelogs before upgrading.

Install

pip install morefs Install stable version

Imports

AsyncLocalFileSystem

from morefs.asyn_local import AsyncLocalFileSystem

DictFileSystem
```
from morefs.dict import DictFileSystem
```

OverlayFileSystem

from morefs.overlay import OverlayFileSystem

filesystem
```
from fsspec import filesystem
```
Used to instantiate morefs filesystems via their protocol string (e.g., 'asynclocal').

Quickstart

Demonstrates the initialization and basic usage of `AsyncLocalFileSystem`, `DictFileSystem`, and `OverlayFileSystem`, including listing directories and reading file content.

from fsspec import filesystem
from morefs.asyn_local import AsyncLocalFileSystem
from morefs.dict import DictFileSystem
from morefs.overlay import OverlayFileSystem
import asyncio

# AsyncLocalFileSystem
async def run_asynclocal_example():
    fs = filesystem("asynclocal") # or AsyncLocalFileSystem()
    # Using AsyncLocalFileSystem requires an async context
    print(f"AsyncLocalFileSystem ls '/': {await fs.ls('/')}")

# DictFileSystem
dict_fs = DictFileSystem({"foo": b"bar", "baz/qux": b"quux"})
print(f"DictFileSystem ls '/': {dict_fs.ls('/')}")
print(f"DictFileSystem read 'foo': {dict_fs.read_bytes('foo')}")

# OverlayFileSystem
lower_fs = DictFileSystem({"foo": b"bar_lower", "common": b"lower_val"})
upper_fs = DictFileSystem({"baz": b"qux_upper", "common": b"upper_val"})
overlay_fs = OverlayFileSystem(lower_fs=lower_fs, upper_fs=upper_fs)
print(f"OverlayFileSystem ls '/': {overlay_fs.ls('/')}")
print(f"OverlayFileSystem read 'foo': {overlay_fs.read_bytes('foo')}")
print(f"OverlayFileSystem read 'baz': {overlay_fs.read_bytes('baz')}")
print(f"OverlayFileSystem read 'common' (upper overrides): {overlay_fs.read_bytes('common')}")

# Run the async example
asyncio.run(run_asynclocal_example())

view raw JSON →