fsspec — Filesystem Interfaces for Python

Warnings

• gotcha Filesystem instances are cached singletons. fsspec.filesystem('s3', key=K1) and fsspec.filesystem('s3', key=K2) with different credentials may return the same cached instance if arguments hash identically. This can cause silent credential cross-contamination.
  Fix: Call fs_class.clear_instance_cache() when you need a fresh instance, or pass skip_instance_cache=True to fsspec.filesystem().
• breaking Calling sync fsspec methods (e.g. fs.ls()) from inside a running asyncio event loop raises NotImplementedError: 'Calling sync() from within a running loop'. This affects Jupyter notebooks, FastAPI handlers, and async frameworks.
  Fix: Inside an async context, instantiate with asynchronous=True and await the private _method coroutines (e.g. await fs._ls()), or use fsspec.asyn.fsspec_loop context manager.
• gotcha fsspec.open() returns an OpenFile placeholder — the remote file is NOT opened until you enter a `with` block. Calling .read() on the raw OpenFile object without a context manager will fail.
  Fix: Always use `with fsspec.open(...) as f: f.read(...)` or call of.open() explicitly and close when done.
• gotcha Cloud backend packages (s3fs, gcsfs, adlfs) are NOT included in the base fsspec install. Attempting to use fsspec.filesystem('s3') without s3fs raises an ImportError with a hint, but the hint may be missed in automated pipelines.
  Fix: Install the required extra: pip install fsspec[s3], fsspec[gcs], or the specific package. Use fsspec.available_protocols() to check what is installed.
• gotcha Async filesystem instances (s3fs, gcsfs) are incompatible with os.fork() (used by PyTorch DataLoader, multiprocessing). After fork, cached instances hold references to dead event loops, causing hangs or cryptic errors.
  Fix: Use multiprocessing with spawn context, or call fs_class.clear_instance_cache() inside the worker initializer after forking.
• deprecated Passing `trim` to fsspec.spec.AbstractBufferedFile is deprecated.
  Fix: Remove the trim argument from any AbstractBufferedFile instantiation.
• breaking fsspec.asyn.maybe_sync was removed. Older pinned versions of s3fs (<=0.5.2) and other ecosystem libraries that imported maybe_sync will break on recent fsspec.
  Fix: Upgrade s3fs, gcsfs, and adlfs to current releases that are compatible with the same fsspec CalVer month.

Install

• pip install fsspec Base install (local, memory, HTTP, ZIP, TAR built-in)
• pip install fsspec[s3] With AWS S3 support (installs s3fs)
• pip install fsspec[gcs] With Google Cloud Storage support (installs gcsfs)
• pip install fsspec[ssh] With SSH/SFTP support
• pip install fsspec[full] All known optional backends

Imports

• fsspec.open

  import fsspec
  with fsspec.open('s3://bucket/file.txt', 'rt') as f: ...

  Top-level convenience function; returns an OpenFile that only opens the resource when entering a `with` context.
• fsspec.filesystem

  import fsspec
  fs = fsspec.filesystem('s3', key='...', secret='...')

  Prefer fsspec.filesystem('protocol') for protocol-agnostic code; direct class imports tie code to a specific backend.
• AbstractFileSystem

  from fsspec.spec import AbstractFileSystem

  Base class for implementing a custom filesystem backend.
• AsyncFileSystem

  from fsspec.asyn import AsyncFileSystem

  Base class for async-capable filesystem backends. Sync wrappers are auto-generated.
• LocalFileSystem

  from fsspec.implementations.local import LocalFileSystem

  Or use fsspec.filesystem('file'). Direct instantiation shown in official docs.
• known_implementations

  from fsspec.registry import known_implementations

  Inspect or extend the registry of protocol → class mappings.
• register_implementation

  from fsspec.registry import register_implementation
  register_implementation('myproto', MyFS)

  Register a custom backend at runtime. Preferred over patching known_implementations directly.
• get_mapper

  import fsspec
  mapper = fsspec.get_mapper('s3://bucket/path/')

  Returns a MutableMapping view of a filesystem path, used by Zarr and other key-value consumers.
• OpenFile

  from fsspec.core import OpenFile

  Serialisable file descriptor; resources are only engaged when used in a `with` context.

Quickstart

Open a remote HTTP file, read/list a local filesystem, and demonstrate the memory filesystem — all with the same API.

import os
import fsspec

# 1. Open any URL transparently (protocol auto-detected from URL)
with fsspec.open(
    "https://raw.githubusercontent.com/fsspec/filesystem_spec/master/README.md",
    "rt",
) as f:
    first_line = f.readline()
    print("README first line:", first_line.strip())

# 2. Use fsspec.filesystem() for repeated operations on one backend
fs = fsspec.filesystem("memory")  # in-process, no I/O
fs.mkdir("/mydir")
with fs.open("/mydir/hello.txt", "wt") as f:
    f.write("Hello from fsspec!")
with fs.open("/mydir/hello.txt", "rt") as f:
    print(f.read())
print("Files:", fs.ls("/mydir"))

# 3. S3 example (needs pip install fsspec[s3])
# Uses env-var credentials — safe pattern for agents
# aws_key = os.environ.get('AWS_ACCESS_KEY_ID', '')
# aws_secret = os.environ.get('AWS_SECRET_ACCESS_KEY', '')
# fs_s3 = fsspec.filesystem('s3', key=aws_key, secret=aws_secret)
# print(fs_s3.ls('my-bucket/'))

# 4. Zarr-style key-value mapping over any backend
mapper = fsspec.get_mapper("memory://zarr-root/")
mapper["chunk-0"] = b"\x00" * 128
print("Mapper keys:", list(mapper))