fsspec — Filesystem Interfaces for Python

Common errors

Warnings

Install

Install compatibility verified last tested: 2026-05-12

Imports

• fsspec.open

  import fsspec
  with fsspec.open('s3://bucket/file.txt', 'rt') as f: ...

  Top-level convenience function; returns an OpenFile that only opens the resource when entering a `with` context.

• fsspec.filesystem
  wrong

  from s3fs import S3FileSystem
  fs = S3FileSystem()

  correct

  import fsspec
  fs = fsspec.filesystem('s3', key='...', secret='...')

  Prefer fsspec.filesystem('protocol') for protocol-agnostic code; direct class imports tie code to a specific backend.

• AbstractFileSystem

  from fsspec.spec import AbstractFileSystem

  Base class for implementing a custom filesystem backend.

• AsyncFileSystem

  from fsspec.asyn import AsyncFileSystem

  Base class for async-capable filesystem backends. Sync wrappers are auto-generated.

• LocalFileSystem

  from fsspec.implementations.local import LocalFileSystem

  Or use fsspec.filesystem('file'). Direct instantiation shown in official docs.

• known_implementations

  from fsspec.registry import known_implementations

  Inspect or extend the registry of protocol → class mappings.

• register_implementation

  from fsspec.registry import register_implementation
  register_implementation('myproto', MyFS)

  Register a custom backend at runtime. Preferred over patching known_implementations directly.

• get_mapper

  import fsspec
  mapper = fsspec.get_mapper('s3://bucket/path/')

  Returns a MutableMapping view of a filesystem path, used by Zarr and other key-value consumers.

• OpenFile

  from fsspec.core import OpenFile

  Serialisable file descriptor; resources are only engaged when used in a `with` context.

Quickstart stale last tested: 2026-04-23

Open a remote HTTP file, read/list a local filesystem, and demonstrate the memory filesystem — all with the same API.

import os
import fsspec

# 1. Open any URL transparently (protocol auto-detected from URL)
with fsspec.open(
    "https://raw.githubusercontent.com/fsspec/filesystem_spec/master/README.md",
    "rt",
) as f:
    first_line = f.readline()
    print("README first line:", first_line.strip())

# 2. Use fsspec.filesystem() for repeated operations on one backend
fs = fsspec.filesystem("memory")  # in-process, no I/O
fs.mkdir("/mydir")
with fs.open("/mydir/hello.txt", "wt") as f:
    f.write("Hello from fsspec!")
with fs.open("/mydir/hello.txt", "rt") as f:
    print(f.read())
print("Files:", fs.ls("/mydir"))

# 3. S3 example (needs pip install fsspec[s3])
# Uses env-var credentials — safe pattern for agents
# aws_key = os.environ.get('AWS_ACCESS_KEY_ID', '')
# aws_secret = os.environ.get('AWS_SECRET_ACCESS_KEY', '')
# fs_s3 = fsspec.filesystem('s3', key=aws_key, secret=aws_secret)
# print(fs_s3.ls('my-bucket/'))

# 4. Zarr-style key-value mapping over any backend
mapper = fsspec.get_mapper("memory://zarr-root/")
mapper["chunk-0"] = b"\x00" * 128
print("Mapper keys:", list(mapper))