S3Fs

2026.2.0 verified Tue May 12 auth: no python install: verified quickstart: stale

S3Fs is a Pythonic filesystem interface to Amazon S3, built on top of aiobotocore and fsspec. The top-level class S3FileSystem exposes familiar file-system operations (ls, cp, mv, du, glob, put, get) and a file open() API that emulates Python's standard file protocol, making it a drop-in for libraries like pandas, dask, and gzip that accept file-like objects. It also supports S3-compatible stores (MinIO, Ceph, R2) via the endpoint_url parameter. Versions follow calendar versioning (YYYY.MM.PATCH); the current release is 2026.2.0, released February 2026, with roughly monthly cadence.

pip install s3fs

Common errors

error ModuleNotFoundError: No module named 's3fs' ↓

cause The 's3fs' library has not been installed in the current Python environment.

fix

pip install s3fs

error fsspec.exceptions.NoS3FSImplementation: No s3fs implementation found, install s3fs to use s3:// ↓

cause When using `fsspec`-dependent libraries like pandas or dask to access S3 paths, `s3fs` must be installed to provide the S3 filesystem implementation.

fix

pip install s3fs

error botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the GetObject operation: Access Denied. ↓

cause The AWS credentials configured for `s3fs` lack the necessary IAM permissions to perform the requested S3 operation on the specified bucket or object.

fix

Ensure your AWS user or role has the required IAM policies (e.g., s3:GetObject, s3:ListBucket) for the target S3 resources, and that your credentials are correctly configured (environment variables, ~/.aws/credentials, or IAM role).

error TypeError: S3FileSystem.__init__() got an unexpected keyword argument 'endpoint_url' ↓

cause The `endpoint_url` parameter for S3-compatible storage should be passed within the `client_kwargs` dictionary, not as a direct argument to `S3FileSystem`.

fix

s3 = s3fs.S3FileSystem(client_kwargs={'endpoint_url': 'http://localhost:9000'})

error FileNotFoundError: No such file or directory: 's3://my-bucket/non-existent-path' ↓

cause The specified S3 path (bucket or object) does not exist or is inaccessible with the provided credentials.

fix

Verify the S3 path for correctness, ensure the bucket exists, and confirm that your AWS credentials have appropriate list and read permissions for the specified location.

Warnings

breaking aiobotocore pins an extremely narrow botocore version range (often a single patch). Installing s3fs alongside boto3 or awscli frequently produces irresolvable dependency conflicts because boto3 requires a different botocore range. ↓

fix Use 'pip install s3fs[boto3]' to get a pre-validated boto3+botocore combination, or prefer conda-forge which pre-solves the triangle. Never pin boto3 and s3fs independently with ^ in Poetry without checking aiobotocore's exact botocore requirement first.

breaking Using multiprocessing with the default 'fork' start method causes deadlocks and hard-to-reproduce bugs because s3fs keeps open async sockets and a background thread. ↓

fix Set multiprocessing.set_start_method('spawn') or 'forkserver' before creating S3FileSystem instances, or avoid sharing S3FileSystem objects across fork boundaries.

breaking s3fs version 2023.12.0 was yanked from PyPI due to an authentication regression. pip may still resolve to it on some platforms if not using --pre filtering. ↓

fix Pin to >=2023.12.1. Run 'pip install s3fs>=2023.12.1' to avoid the yanked release.

gotcha The directory listing cache (dircache) is not invalidated automatically. If an object is written or resized externally (e.g. by boto3 or another process) after fs.ls() or fs.info() has cached its metadata, subsequent reads via the same S3FileSystem instance will use stale size information and may return corrupted or truncated data. ↓

fix Call fs.invalidate_cache() after external writes, or pass refresh=True to fs.info(). For high-churn workloads consider listings_expiry_time= when constructing S3FileSystem.

gotcha File access is always binary. Text mode ('r', 'w') is technically accepted but returns bytes or requires an explicit encoding wrapper. readline() and line iteration work but the underlying stream is always bytes. ↓

fix Use 'rb'/'wb' modes explicitly. For text, wrap with io.TextIOWrapper: io.TextIOWrapper(fs.open('bucket/file', 'rb'), encoding='utf-8').

gotcha S3FileSystem instances are cached as singletons by default (skip_instance_cache=False). Two calls with the same credentials return the same object, which can cause credential or config bleed between parts of an application that expect independent connections. ↓

fix Pass skip_instance_cache=True when constructing S3FileSystem if you need isolated instances, e.g. with different endpoint_urls or IAM roles.

gotcha Writes to S3 are not flushed until the file is closed (or the multipart threshold of ~150 MiB is hit). Calling f.write() without closing inside a context manager means data is buffered locally and nothing is committed to S3 on partial writes. ↓

fix Always use 'with fs.open(..., "wb") as f:' context managers for writes. Do not rely on explicit flush() calls for durability; only close() / __exit__ commits the upload.

breaking S3FileSystem requires valid AWS credentials to be configured (e.g., via environment variables, IAM roles, or ~/.aws/credentials). Failure to provide these results in an 'AuthorizationHeaderMalformed' error. ↓

fix Ensure AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables are set, or that an appropriate IAM role is available to the execution environment. Alternatively, pass explicit credentials to the S3FileSystem constructor.

breaking An `AuthorizationHeaderMalformed` error, often stating 'a non-empty Access Key (AKID) must be provided in the credential', means that s3fs could not find or use valid AWS credentials. This typically happens when environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) are unset or empty, or explicit `key` and `secret` parameters are missing/incorrect from the `S3FileSystem` constructor. ↓

fix Ensure AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables are correctly set, or pass `key` and `secret` parameters directly to the S3FileSystem constructor. If using IAM roles, verify the instance profile or role assumption is properly configured and providing valid temporary credentials.

Install

pip install 's3fs[boto3]'

conda install -c conda-forge s3fs

Install compatibility verified last tested: 2026-05-12

python os / libc variant status wheel install import disk

3.10 alpine (musl) boto3 - - 0.93s 61.0M

3.10 alpine (musl) s3fs - - 0.97s 61.0M

3.10 slim (glibc) boto3 - - 0.73s 63M

3.10 slim (glibc) s3fs - - 0.72s 63M

3.11 alpine (musl) boto3 - - 1.24s 65.3M

3.11 alpine (musl) s3fs - - 1.24s 65.3M

3.11 slim (glibc) boto3 - - 1.13s 68M

3.11 slim (glibc) s3fs - - 1.02s 68M

3.12 alpine (musl) boto3 - - 1.31s 56.7M

3.12 alpine (musl) s3fs - - 1.31s 56.7M

3.12 slim (glibc) boto3 - - 1.26s 59M

3.12 slim (glibc) s3fs - - 1.39s 59M

3.13 alpine (musl) boto3 - - 1.25s 56.1M

3.13 alpine (musl) s3fs - - 1.27s 56.1M

3.13 slim (glibc) boto3 - - 1.24s 58M

3.13 slim (glibc) s3fs - - 1.31s 58M

3.9 alpine (musl) boto3 - - 0.92s 62.1M

3.9 alpine (musl) s3fs - - 0.93s 60.0M

3.9 slim (glibc) boto3 - - 0.81s 65M

3.9 slim (glibc) s3fs - - 0.80s 63M

Imports

S3FileSystem
wrong
```
from s3fs.core import S3FileSystem
```
correct
```
import s3fs
s3 = s3fs.S3FileSystem()
```
S3FileSystem is the only public entrypoint; importing from s3fs.core is internal and may break across releases. Always use import s3fs then s3fs.S3FileSystem().
S3File
```
with s3fs.S3FileSystem().open('bucket/key', 'rb') as f: ...
```
S3File is returned by open(); never instantiate it directly. Only binary modes are supported: r, w, a, rb, wb, ab.
open via fsspec URL
```
import fsspec
with fsspec.open('s3://bucket/key', 'rb') as f: ...
```
fsspec.open with an s3:// URL will automatically dispatch to S3FileSystem; pass storage_options dict for credentials.

Quickstart stale last tested: 2026-04-23

Connect with explicit credentials from environment variables, list a bucket, read a file, and write a file.

import os
import s3fs

# Credentials via env vars (boto chain also checks ~/.aws/credentials, IAM roles, etc.)
fs = s3fs.S3FileSystem(
    key=os.environ.get('AWS_ACCESS_KEY_ID', ''),
    secret=os.environ.get('AWS_SECRET_ACCESS_KEY', ''),
    # token=os.environ.get('AWS_SESSION_TOKEN', ''),  # uncomment for STS/assumed-role
    # endpoint_url='https://s3.example.com',          # uncomment for MinIO / S3-compatible
)

# List bucket contents
bucket = os.environ.get('S3_BUCKET', 'my-bucket')
print(fs.ls(bucket))

# Read a file
with fs.open(f'{bucket}/hello.txt', 'rb') as f:
    print(f.read())

# Write a file (must flush >5 MiB for multipart; context manager handles this)
with fs.open(f'{bucket}/output.txt', 'wb') as f:
    f.write(b'hello s3fs')

# Works transparently with pandas via storage_options
import pandas as pd
df = pd.read_csv(
    f's3://{bucket}/data.csv',
    storage_options={
        'key': os.environ.get('AWS_ACCESS_KEY_ID', ''),
        'secret': os.environ.get('AWS_SECRET_ACCESS_KEY', ''),
    },
)
print(df.head())