S3Fs

Warnings

• breaking aiobotocore pins an extremely narrow botocore version range (often a single patch). Installing s3fs alongside boto3 or awscli frequently produces irresolvable dependency conflicts because boto3 requires a different botocore range.
  Fix: Use 'pip install s3fs[boto3]' to get a pre-validated boto3+botocore combination, or prefer conda-forge which pre-solves the triangle. Never pin boto3 and s3fs independently with ^ in Poetry without checking aiobotocore's exact botocore requirement first.
• breaking Using multiprocessing with the default 'fork' start method causes deadlocks and hard-to-reproduce bugs because s3fs keeps open async sockets and a background thread.
  Fix: Set multiprocessing.set_start_method('spawn') or 'forkserver' before creating S3FileSystem instances, or avoid sharing S3FileSystem objects across fork boundaries.
• breaking s3fs version 2023.12.0 was yanked from PyPI due to an authentication regression. pip may still resolve to it on some platforms if not using --pre filtering.
  Fix: Pin to >=2023.12.1. Run 'pip install s3fs>=2023.12.1' to avoid the yanked release.
• gotcha The directory listing cache (dircache) is not invalidated automatically. If an object is written or resized externally (e.g. by boto3 or another process) after fs.ls() or fs.info() has cached its metadata, subsequent reads via the same S3FileSystem instance will use stale size information and may return corrupted or truncated data.
  Fix: Call fs.invalidate_cache() after external writes, or pass refresh=True to fs.info(). For high-churn workloads consider listings_expiry_time= when constructing S3FileSystem.
• gotcha File access is always binary. Text mode ('r', 'w') is technically accepted but returns bytes or requires an explicit encoding wrapper. readline() and line iteration work but the underlying stream is always bytes.
  Fix: Use 'rb'/'wb' modes explicitly. For text, wrap with io.TextIOWrapper: io.TextIOWrapper(fs.open('bucket/file', 'rb'), encoding='utf-8').
• gotcha S3FileSystem instances are cached as singletons by default (skip_instance_cache=False). Two calls with the same credentials return the same object, which can cause credential or config bleed between parts of an application that expect independent connections.
  Fix: Pass skip_instance_cache=True when constructing S3FileSystem if you need isolated instances, e.g. with different endpoint_urls or IAM roles.
• gotcha Writes to S3 are not flushed until the file is closed (or the multipart threshold of ~150 MiB is hit). Calling f.write() without closing inside a context manager means data is buffered locally and nothing is committed to S3 on partial writes.
  Fix: Always use 'with fs.open(..., "wb") as f:' context managers for writes. Do not rely on explicit flush() calls for durability; only close() / __exit__ commits the upload.

Install

• pip install s3fs Standard install
• pip install 's3fs[boto3]' Install with boto3 extra (pins a compatible botocore automatically)
• conda install -c conda-forge s3fs conda-forge (resolves aiobotocore/botocore pins for you)

Imports

• S3FileSystem

  import s3fs
  s3 = s3fs.S3FileSystem()

  S3FileSystem is the only public entrypoint; importing from s3fs.core is internal and may break across releases. Always use import s3fs then s3fs.S3FileSystem().
• S3File

  with s3fs.S3FileSystem().open('bucket/key', 'rb') as f: ...

  S3File is returned by open(); never instantiate it directly. Only binary modes are supported: r, w, a, rb, wb, ab.
• open via fsspec URL

  import fsspec
  with fsspec.open('s3://bucket/key', 'rb') as f: ...

  fsspec.open with an s3:// URL will automatically dispatch to S3FileSystem; pass storage_options dict for credentials.

Quickstart

Connect with explicit credentials from environment variables, list a bucket, read a file, and write a file.

import os
import s3fs

# Credentials via env vars (boto chain also checks ~/.aws/credentials, IAM roles, etc.)
fs = s3fs.S3FileSystem(
    key=os.environ.get('AWS_ACCESS_KEY_ID', ''),
    secret=os.environ.get('AWS_SECRET_ACCESS_KEY', ''),
    # token=os.environ.get('AWS_SESSION_TOKEN', ''),  # uncomment for STS/assumed-role
    # endpoint_url='https://s3.example.com',          # uncomment for MinIO / S3-compatible
)

# List bucket contents
bucket = os.environ.get('S3_BUCKET', 'my-bucket')
print(fs.ls(bucket))

# Read a file
with fs.open(f'{bucket}/hello.txt', 'rb') as f:
    print(f.read())

# Write a file (must flush >5 MiB for multipart; context manager handles this)
with fs.open(f'{bucket}/output.txt', 'wb') as f:
    f.write(b'hello s3fs')

# Works transparently with pandas via storage_options
import pandas as pd
df = pd.read_csv(
    f's3://{bucket}/data.csv',
    storage_options={
        'key': os.environ.get('AWS_ACCESS_KEY_ID', ''),
        'secret': os.environ.get('AWS_SECRET_ACCESS_KEY', ''),
    },
)
print(df.head())