{"id":260,"library":"s3fs","title":"S3Fs","description":"S3Fs is a Pythonic filesystem interface to Amazon S3, built on top of aiobotocore and fsspec. The top-level class S3FileSystem exposes familiar file-system operations (ls, cp, mv, du, glob, put, get) and a file open() API that emulates Python's standard file protocol, making it a drop-in for libraries like pandas, dask, and gzip that accept file-like objects. It also supports S3-compatible stores (MinIO, Ceph, R2) via the endpoint_url parameter. Versions follow calendar versioning (YYYY.MM.PATCH); the current release is 2026.2.0, released February 2026, with roughly monthly cadence.","status":"active","version":"2026.2.0","language":"python","source_language":"en","source_url":"https://github.com/fsspec/s3fs","tags":["aws","s3","filesystem","fsspec","aiobotocore","async","cloud-storage","pandas","dask"],"install":[{"cmd":"pip install s3fs","lang":"bash","label":"Standard install"},{"cmd":"pip install 's3fs[boto3]'","lang":"bash","label":"Install with boto3 extra (pins a compatible botocore automatically)"},{"cmd":"conda install -c conda-forge s3fs","lang":"bash","label":"conda-forge (resolves aiobotocore/botocore pins for you)"}],"dependencies":[{"reason":"Core async AWS client used internally; pins a narrow botocore range — this is the primary source of dependency conflicts when boto3 is also installed","package":"aiobotocore","optional":false},{"reason":"Abstract filesystem interface that S3FileSystem inherits from; s3fs and fsspec versions must be kept in sync","package":"fsspec","optional":false},{"reason":"Available as the [boto3] extra; needed only if you use get_delegated_s3pars() or STS credential delegation","package":"boto3","optional":true},{"reason":"Transitive dependency of aiobotocore; guards ClientPayloadError handling for incomplete-read retries","package":"aiohttp","optional":true}],"imports":[{"note":"S3FileSystem is the only public entrypoint; importing from s3fs.core is internal and may break across releases. Always use import s3fs then s3fs.S3FileSystem().","wrong":"from s3fs.core import S3FileSystem","symbol":"S3FileSystem","correct":"import s3fs\ns3 = s3fs.S3FileSystem()"},{"note":"S3File is returned by open(); never instantiate it directly. Only binary modes are supported: r, w, a, rb, wb, ab.","symbol":"S3File","correct":"with s3fs.S3FileSystem().open('bucket/key', 'rb') as f: ..."},{"note":"fsspec.open with an s3:// URL will automatically dispatch to S3FileSystem; pass storage_options dict for credentials.","symbol":"open via fsspec URL","correct":"import fsspec\nwith fsspec.open('s3://bucket/key', 'rb') as f: ..."}],"quickstart":{"code":"import os\nimport s3fs\n\n# Credentials via env vars (boto chain also checks ~/.aws/credentials, IAM roles, etc.)\nfs = s3fs.S3FileSystem(\n    key=os.environ.get('AWS_ACCESS_KEY_ID', ''),\n    secret=os.environ.get('AWS_SECRET_ACCESS_KEY', ''),\n    # token=os.environ.get('AWS_SESSION_TOKEN', ''),  # uncomment for STS/assumed-role\n    # endpoint_url='https://s3.example.com',          # uncomment for MinIO / S3-compatible\n)\n\n# List bucket contents\nbucket = os.environ.get('S3_BUCKET', 'my-bucket')\nprint(fs.ls(bucket))\n\n# Read a file\nwith fs.open(f'{bucket}/hello.txt', 'rb') as f:\n    print(f.read())\n\n# Write a file (must flush >5 MiB for multipart; context manager handles this)\nwith fs.open(f'{bucket}/output.txt', 'wb') as f:\n    f.write(b'hello s3fs')\n\n# Works transparently with pandas via storage_options\nimport pandas as pd\ndf = pd.read_csv(\n    f's3://{bucket}/data.csv',\n    storage_options={\n        'key': os.environ.get('AWS_ACCESS_KEY_ID', ''),\n        'secret': os.environ.get('AWS_SECRET_ACCESS_KEY', ''),\n    },\n)\nprint(df.head())\n","lang":"python","description":"Connect with explicit credentials from environment variables, list a bucket, read a file, and write a file."},"warnings":[{"fix":"Use 'pip install s3fs[boto3]' to get a pre-validated boto3+botocore combination, or prefer conda-forge which pre-solves the triangle. Never pin boto3 and s3fs independently with ^ in Poetry without checking aiobotocore's exact botocore requirement first.","message":"aiobotocore pins an extremely narrow botocore version range (often a single patch). Installing s3fs alongside boto3 or awscli frequently produces irresolvable dependency conflicts because boto3 requires a different botocore range.","severity":"breaking","affected_versions":"all"},{"fix":"Set multiprocessing.set_start_method('spawn') or 'forkserver' before creating S3FileSystem instances, or avoid sharing S3FileSystem objects across fork boundaries.","message":"Using multiprocessing with the default 'fork' start method causes deadlocks and hard-to-reproduce bugs because s3fs keeps open async sockets and a background thread.","severity":"breaking","affected_versions":"all"},{"fix":"Pin to >=2023.12.1. Run 'pip install s3fs>=2023.12.1' to avoid the yanked release.","message":"s3fs version 2023.12.0 was yanked from PyPI due to an authentication regression. pip may still resolve to it on some platforms if not using --pre filtering.","severity":"breaking","affected_versions":"2023.12.0"},{"fix":"Call fs.invalidate_cache() after external writes, or pass refresh=True to fs.info(). For high-churn workloads consider listings_expiry_time= when constructing S3FileSystem.","message":"The directory listing cache (dircache) is not invalidated automatically. If an object is written or resized externally (e.g. by boto3 or another process) after fs.ls() or fs.info() has cached its metadata, subsequent reads via the same S3FileSystem instance will use stale size information and may return corrupted or truncated data.","severity":"gotcha","affected_versions":"all"},{"fix":"Use 'rb'/'wb' modes explicitly. For text, wrap with io.TextIOWrapper: io.TextIOWrapper(fs.open('bucket/file', 'rb'), encoding='utf-8').","message":"File access is always binary. Text mode ('r', 'w') is technically accepted but returns bytes or requires an explicit encoding wrapper. readline() and line iteration work but the underlying stream is always bytes.","severity":"gotcha","affected_versions":"all"},{"fix":"Pass skip_instance_cache=True when constructing S3FileSystem if you need isolated instances, e.g. with different endpoint_urls or IAM roles.","message":"S3FileSystem instances are cached as singletons by default (skip_instance_cache=False). Two calls with the same credentials return the same object, which can cause credential or config bleed between parts of an application that expect independent connections.","severity":"gotcha","affected_versions":"all"},{"fix":"Always use 'with fs.open(..., \"wb\") as f:' context managers for writes. Do not rely on explicit flush() calls for durability; only close() / __exit__ commits the upload.","message":"Writes to S3 are not flushed until the file is closed (or the multipart threshold of ~150 MiB is hit). Calling f.write() without closing inside a context manager means data is buffered locally and nothing is committed to S3 on partial writes.","severity":"gotcha","affected_versions":"all"},{"fix":"Ensure AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables are set, or that an appropriate IAM role is available to the execution environment. Alternatively, pass explicit credentials to the S3FileSystem constructor.","message":"S3FileSystem requires valid AWS credentials to be configured (e.g., via environment variables, IAM roles, or ~/.aws/credentials). Failure to provide these results in an 'AuthorizationHeaderMalformed' error.","severity":"breaking","affected_versions":"all"},{"fix":"Ensure AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables are correctly set, or pass `key` and `secret` parameters directly to the S3FileSystem constructor. If using IAM roles, verify the instance profile or role assumption is properly configured and providing valid temporary credentials.","message":"An `AuthorizationHeaderMalformed` error, often stating 'a non-empty Access Key (AKID) must be provided in the credential', means that s3fs could not find or use valid AWS credentials. This typically happens when environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) are unset or empty, or explicit `key` and `secret` parameters are missing/incorrect from the `S3FileSystem` constructor.","severity":"breaking","affected_versions":"all"}],"env_vars":null,"last_verified":"2026-05-12T12:27:07.002Z","next_check":"2026-06-25T00:00:00.000Z","problems":[{"fix":"pip install s3fs","cause":"The 's3fs' library has not been installed in the current Python environment.","error":"ModuleNotFoundError: No module named 's3fs'"},{"fix":"pip install s3fs","cause":"When using `fsspec`-dependent libraries like pandas or dask to access S3 paths, `s3fs` must be installed to provide the S3 filesystem implementation.","error":"fsspec.exceptions.NoS3FSImplementation: No s3fs implementation found, install s3fs to use s3://"},{"fix":"Ensure your AWS user or role has the required IAM policies (e.g., `s3:GetObject`, `s3:ListBucket`) for the target S3 resources, and that your credentials are correctly configured (environment variables, `~/.aws/credentials`, or IAM role).","cause":"The AWS credentials configured for `s3fs` lack the necessary IAM permissions to perform the requested S3 operation on the specified bucket or object.","error":"botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the GetObject operation: Access Denied."},{"fix":"s3 = s3fs.S3FileSystem(client_kwargs={'endpoint_url': 'http://localhost:9000'})","cause":"The `endpoint_url` parameter for S3-compatible storage should be passed within the `client_kwargs` dictionary, not as a direct argument to `S3FileSystem`.","error":"TypeError: S3FileSystem.__init__() got an unexpected keyword argument 'endpoint_url'"},{"fix":"Verify the S3 path for correctness, ensure the bucket exists, and confirm that your AWS credentials have appropriate list and read permissions for the specified location.","cause":"The specified S3 path (bucket or object) does not exist or is inaccessible with the provided credentials.","error":"FileNotFoundError: No such file or directory: 's3://my-bucket/non-existent-path'"}],"ecosystem":"pypi","meta_description":null,"install_score":100,"install_tag":"verified","quickstart_score":0,"quickstart_tag":"stale","pypi_latest":null,"install_checks":{"last_tested":"2026-05-12","tag":"verified","tag_description":"installs cleanly on critical runtimes, fast import, recently tested","results":[{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":"boto3","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.93,"mem_mb":23.6,"disk_size":"61.0M"},{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.97,"mem_mb":23.6,"disk_size":"61.0M"},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":"boto3","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.73,"mem_mb":23.6,"disk_size":"63M"},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.72,"mem_mb":23.6,"disk_size":"63M"},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":"boto3","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":1.24,"mem_mb":25.7,"disk_size":"65.3M"},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":1.24,"mem_mb":25.7,"disk_size":"65.3M"},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":"boto3","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":1.13,"mem_mb":25.7,"disk_size":"68M"},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":1.02,"mem_mb":25.7,"disk_size":"68M"},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":"boto3","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":1.31,"mem_mb":25.8,"disk_size":"56.7M"},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":1.31,"mem_mb":25.8,"disk_size":"56.7M"},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":"boto3","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":1.26,"mem_mb":25.8,"disk_size":"59M"},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":1.39,"mem_mb":25.8,"disk_size":"59M"},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":"boto3","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":1.25,"mem_mb":26.3,"disk_size":"56.1M"},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":1.27,"mem_mb":26.3,"disk_size":"56.1M"},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":"boto3","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":1.24,"mem_mb":26.3,"disk_size":"58M"},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":1.31,"mem_mb":26.3,"disk_size":"58M"},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":"boto3","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.92,"mem_mb":22.8,"disk_size":"62.1M"},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.93,"mem_mb":22.8,"disk_size":"60.0M"},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":"boto3","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.81,"mem_mb":22.8,"disk_size":"65M"},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.8,"mem_mb":22.8,"disk_size":"63M"}]},"quickstart_checks":{"last_tested":"2026-04-23","tag":"stale","tag_description":"widespread failures or data too old to trust","results":[{"runtime":"python:3.10-alpine","exit_code":1},{"runtime":"python:3.10-slim","exit_code":1},{"runtime":"python:3.11-alpine","exit_code":1},{"runtime":"python:3.11-slim","exit_code":1},{"runtime":"python:3.12-alpine","exit_code":1},{"runtime":"python:3.12-slim","exit_code":1},{"runtime":"python:3.13-alpine","exit_code":1},{"runtime":"python:3.13-slim","exit_code":1},{"runtime":"python:3.9-alpine","exit_code":1},{"runtime":"python:3.9-slim","exit_code":1}]}}