Amazon S3 Filesystem for PyFilesystem2

Warnings

• breaking Version 1.1.1 (released 2019) introduced dependency bumps (PyFilesystem2 to 2.4, boto3 to 1.9). This may cause compatibility issues if your project relies on older versions of these underlying libraries, or if their APIs changed in ways not backward compatible with existing code.
  Fix: Ensure your project's `fs` and `boto3` versions are compatible with `fs-s3fs==1.1.1` and consider potential API changes in those libraries.
• gotcha The library has not been updated since August 2019. While functional, it may not be fully compatible with the absolute latest versions of `boto3` or `PyFilesystem2` released since then, or may lack support for newer S3 features (e.g., specific storage classes, access points, etc.).
  Fix: Thoroughly test `fs-s3fs` with your specific versions of `boto3` and `PyFilesystem2`. For new projects, evaluate if a more actively maintained S3 filesystem backend is available if you require cutting-edge features or strict compatibility with the latest dependency versions.
• gotcha Credentials for S3 are handled by `boto3`. If `access_key` and `secret_key` are not explicitly passed to the `S3FS` constructor, `boto3` will use its standard credential chain (environment variables like `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, IAM roles, AWS config files). Incorrect credential setup is a common source of connection issues.
  Fix: Always ensure valid AWS credentials are available to `boto3` through environment variables, IAM roles, or by explicitly passing `access_key` and `secret_key` (and optionally `region_name`) to the `S3FS` constructor.
• gotcha The `bucket_name` parameter is mandatory for the `S3FS` constructor. Omitting it will result in an `TypeError` or unexpected behavior, as the filesystem needs a specific S3 bucket as its root to operate correctly.
  Fix: Always provide the `bucket_name` argument when instantiating `S3FS`.

Install

• pip install fs-s3fs Install latest version

Imports

• S3FS

  from s3fs import S3FS

Quickstart

This quickstart demonstrates how to initialize an `S3FS` object, connect to an S3 bucket, create/read/list/delete files and directories. Ensure AWS credentials are set via environment variables or passed directly, and replace `your-unique-s3-bucket-name` with an actual S3 bucket you have access to.

import os
from s3fs import S3FS

# For local testing, ensure AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are set
# in your environment variables. You can also pass them directly to the constructor.
# AWS_REGION_NAME can also be specified if needed.

access_key = os.environ.get('AWS_ACCESS_KEY_ID', 'YOUR_ACCESS_KEY')
secret_key = os.environ.get('AWS_SECRET_ACCESS_KEY', 'YOUR_SECRET_KEY')
bucket_name = 'your-unique-s3-bucket-name' # IMPORTANT: Replace with an existing S3 bucket name

if access_key == 'YOUR_ACCESS_KEY' or secret_key == 'YOUR_SECRET_KEY':
    print("WARNING: Please set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.")
    print("         This quickstart will likely fail without valid AWS credentials.")

try:
    # Initialize S3FS. If access_key/secret_key are None, boto3 will look for env vars or IAM roles.
    fs_s3 = S3FS(bucket_name=bucket_name,
                 access_key=access_key if access_key != 'YOUR_ACCESS_KEY' else None,
                 secret_key=secret_key if secret_key != 'YOUR_SECRET_KEY' else None)

    print(f"Successfully connected to S3 bucket: {bucket_name}")

    # Example 1: List contents of the bucket root
    print(f"\nContents of '{bucket_name}' (root):")
    for info in fs_s3.listdir('/', detail=True):
        print(f"- {info['name']} (type: {info['type']}) {'(directory)' if info['is_dir'] else ''}")

    # Example 2: Create a directory
    new_dir = 'test_s3fs_dir'
    fs_s3.makedir(new_dir, recreate=True)
    print(f"\nDirectory '{new_dir}' created.")

    # Example 3: Create a file inside the new directory
    file_path = f"{new_dir}/hello.txt"
    with fs_s3.open(file_path, 'w') as f:
        f.write("Hello from fs-s3fs on S3!")
    print(f"File '{file_path}' created with content.")

    # Example 4: Read the file
    with fs_s3.open(file_path, 'r') as f:
        content = f.read()
    print(f"Content of '{file_path}': '{content}'")

    # Example 5: Remove the file (clean up)
    fs_s3.remove(file_path)
    print(f"File '{file_path}' removed.")

    # Example 6: Remove the directory
    fs_s3.removedir(new_dir)
    print(f"Directory '{new_dir}' removed.")

except Exception as e:
    print(f"\nAn error occurred: {e}")
finally:
    if 'fs_s3' in locals():
        fs_s3.close() # Important to close the filesystem
        print("\nS3FS connection closed.")