TensorFlow I/O

0.37.1 · active · verified Wed Apr 15

TensorFlow I/O is an extension library that provides a collection of file systems and file formats not natively available in TensorFlow's built-in support. It enhances data input/output capabilities for machine learning workflows, integrating seamlessly with TensorFlow's `tf.data` API. The current version is 0.37.1 and it maintains a frequent release cadence, often with monthly or bi-monthly updates to support new TensorFlow versions and features.

Warnings

breaking TensorFlow I/O requires a matching version of TensorFlow to ensure compatibility. Mismatched versions can lead to runtime errors or `UnimplementedError` (e.g., `File system scheme 's3' not implemented`). Always check the TensorFlow I/O release notes or PyPI page for the recommended TensorFlow compatibility table.
Fix: Ensure `tensorflow-io` and `tensorflow` versions are compatible. For example, `tensorflow-io==0.37.x` is compatible with `tensorflow==2.16.x`. Upgrade or downgrade one of the packages to match the recommended version.
breaking TensorFlow I/O has specific Python version requirements (e.g., `<3.13, >=3.7` for version 0.37.1). Using an unsupported Python version will result in `ERROR: No matching distribution found` during installation.
Fix: Install `tensorflow-io` in a Python environment that satisfies its `Requires-Python` metadata. For example, if using Python 3.13, you will need to wait for a `tensorflow-io` release that supports it.
breaking Starting with TensorFlow 2.7, core TensorFlow migrated S3 and HDFS filesystem support to the `tensorflow-io` project. Direct usage of `tf.io.gfile` for S3 paths without importing `tensorflow_io` will result in `tensorflow.python.framework.errors_impl.UnimplementedError: File system scheme 's3' not implemented`.
Fix: Ensure `tensorflow-io` is installed and explicitly imported (`import tensorflow_io as tfio`) in your Python code before attempting to access S3 or HDFS filesystems via `tf.io.gfile`.
gotcha Importing `tensorflow_io` after TensorFlow has already implicitly registered a default S3 filesystem (which can happen in some environments or with specific TensorFlow versions) might lead to `tensorflow.python.framework.errors_impl.AlreadyExistsError: File system for s3 already registered`.
Fix: To avoid filesystem registration conflicts, import `tensorflow_io` as early as possible in your script, preferably right after `import tensorflow as tf`.
gotcha TensorFlow I/O does not officially provide pre-built wheel distributions for Windows. Attempting `pip install tensorflow-io` on Windows will often result in `ERROR: No matching distribution found`, even if the Python version is theoretically compatible.
Fix: For Windows users, it is recommended to use the Windows Subsystem for Linux (WSL) to run `tensorflow-io` within a Linux environment, or build from source if necessary (which is complex).
gotcha When building `tensorflow-io` from source or running tests directly, especially on macOS with older Python versions (e.g., system default Python 3.8.2 on macOS 10.15.7), compiler options (`-arch arm64 -arch x86_64`) can cause regex installation errors.
Fix: When building from source on affected macOS systems, export `ARCHFLAGS="-arch x86_64"` before compilation to explicitly target x86_64 architecture and avoid conflicts. The `TFIO_DATAPATH` environment variable also needs to be set when running tests or installing from source locally.

Install

pip install tensorflow-io Install stable version

Imports

tfio
```
import tensorflow_io as tfio
```

Quickstart

This quickstart demonstrates loading the MNIST dataset directly from compressed URLs using `tfio.IODataset.from_mnist`, showcasing TensorFlow I/O's ability to handle remote filesystems and various data formats. The data is then preprocessed and used to train a simple Keras model.

import tensorflow as tf
import tensorflow_io as tfio

# Read the MNIST data into an IODataset directly from URLs
dataset_url = "https://storage.googleapis.com/cvdf-datasets/mnist/"
d_train = tfio.IODataset.from_mnist(
    dataset_url + "train-images-idx3-ubyte.gz",
    dataset_url + "train-labels-idx1-ubyte.gz",
)

# Shuffle the elements of the dataset.
d_train = d_train.shuffle(buffer_size=1024)

# By default image data is uint8, so convert to float32 using map().
d_train = d_train.map(lambda x, y: (tf.image.convert_image_dtype(x, tf.float32), y))

# Prepare batches the data just like any other tf.data.Dataset
d_train = d_train.batch(32)

# Build a simple Keras model
model = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(input_shape=(28, 28)),
    tf.keras.layers.Dense(512, activation=tf.nn.relu),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(10, activation=tf.nn.softmax),
])

# Compile and fit the model (example uses a small number of steps for brevity)
model.compile(
    optimizer='adam',
    loss='sparse_categorical_crossentropy',
    metrics=['accuracy']
)

print("Fitting model...")
# Use a small number of steps per epoch for quick demonstration
model.fit(d_train, epochs=1, steps_per_epoch=2)
print("Model fitting complete.")

view raw JSON →