{"id":6909,"library":"tensorflow-io","title":"TensorFlow I/O","description":"TensorFlow I/O is an extension library that provides a collection of file systems and file formats not natively available in TensorFlow's built-in support. It enhances data input/output capabilities for machine learning workflows, integrating seamlessly with TensorFlow's `tf.data` API. The current version is 0.37.1 and it maintains a frequent release cadence, often with monthly or bi-monthly updates to support new TensorFlow versions and features.","status":"active","version":"0.37.1","language":"en","source_language":"en","source_url":"https://github.com/tensorflow/io","tags":["tensorflow","data-io","io","data-formats","filesystems","machine-learning"],"install":[{"cmd":"pip install tensorflow-io","lang":"bash","label":"Install stable version"}],"dependencies":[{"reason":"TensorFlow I/O extends TensorFlow's data handling capabilities and requires a compatible TensorFlow installation to function.","package":"tensorflow"}],"imports":[{"symbol":"tfio","correct":"import tensorflow_io as tfio"}],"quickstart":{"code":"import tensorflow as tf\nimport tensorflow_io as tfio\n\n# Read the MNIST data into an IODataset directly from URLs\ndataset_url = \"https://storage.googleapis.com/cvdf-datasets/mnist/\"\nd_train = tfio.IODataset.from_mnist(\n    dataset_url + \"train-images-idx3-ubyte.gz\",\n    dataset_url + \"train-labels-idx1-ubyte.gz\",\n)\n\n# Shuffle the elements of the dataset.\nd_train = d_train.shuffle(buffer_size=1024)\n\n# By default image data is uint8, so convert to float32 using map().\nd_train = d_train.map(lambda x, y: (tf.image.convert_image_dtype(x, tf.float32), y))\n\n# Prepare batches the data just like any other tf.data.Dataset\nd_train = d_train.batch(32)\n\n# Build a simple Keras model\nmodel = tf.keras.models.Sequential([\n    tf.keras.layers.Flatten(input_shape=(28, 28)),\n    tf.keras.layers.Dense(512, activation=tf.nn.relu),\n    tf.keras.layers.Dropout(0.2),\n    tf.keras.layers.Dense(10, activation=tf.nn.softmax),\n])\n\n# Compile and fit the model (example uses a small number of steps for brevity)\nmodel.compile(\n    optimizer='adam',\n    loss='sparse_categorical_crossentropy',\n    metrics=['accuracy']\n)\n\nprint(\"Fitting model...\")\n# Use a small number of steps per epoch for quick demonstration\nmodel.fit(d_train, epochs=1, steps_per_epoch=2)\nprint(\"Model fitting complete.\")","lang":"python","description":"This quickstart demonstrates loading the MNIST dataset directly from compressed URLs using `tfio.IODataset.from_mnist`, showcasing TensorFlow I/O's ability to handle remote filesystems and various data formats. The data is then preprocessed and used to train a simple Keras model."},"warnings":[{"fix":"Ensure `tensorflow-io` and `tensorflow` versions are compatible. For example, `tensorflow-io==0.37.x` is compatible with `tensorflow==2.16.x`. Upgrade or downgrade one of the packages to match the recommended version.","message":"TensorFlow I/O requires a matching version of TensorFlow to ensure compatibility. Mismatched versions can lead to runtime errors or `UnimplementedError` (e.g., `File system scheme 's3' not implemented`). Always check the TensorFlow I/O release notes or PyPI page for the recommended TensorFlow compatibility table.","severity":"breaking","affected_versions":"<0.37.0"},{"fix":"Install `tensorflow-io` in a Python environment that satisfies its `Requires-Python` metadata. For example, if using Python 3.13, you will need to wait for a `tensorflow-io` release that supports it.","message":"TensorFlow I/O has specific Python version requirements (e.g., `<3.13, >=3.7` for version 0.37.1). Using an unsupported Python version will result in `ERROR: No matching distribution found` during installation.","severity":"breaking","affected_versions":"All versions"},{"fix":"Ensure `tensorflow-io` is installed and explicitly imported (`import tensorflow_io as tfio`) in your Python code before attempting to access S3 or HDFS filesystems via `tf.io.gfile`.","message":"Starting with TensorFlow 2.7, core TensorFlow migrated S3 and HDFS filesystem support to the `tensorflow-io` project. Direct usage of `tf.io.gfile` for S3 paths without importing `tensorflow_io` will result in `tensorflow.python.framework.errors_impl.UnimplementedError: File system scheme 's3' not implemented`.","severity":"breaking","affected_versions":">=2.7.0 of TensorFlow"},{"fix":"To avoid filesystem registration conflicts, import `tensorflow_io` as early as possible in your script, preferably right after `import tensorflow as tf`.","message":"Importing `tensorflow_io` after TensorFlow has already implicitly registered a default S3 filesystem (which can happen in some environments or with specific TensorFlow versions) might lead to `tensorflow.python.framework.errors_impl.AlreadyExistsError: File system for s3 already registered`.","severity":"gotcha","affected_versions":"All versions (intermittent)"},{"fix":"For Windows users, it is recommended to use the Windows Subsystem for Linux (WSL) to run `tensorflow-io` within a Linux environment, or build from source if necessary (which is complex).","message":"TensorFlow I/O does not officially provide pre-built wheel distributions for Windows. Attempting `pip install tensorflow-io` on Windows will often result in `ERROR: No matching distribution found`, even if the Python version is theoretically compatible.","severity":"gotcha","affected_versions":"All versions on Windows"},{"fix":"When building from source on affected macOS systems, export `ARCHFLAGS=\"-arch x86_64\"` before compilation to explicitly target x86_64 architecture and avoid conflicts. The `TFIO_DATAPATH` environment variable also needs to be set when running tests or installing from source locally.","message":"When building `tensorflow-io` from source or running tests directly, especially on macOS with older Python versions (e.g., system default Python 3.8.2 on macOS 10.15.7), compiler options (`-arch arm64 -arch x86_64`) can cause regex installation errors.","severity":"gotcha","affected_versions":"Source builds on specific macOS/Python combinations"}],"env_vars":null,"last_verified":"2026-04-15T00:00:00.000Z","next_check":"2026-07-14T00:00:00.000Z","problems":[]}