TFX Basic Shared Libraries (tfx-bsl)

Common errors

• tensorflow.python.framework.errors_impl.NotFoundError: ...tfx_bsl_extension.so: undefined symbol: _ZN10tensorflow7tfrt_gpu...

  cause Mismatch between the TensorFlow version and the `tfx-bsl` C++ extension. `tfx-bsl` was compiled against a different TensorFlow Application Binary Interface (ABI).
  fix Ensure `tfx-bsl` and `tensorflow` are installed from compatible versions. The safest way is to install `tfx` itself, which manages these dependencies, or consult the TFX compatibility matrix (`https://www.tensorflow.org/tfx/releases#python_package_compatibility`).

• ImportError: cannot import name 'DataType' from 'pyarrow.lib'

  cause Incompatible versions of `tfx-bsl` and `pyarrow`. `pyarrow`'s internal API for `DataType` changed in an incompatible way.
  fix Upgrade or downgrade `pyarrow` to a version compatible with your `tfx-bsl` installation. If installing `tfx`, this dependency should be managed automatically. If installing manually, check `tfx-bsl`'s `install_requires` for `pyarrow` constraints.

• TypeError: descriptor 'decode' requires a 'tfx_bsl.coders.example_coder.ExampleToRecordBatchDecoder' object but received a 'bytes' object

  cause Attempting to call `decode` directly on a single byte string instead of an instance of the `decoder` class, or passing a single example instead of an iterable of examples.
  fix Ensure you instantiate the decoder first (e.g., `decoder = ExampleToRecordBatchDecoder()`) and pass an iterable (list, tuple) of serialized examples: `record_batch = decoder.decode([example1_bytes, example2_bytes])`.

Warnings

• breaking Version mismatches between `tfx-bsl`, `tensorflow`, `tfx`, and `apache-beam` are the most common cause of runtime errors. Ensure all TFX-related packages are installed with compatible versions, ideally from the same TFX release train.
  Fix: Always install `tfx` (which pins `tfx-bsl`) or refer to the official TFX compatibility matrix: `https://www.tensorflow.org/tfx/releases#python_package_compatibility`.
• gotcha Python 3.8 and earlier are not supported by recent `tfx-bsl` versions. Python 4.x is also not yet supported. The package specifically targets Python 3.9-3.11.
  Fix: Ensure your environment uses Python 3.9, 3.10, or 3.11. Check `requires_python` from PyPI for the exact range for your specific `tfx-bsl` version.
• gotcha Using `pip install tfx-bsl` without explicitly installing `tensorflow` or its `[tensorflow]` extra will result in a version of `tfx-bsl` that might not be fully functional or compatible with your existing TensorFlow installation.
  Fix: Prefer `pip install tfx` (which includes `tfx-bsl` and `tensorflow`) or `pip install tfx-bsl[tensorflow]` to ensure core dependencies are aligned.
• breaking TFX-BSL relies on custom C++ extensions. Issues with build environments, incompatible compilers, or missing system dependencies (like specific glibc versions) can lead to `ImportError` or segmentation faults.
  Fix: Use pre-built wheels if possible. If building from source, ensure your C++ toolchain (gcc/g++) is compatible with TensorFlow's requirements and that all necessary libraries are present. Often, this means sticking to official Docker images or tested environments.

Install

• pip install tfx-bsl Install core tfx-bsl
• pip install tfx-bsl[tensorflow] Install with TensorFlow dependency (recommended for full functionality)

Imports

• ExampleToRecordBatchDecoder

  from tfx_bsl.coders.example_coder import ExampleToRecordBatchDecoder

  Commonly used for converting `tf.train.Example` protobufs to Apache Arrow `RecordBatch`.
• RecordBatchToExamplesEncoder

  from tfx_bsl.coders.example_coder import RecordBatchToExamplesEncoder

  Used for converting Apache Arrow `RecordBatch` back to `tf.train.Example` protobufs.

Quickstart

Demonstrates how to convert `tf.train.Example` protobufs to an Apache Arrow `RecordBatch` and back using `tfx_bsl`'s `ExampleToRecordBatchDecoder` and `RecordBatchToExamplesEncoder`. This is a core data transformation task that `tfx-bsl` facilitates for TFX components.

import tensorflow as tf
import pyarrow as pa
from tfx_bsl.coders.example_coder import ExampleToRecordBatchDecoder, RecordBatchToExamplesEncoder

# 1. Create a list of serialized tf.train.Example
examples_list = [
    tf.train.Example(features=tf.train.Features(feature={
        'feature1': tf.train.Feature(int64_list=tf.train.Int64List(value=[1, 2])),
        'feature2': tf.train.Feature(float_list=tf.train.FloatList(value=[1.0, 2.0]))
    })).SerializeToString(),
    tf.train.Example(features=tf.train.Features(feature={
        'feature1': tf.train.Feature(int64_list=tf.train.Int64List(value=[3])),
        'feature2': tf.train.Feature(float_list=tf.train.FloatList(value=[3.0]))
    })).SerializeToString()
]

# 2. Decode TF.Examples to an Apache Arrow RecordBatch
decoder = ExampleToRecordBatchDecoder()
record_batch = decoder.decode(examples_list)

print(f"\nDecoded RecordBatch schema:\n{record_batch.schema}")
print(f"Decoded RecordBatch content:\n{record_batch}")

# 3. Encode the Apache Arrow RecordBatch back to TF.Examples
encoder = RecordBatchToExamplesEncoder(record_batch.schema)
encoded_examples_iterator = encoder.encode(record_batch)
encoded_examples_list = list(encoded_examples_iterator)

print(f"\nRe-encoded examples (first one):\n{tf.train.Example().FromString(encoded_examples_list[0])}")

# Verify round-trip (simplified check)
assert len(examples_list) == len(encoded_examples_list)
print("\nSuccessfully decoded to Arrow and re-encoded to TF.Example.")