TensorFlow Model Analysis

Common errors

• ValueError: No examples found for slicing spec

  cause The input data or model setup isn't correctly providing features or predictions that TFMA can use for the defined slicing specifications. This often means the `label_key` or features defined in `slicing_specs` do not exist in the parsed `tf.train.Example` protos.
  fix Verify that your `tf.train.Example` protos contain the features specified in your `eval_config.slicing_specs` and `model_specs.label_key`. Also, ensure the `signature_name` and `prediction_key` in `ModelSpec` are correct so the model can make predictions.

• ModuleNotFoundError: No module named 'apache_beam'

  cause `apache-beam` is a mandatory dependency for TFMA and was not installed or is not accessible in the current environment.
  fix Install Apache Beam: `pip install apache-beam`. If planning to use cloud runners like Dataflow, include the relevant extras, e.g., `pip install apache-beam[gcp]`.

• The `prediction_key` specified in `ModelSpec` did not match any predictions in the model output. Available predictions: [...]

  cause The `prediction_key` provided in `tfma.ModelSpec` does not correspond to a key in the dictionary output by the model's serving signature.
  fix Examine your `SavedModel`'s serving signature outputs (e.g., using `saved_model_cli show --dir <model_path> --tag_set serve`) and ensure `prediction_key` matches one of the output tensor names. If the model outputs a single tensor (not a dict), you might need to omit `prediction_key` or ensure the model wraps it in a dict with a specific key.

• AttributeError: 'tensorflow.python.framework.ops.EagerTensor' object has no attribute 'decode_raw'

  cause This error often occurs when `tf.Example` protos are not correctly parsed before being passed to the model, or when the `tfma.FeatureSpec` (often inferred or default) does not align with the actual data types in your `tf.train.Example`.
  fix Ensure your `tf.train.Example` features are correctly defined (e.g., using `tf.train.FloatList` for floats, `tf.train.BytesList` for strings) and that your `ModelSpec` (or any custom `input_fn` if used) correctly specifies how to parse these features into tensors for the model.

Warnings

• breaking TFMA has strict requirements on Python, TensorFlow, and Apache Beam versions. Upgrading one without considering the others can lead to installation or runtime errors. For instance, Python 3.8 support was dropped in 0.45.0, and `tensorflow` must be `~2.11` and `apache-beam` `~2.54` for 0.48.0.
  Fix: Always check the official `setup.py` or `pyproject.toml` for exact dependency ranges. Ensure your Python environment matches the `requires_python` range and that `tensorflow` and `apache-beam` versions are compatible.
• gotcha TFMA expects input data in `tf.train.Example` format, typically stored in TFRecord files. It cannot directly consume CSV, JSON, or other raw formats without prior conversion.
  Fix: Pre-process your data into `tf.train.Example` protos and write them to TFRecord files. Libraries like `tfx_bsl.public.tf_example_io` can assist with this.
• gotcha Incorrectly configuring `ModelSpec` parameters (`signature_name`, `label_key`, `prediction_key`) is a common source of errors. These must precisely match your `SavedModel`'s serving signature and the feature/output keys.
  Fix: Inspect your `SavedModel` using `saved_model_cli show` to confirm signature names and input/output tensor keys. Ensure `label_key` matches your TFRecord feature name and `prediction_key` matches the relevant output key from your model's prediction dictionary.
• gotcha Apache Beam (which TFMA uses for its pipelines) requires specific extras for different runners (e.g., `[direct]` for local, `[gcp]` for Dataflow). Missing these can cause 'ModuleNotFoundError' or runtime errors when trying to use a runner.
  Fix: Install `apache-beam` with the necessary extras, for example: `pip install apache-beam[direct,gcp]`. Ensure your Beam pipeline options (`beam_options` in `tfma.run_model_analysis`) are correctly configured for your chosen runner.

Install

• pip install tensorflow-model-analysis apache-beam[direct,gcp] tensorflow Install TFMA with Beam dependencies

Imports

• tfma

  import tensorflow_model_analysis as tfma

Quickstart

This quickstart demonstrates how to set up a dummy TensorFlow model, create synthetic TFRecord data, define an `EvalConfig`, and run a basic model analysis with TFMA locally. It shows how to obtain overall and sliced metrics.

import tensorflow as tf
import tensorflow_model_analysis as tfma
import os
import shutil

# 1. Create a simple Keras model and save it
# This model expects a 'feature_1' input and outputs 'prediction'.
class SimplePredictionModel(tf.keras.Model):
    def __init__(self):
        super().__init__()
        self.dense = tf.keras.layers.Dense(1, activation='sigmoid')

    @tf.function(input_signature=[
        tf.TensorSpec(shape=[None], dtype=tf.float32, name='feature_1')
    ])
    def serving_default(self, feature_1):
        return {'predictions': self.dense(tf.expand_dims(feature_1, axis=-1))}

model = SimplePredictionModel()
# Initialize weights by calling the serving function once
_ = model.serving_default(tf.constant([1.0, 2.0]))

model_dir = '/tmp/tfma_quickstart_model'
if os.path.exists(model_dir): shutil.rmtree(model_dir)
tf.saved_model.save(model, model_dir, signatures={'serving_default': model.serving_default})

# 2. Create dummy data as TFRecord (TFMA expects tf.train.Example protos)
data_path = '/tmp/tfma_quickstart_data.tfrecord'
if os.path.exists(data_path): os.remove(data_path)

examples_proto = []
for i in range(10):
    feature_val = float(i)
    label_val = 1.0 if i % 2 == 0 else 0.0
    example = tf.train.Example(features=tf.train.Features(feature={
        'feature_1': tf.train.Feature(float_list=tf.train.FloatList(value=[feature_val])),
        'label': tf.train.Feature(float_list=tf.train.FloatList(value=[label_val])),
    }))
    examples_proto.append(example.SerializeToString())

with tf.io.TFRecordWriter(data_path) as writer:
    for ex in examples_proto:
        writer.write(ex)

# 3. Define EvalConfig
eval_config = tfma.EvalConfig(
    model_specs=[tfma.ModelSpec(
        signature_name='serving_default',
        label_key='label',
        prediction_key='predictions' # Key from model output dict
    )],
    metrics_specs=[
        tfma.MetricsSpec(
            metrics=[
                tfma.MetricConfig(class_name='ExampleCount'),
                tfma.MetricConfig(class_name='Accuracy')
            ]
        )
    ],
    slicing_specs=[
        tfma.SlicingSpec(), # Overall slice
        tfma.SlicingSpec(feature_keys=['feature_1']) # Slice by feature_1
    ]
)

# 4. Run Model Analysis
output_dir = '/tmp/tfma_quickstart_output'
if os.path.exists(output_dir): shutil.rmtree(output_dir)

print(f"Running TFMA with model: {model_dir}, data: {data_path}, output: {output_dir}")
results = tfma.run_model_analysis(
    model_location=model_dir,
    data_location=data_path,
    eval_config=eval_config,
    output_path=output_dir,
    # TFMA uses Apache Beam for execution. For local quickstart,
    # default DirectRunner is used. For cloud, configure Beam options.
    # e.g., beam_options=os.environ.get('BEAM_OPTIONS', '').split()
)

print(f"TFMA analysis complete. Results written to: {output_dir}")
# To inspect results (e.g., in a Jupyter Notebook):
# from tensorflow_model_analysis.notebook import visualization
# visualization.display_metrics(output_dir)