TensorFlow Metadata

Warnings

• breaking Frequent and critical dependency conflicts with the `protobuf` library. `tensorflow-metadata` often pins `protobuf` to specific major/minor versions, which can clash with other libraries in the TensorFlow ecosystem.
  Fix: Consult `tensorflow-metadata`'s `setup.py` or `RELEASE.md` for exact `protobuf` version requirements for your Python version (e.g., `protobuf>=4.25.2,<5` for Python 3.11). Consider using a virtual environment and carefully managing dependencies.
• deprecated Support for Python 3.8 was deprecated starting from version 1.15.0.
  Fix: Upgrade to Python 3.9 or higher. The library currently supports Python >=3.9,<4.
• gotcha Nightly builds of `tensorflow-metadata` (and related TF projects) are explicitly stated to be unstable and prone to breakages, with fixes potentially taking a week or more.
  Fix: Always use the stable versions available on PyPI (`pip install tensorflow-metadata`) for production or reliable development. Only use nightly builds if you need the absolute latest features and are prepared to handle instability.
• breaking Version 1.15.0 introduced a semantic change to how `min/max/avg/tot num-values` are calculated for nested features, now relying on the innermost level.
  Fix: If you rely on statistics for nested features, re-evaluate existing pipelines and logic when upgrading to 1.15.0 or later, as the reported values might change.
• breaking The field `NaturalLanguageDomain.location_constraint_regex` was removed in version 1.15.0. It was previously documented as 'please do not use' and was never fully implemented.
  Fix: Remove any usage of `NaturalLanguageDomain.location_constraint_regex` from your code.

Install

• pip install tensorflow-metadata Latest stable version

Imports

• Schema

  from tf_metadata.proto.v0 import schema_pb2

  For defining data schemas.
• Statistics

  from tf_metadata.proto.v0 import statistics_pb2

  For working with dataset statistics.
• MetadataStore

  from ml_metadata.metadata_store import metadata_store_pb2

  Used by ML Metadata (MLMD) for artifact and execution tracking, built on TFMD protobufs.

Quickstart

This quickstart demonstrates how to define a simple data schema using `tensorflow-metadata`'s protobuf definitions. It shows how to add features with different types and constraints, then serializes and deserializes the schema for storage or transfer.

from tf_metadata.proto.v0 import schema_pb2

# Create a simple schema definition
schema = schema_pb2.Schema()

# Add a feature named 'age' of type INT
feature_age = schema.feature.add()
feature_age.name = "age"
feature_age.type = schema_pb2.FeatureType.INT
feature_age.int_domain.is_categorical = False
feature_age.presence.min_fraction = 1.0 # 'age' must always be present
feature_age.int_domain.min = 0
feature_age.int_domain.max = 120

# Add a feature named 'city' of type BYTES (string), which is categorical
feature_city = schema.feature.add()
feature_city.name = "city"
feature_city.type = schema_pb2.FeatureType.BYTES
feature_city.string_domain.is_categorical = True
feature_city.string_domain.value.extend(["New York", "London", "Tokyo"])

print("Generated Schema (protobuf format):")
print(schema)

# Serialize the schema to bytes
serialized_schema = schema.SerializeToString()
print(f"\nSerialized Schema (bytes): {len(serialized_schema)} bytes")

# Deserialize the schema back from bytes
deserialized_schema = schema_pb2.Schema()
deserialized_schema.ParseFromString(serialized_schema)
print("\nDeserialized Schema:")
print(deserialized_schema)