Protocol Buffers (protobuf) for Python

Warnings

• breaking Python major version bumped to 7 with the 7.34.0 release (previous line was 6.x). Boolean values are now rejected when setting enum or int fields—the API raises a TypeError instead of implicitly converting them. The deprecated float_precision option in json_format and float_format/double_format in text_format were also removed.
  Fix: Upgrade to >=7.34.1. Replace boolean literals with int values when assigning to int/enum fields (e.g. use 1 instead of True). Remove any use of float_precision= in json_format.MessageToJson() and float_format=/double_format= in text_format calls.
• breaking Gencode/runtime version mismatch raises google.protobuf.runtime_version.VersionError at import time. Generated _pb2.py files embed a minimum runtime version; loading them against an older installed protobuf package fails hard. This is especially common when grpcio-tools generates code with a newer bundled protoc than the protobuf runtime you have installed.
  Fix: Always regenerate _pb2.py files with the same protoc version as your installed runtime. Pin grpcio-tools and protobuf together in requirements. Generated code for major version V works only with runtime major versions V and V+1.
• breaking Python 4.21.0 (2022) switched the C extension to the upb library. Sharing message objects between Python and C++ (e.g. via SWIG or pybind11) stopped working by default. Libraries like older TensorFlow that relied on this crash with AttributeError on import.
  Fix: Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp and install the Python/C++ extension if you need Python↔C++ message sharing, or upgrade dependent libraries (e.g. TF >=2.12) that have been updated to use upb.
• breaking message.UnknownFields() was deprecated in v5.25 and removed in v6.26+. Calling it raises AttributeError.
  Fix: Replace msg.UnknownFields() with: from google.protobuf import unknown_fields; unknown_fields.UnknownFieldSet(msg)
• gotcha Accessing an undefined key in a proto map field creates that key with a zero/false/empty value (defaultdict-like behaviour). This silently mutates the message during read-only access, which can cause unexpected serialization differences and test failures.
  Fix: Use 'key in msg.map_field' to check existence before accessing. Use msg.map_field.get(key) where available, or msg.map_field.GetOrCreate(key) for message-typed maps.
• gotcha The Python package name declared in a .proto file does NOT affect generated Python module names or import paths. Python packages are determined purely by directory structure relative to the --proto_path flag. Hyphens in filenames are silently converted to underscores (foo-bar.proto → foo_bar_pb2.py).
  Fix: Pass --proto_path rooted at the common ancestor of all .proto files. Ensure generated _pb2.py files are on PYTHONPATH. Well-known google/* protos must always be imported as 'google/protobuf/...' in .proto files.
• gotcha Do not subclass generated message classes. They use a metaclass and internal descriptor machinery that makes subclassing produce subtle bugs ('fragile base class' problems). The official docs explicitly warn against it.
  Fix: Compose rather than inherit: hold a proto message as a field, or use helper functions. For custom serialization logic, wrap the message in a plain Python class.

Install

• pip install protobuf Runtime library only (most common)
• pip install grpcio-tools Includes protoc compiler plugin for gRPC projects
• # Install the standalone protoc compiler (platform-specific) # macOS: brew install protobuf # Ubuntu: apt-get install -y protobuf-compiler # Or download from https://github.com/protocolbuffers/protobuf/releases Install protoc compiler (required to generate _pb2.py files)

Imports

• MessageToJson / ParseDict

  from google.protobuf import json_format
  json_format.MessageToJson(msg)
  json_format.ParseDict(d, MyMessage())

  Always import via google.protobuf package namespace. Direct attribute access on a bare import can fail if the google namespace package is split.
• MessageToString / Parse (text format)

  from google.protobuf import text_format
  text_format.MessageToString(msg)
  text_format.Parse(text, MyMessage())

  text_format lives under google.protobuf, not at the top level of the protobuf package.
• descriptor_pool / DescriptorPool

  from google.protobuf import descriptor_pool
  pool = descriptor_pool.Default()

  Used for dynamic message creation and reflection; pool is a singleton—do not construct new ones unnecessarily.
• Generated message class (user proto)

  # After: protoc --python_out=. my_message.proto
  from my_message_pb2 import MyMessage
  msg = MyMessage(field1='hello', field2=42)

  Generated files always have the _pb2.py suffix. The proto package declaration does NOT affect Python module names—only directory structure matters.
• Well-known types (Timestamp, Duration, Any, Struct …)

  from google.protobuf.timestamp_pb2 import Timestamp
  from google.protobuf.any_pb2 import Any

  Well-known types ship with the runtime under google.protobuf.*_pb2. They must be imported as google.protobuf.* in .proto files too (e.g. import 'google/protobuf/timestamp.proto').
• UnknownFieldSet

  from google.protobuf import unknown_fields
  unk = unknown_fields.UnknownFieldSet(msg)

  message.UnknownFields() was deprecated in v5.25 and removed in v6.26+. Use the unknown_fields module instead.

Quickstart

Serialize and deserialize a message using a well-known type (no custom .proto compilation required). Demonstrates SerializeToString / ParseFromString and JSON round-trip.

# pip install protobuf
# No custom .proto needed for this example — uses the built-in Timestamp well-known type.

from google.protobuf.timestamp_pb2 import Timestamp
from google.protobuf import json_format
import time

# --- Create and populate a message ---
ts = Timestamp()
ts.GetCurrentTime()          # sets seconds + nanos to now

# --- Binary serialization round-trip ---
binary = ts.SerializeToString()
ts2 = Timestamp()
ts2.ParseFromString(binary)  # returns number of bytes consumed
assert ts == ts2, "Round-trip failed"

# --- JSON serialization ---
json_str = json_format.MessageToJson(ts)
print("JSON:", json_str)

ts3 = json_format.Parse(json_str, Timestamp())
assert ts == ts3, "JSON round-trip failed"
print("All assertions passed.")

# --- Typical workflow with a custom proto ---
# 1. Write my_message.proto:
#      syntax = "proto3";
#      message Person { string name = 1; int32 id = 2; }
# 2. Compile:
#      protoc --python_out=. my_message.proto
# 3. Use generated code:
#      from my_message_pb2 import Person
#      p = Person(name='Alice', id=42)
#      data = p.SerializeToString()
#      p2 = Person()
#      p2.ParseFromString(data)