Hatch Protobuf Plugin

0.5.0 · active · verified Thu Apr 16

Hatch-protobuf is a Hatch build plugin designed to automatically generate Python files from Protocol Buffers (.proto) definitions. It leverages `grpcio-tools` internally to invoke the `protoc` compiler, streamlining the integration of Protobuf schema compilation into the standard Python build process. As of version 0.5.0, it is actively maintained, though its release cadence is tied to the broader Hatch and Hatchling ecosystem.

Common errors

```
RuntimeError: ERROR: ['grpc_tools.protoc', '--proto_path=...', '--python_out=...', '...']
```
cause The `protoc` command invoked by `grpc_tools.protoc.main` failed. This often indicates issues with file paths, syntax errors in `.proto` files, or `grpcio-tools`/`protobuf` not being correctly installed or accessible.

fix Ensure `grpcio-tools` and `protobuf` are correctly specified in your build hook dependencies with compatible versions. Double-check `proto-paths` and `output-path` in `pyproject.toml` for correctness. Manually run `protoc` on your `.proto` files with similar arguments to debug syntax issues.
```
ModuleNotFoundError: No module named 'my_package.protos.my_message_pb2'
```
cause The Python files generated by `hatch-protobuf` are not being placed in a location discoverable by Python, or the Python package structure is incorrect, or the project is not installed in development mode.

fix Verify that `output-path` in `pyproject.toml` points to a valid location within your Python package structure (e.g., `src/my_package/protos`). For development, ensure your project is installed in editable mode (e.g., `hatch env run pip install -e .` from your project root) so the generated files are symlinked or copied into the editable package location.
```
TypeError: environment_type_plugin_class() got an unexpected keyword argument 'keep_env'
```
cause This error or similar `TypeError` can occur when using an older version of `hatch-protobuf` with a newer version of `Hatch` due to changes in Hatch's environment plugin interface.

fix Update `hatch-protobuf` to its latest compatible version, as changes in Hatch (e.g., Hatch 1.16.5 fixed a breaking change in virtualenv, and older Hatch versions might drop Python 3.9 support). Ensure `hatch-protobuf`'s version constraint allows for the latest patch releases.

Warnings

breaking Hatch 1.16.0 introduced changes in how build hook dependencies are handled, which may cause `hatch build` to fail if `hatch-protobuf` is listed only within `[tool.hatch.build.hooks.protobuf].dependencies`.
Fix: Ensure `hatch-protobuf` itself is listed in `build-system.requires` in your `pyproject.toml`, as shown in the quickstart example.
gotcha The plugin is designed to import `.proto` definitions from non-editable (regular `pip install`) dependencies. Importing from editable dependencies (e.g., `pip install -e` or `uv workspaces`) might not work due to different directory layouts that `protoc` cannot correctly resolve.
Fix: For `.proto` files provided by other Python packages, ensure those packages are installed normally (not in editable mode). For local `.proto` files, manage them within your project's `proto-paths`.
gotcha Care must be taken when configuring `proto-paths` and `import_site_packages`. Setting `import_site_packages = true` causes `protoc` to add your site-packages to its `--proto_path`, potentially leading to re-generation of Python files for existing system-wide Protobuf definitions (like `googleapis-common-protos`), which can conflict with pre-built versions and stomp over your `site-packages` directory.
Fix: Only use `import_site_packages = true` if you explicitly intend to generate code that might conflict with or override existing `site-packages` protos. For most cases, explicitly define `proto-paths` for your project's `.proto` files.
gotcha Newer versions of the `protoc` compiler (used internally) enforce stricter limits on the length of symbol names (e.g., field names). Very long names in your `.proto` definitions can lead to compilation errors.
Fix: Review and shorten excessively long symbol names in your `.proto` files if you encounter errors related to name length limits.

Install

pip install hatch-protobuf Install only the plugin
pip install hatch-protobuf grpcio-tools protobuf Install plugin and common Protobuf dependencies

Imports

GeneratedProtobufModule
```
from my_package.protos import my_message_pb2
```
Hatch-protobuf is a build hook; end-users import the Python modules generated from .proto files, typically suffixed with '_pb2.py' and located according to your project's `output-path` configuration.

Quickstart

To quickly use `hatch-protobuf`, first configure your `pyproject.toml` file to include the plugin and define the paths for your `.proto` files and where the generated Python modules should be placed. Ensure `grpcio-tools` and `protobuf` are listed as dependencies for the build hook. Then, place your `.proto` definitions in the specified `proto-paths`. After running `hatch build` or setting up your development environment (e.g., `hatch env run pip install -e .`), the generated Python modules can be imported and used like any other Protobuf-generated code.

# pyproject.toml
[build-system]
requires = ["hatchling>=1.12.0", "hatch-protobuf~=0.5.0"]
build-backend = "hatchling.build"

[tool.hatch.build.hooks.protobuf]
proto-paths = ["src/my_package/protos"] # Directory containing your .proto files
output-path = "src/my_package/protos" # Where to place generated Python files
dependencies = [
    "grpcio-tools~=1.48", # Specific versions often needed for compatibility
    "protobuf>=3.19"
]

# src/my_package/protos/example.proto
syntax = "proto3";

package my_package.protos;

message MyMessage {
  string name = 1;
  int32 id = 2;
}

# src/my_package/main.py (after running hatch build or in dev environment)
from my_package.protos import example_pb2

message = example_pb2.MyMessage(name="Alice", id=123)
print(f"Created Protobuf message: {message.name}, {message.id}")

view raw JSON →