Avro Record Class and Specific Record Reader Generator

Warnings

• gotcha avro-gen3 generates specific record classes as `DictWrapper` instances and does NOT provide an overloaded `DictWriter`. This means that generated specific records, while offering type-hinted access, behave like regular Python dictionaries for serialization purposes with standard Avro `DatumWriter`.
  Fix: Be aware that direct dictionary writing or standard `DatumWriter` usage will not enforce schema during the write operation beyond what the underlying `apache-avro` library provides. The primary benefit of `avro-gen3` is compile-time type checking and IDE support via generated classes.
• breaking Breaking change in `apache-avro` versions 1.10 and later moved `AvroTypeException` to a different package, which can cause `AttributeError: module 'avro.io' has no attribute 'AvroTypeException'` if `avro-gen3` generated code (or its dependencies) expects the old location. This often manifests when custom properties contain non-string values.
  Fix: Ensure `avro-gen3` is updated to a version compatible with your `apache-avro` library (e.g., `avro-gen3==0.7.16` or newer). Pin your `apache-avro` dependency to a compatible version if necessary (e.g., `<1.10` or `>=1.10` based on `avro-gen3`'s constraints). Regenerate classes if the error persists after updating.
• gotcha When defining optional fields in Avro schemas, the `type` must be a union with `"null"` as the *first* type, and a `default` value must be specified as the literal `null` (not the string `"null"`). Incorrectly formatted optional fields can lead to consumer-side exceptions even if the schema appears valid for encoding.
  Fix: Always declare optional fields like `"fields": [{"name": "optional_field", "type": ["null", "string"], "default": null}]`.
• gotcha Generated Avro classes are organized into submodules reflecting their Avro namespaces within the output directory. Importing them requires correctly constructing the Python import path based on the Avro namespace and the chosen output directory.
  Fix: If your Avro schema has `"namespace": "com.example.app"` and you generate into `my_generated_code/`, you must import with `from my_generated_code.com.example.app import MyRecord`.

Install

• pip install avro-gen3 Install stable version

Imports

• write_schema_files

  from avrogen import write_schema_files

  Used to generate Python classes from Avro schemas.
• GeneratedRecordClass

  from <output_directory>.<avro_namespace_path> import <RecordName>

  Generated record classes are created in a user-specified output directory, organized by Avro namespace, and must be imported from that dynamic path, not directly from 'avrogen'.
• SpecificDatumReader

  from <output_directory> import SpecificDatumReader

  The generated SpecificDatumReader is created in the root of the output directory for specific record deserialization.

Quickstart

This quickstart demonstrates how to define an Avro schema, use `avro-gen3` to generate Python classes for it, and then serialize/deserialize data using these generated classes. It highlights the dynamic import of generated classes based on the schema's namespace and the use of the generated SCHEMA object with standard Avro I/O tools.

import os
import sys
import tempfile
from pathlib import Path
from avrogen import write_schema_files

# 1. Define a simple Avro schema
avro_schema_json = '''
{
  "type": "record",
  "name": "User",
  "namespace": "com.example.app",
  "fields": [
    {"name": "name", "type": "string"},
    {"name": "favorite_number", "type": ["int", "null"], "default": null}
  ]
}
'''

# 2. Define an output directory for generated classes
with tempfile.TemporaryDirectory() as tmpdir_name:
    output_dir = Path(tmpdir_name)
    print(f"Generated Avro classes will be written to: {output_dir}")

    # 3. Generate Python classes from the Avro schema
    write_schema_files(avro_schema_json, output_dir)

    # Add the output directory to sys.path to enable import
    sys.path.insert(0, str(output_dir))

    try:
        # 4. Import the generated classes and reader
        # The Avro namespace 'com.example.app' translates to a path within the output_dir
        from com.example.app import User  # Access the generated User class
        from avro.io import DatumWriter, DatumReader
        from avro.datafile import DataFileWriter, DataFileReader

        # 5. Create an instance of the generated class
        user_record = User(name="Alice", favorite_number=123)
        print(f"Created user record: {user_record}")
        print(f"User name: {user_record.name}, Favorite number: {user_record.favorite_number}")

        # 6. Serialize and deserialize using standard Avro tools with the generated schema/classes
        # Note: avro-gen3 wraps DatumReader but for DataFileWriter/Reader, you still use avro's types
        # For simpler examples, we might use the original avro library's DatumWriter/Reader directly
        # The main benefit of avro-gen3 is the type-hinted classes.

        # The generated classes are DictWrapper instances, compatible with standard Avro I/O
        output_file = output_dir / "users.avro"
        writer = DataFileWriter(open(output_file, "wb"), DatumWriter(), user_record.SCHEMA)
        writer.append(user_record._inner_dict) # avro-gen3 records are dict wrappers
        writer.close()

        reader = DataFileReader(open(output_file, "rb"), DatumReader())
        for read_user_dict in reader:
            # When reading back, DatumReader returns dicts. You'd re-wrap if desired.
            read_user = User(**read_user_dict)
            print(f"Deserialized user: {read_user.name}, {read_user.favorite_number}")
        reader.close()

    finally:
        # Clean up sys.path
        sys.path.remove(str(output_dir))