Apache Avro (Python 3)

Common errors

• ModuleNotFoundError: No module named 'avro.schema'

  cause This error occurs when the 'avro' package is installed instead of 'avro-python3' in a Python 3 environment.
  fix Install the correct package using 'pip install avro-python3'.

• AttributeError: module 'avro' has no attribute 'schema'

  cause This error arises when the 'avro' package, intended for Python 2, is used in a Python 3 environment.
  fix Uninstall 'avro' and install 'avro-python3' using 'pip uninstall avro' followed by 'pip install avro-python3'.

• ModuleNotFoundError: No module named 'pycodestyle'

  cause This error occurs due to a missing 'pycodestyle' dependency when installing 'avro-python3' version 1.9.2.
  fix Install 'pycodestyle' manually using 'pip install pycodestyle' before installing 'avro-python3'.

• NameError: name 'file' is not defined

  cause This error occurs when using the 'avro.tool' module's 'dump' command in Python 3, due to the use of 'file' instead of 'open'.
  fix Update to 'avro-python3' version 1.10.0 or later, where this issue is resolved.

• AvroTypeException: The datum is not an example of the schema

  cause This error occurs when attempting to serialize data containing unsupported types, such as 'date' objects, using 'avro-python3'.
  fix Convert 'date' objects to strings or integers before serialization, or switch to the 'avro' package which supports logical types.

Warnings

• gotcha Ensure you install `avro-python3` for Python 3 projects. The older `avro` package (without the `-python3` suffix) is largely unmaintained for Python 3 and may cause compatibility issues or unexpected behavior. This distinction is crucial for modern Python development.
  Fix: Always use `pip install avro-python3` and import directly from the `avro` namespace (e.g., `import avro.schema`).
• gotcha Misunderstanding Avro's schema evolution rules (e.g., adding/removing fields, changing types, default values) can lead to data deserialization errors, especially when different versions of producers and consumers interact.
  Fix: Familiarize yourself with the Avro specification's rules for schema evolution. Use `null` for new optional fields, ensure compatible type changes, and rigorously test schema updates across your ecosystem.
• gotcha Avro data files (`.avro` extension) are binary and not human-readable. Attempting to inspect them directly with a text editor will yield unintelligible, garbled output, which can be confusing for new users.
  Fix: To inspect Avro data, use the `avro-python3` library itself to read and deserialize the data, or utilize Avro-specific tools that can render the binary content into a human-readable format like JSON.
• gotcha Encountering a `ValueError: I/O operation on closed file.` indicates that an attempt was made to perform an I/O operation (such as `getvalue()`, `read()`, or `write()`) on a file-like object after it has already been closed. This is a common programming error related to resource management.
  Fix: Ensure that all necessary I/O operations are completed *before* closing a file-like object. For `io.BytesIO` objects, `getvalue()` can often be called directly without needing to close the stream first, or `close()` should be deferred until after all data extraction is complete. Review the sequence of operations on stream objects to confirm that `close()` is not invoked prematurely.
• gotcha Attempting to perform I/O operations (like `getvalue()` or `read()`) on a file-like object after its underlying stream has been closed will result in a `ValueError: I/O operation on closed file`.
  Fix: Ensure that any file-like objects (e.g., `io.BytesIO`, file handlers) are not closed prematurely before all necessary read/write operations, including final content retrieval (like `getvalue()`), have been completed. Review the code to identify where the I/O object might be prematurely closed or its underlying buffer invalidated.

Install

• pip install avro-python3 Install stable version

Imports

• avro.schema

  import avro.schema

  For parsing and working with Avro schemas.
• avro.io

  import avro.io

  For reading and writing Avro data using DatumReader/DatumWriter.
• avro.datafile

  import avro.datafile

  For working with Avro data files using DataFileReader/DataFileWriter.

Quickstart

This example demonstrates how to define an Avro schema, serialize Python dictionary data into the Avro binary format, and then deserialize it back into Python objects using an in-memory BytesIO stream to simulate file operations.

import avro.schema
import avro.io
import avro.datafile
import io

# 1. Define the Avro schema
schema_str = """
{
    "type": "record",
    "name": "User",
    "fields": [
        {"name": "name", "type": "string"},
        {"name": "favorite_number", "type": ["int", "null"]},
        {"name": "favorite_color", "type": ["string", "null"]}
    ]
}
"""
schema = avro.schema.parse(schema_str)

# 2. Write data to a BytesIO object (simulating a file)
writer = avro.io.DatumWriter(schema)
bytes_writer = io.BytesIO()
data_file_writer = avro.datafile.DataFileWriter(bytes_writer, writer, schema)

data_file_writer.append({"name": "Alyssa", "favorite_number": 256, "favorite_color": None})
data_file_writer.append({"name": "Ben", "favorite_number": 7, "favorite_color": "red"})
data_file_writer.close()

# Get the serialized data
avro_data = bytes_writer.getvalue()

# 3. Read data from the BytesIO object
bytes_reader = io.BytesIO(avro_data)
reader = avro.io.DatumReader(schema)
data_file_reader = avro.datafile.DataFileReader(bytes_reader, reader)

read_records = [record for record in data_file_reader]
data_file_reader.close()

# Print the read records
print(read_records)
# Expected output:
# [{'name': 'Alyssa', 'favorite_number': 256, 'favorite_color': None}, {'name': 'Ben', 'favorite_number': 7, 'favorite_color': 'red'}]