Apache Avro Python

Warnings

• deprecated The `avro-python3` PyPI package is deprecated. Users should now install and use the `avro` package, which supports both Python 2 (legacy) and Python 3. The `avro-python3` package will be removed in the near future.
  Fix: Ensure you are installing `avro` (i.e., `pip install avro`). If migrating from `avro-python3`, be aware of minor API differences, such as function capitalization (e.g., `avro.schema.parse` vs `avro.schema.Parse`).
• gotcha Installing the `avro` package (intended for Python 3+) in older Python 2 environments or incorrectly expecting Python 2 behavior in Python 3 can lead to `SyntaxError` due to incompatible syntax (e.g., `except Exception, e:`).
  Fix: Always use a Python 3 environment (>=3.9 as per package requirements) and ensure `pip install avro` is performed. The `avro` package has consolidated Python 3 support.
• gotcha The official Python Avro library is implemented in pure Python, which can lead to slow performance when processing large volumes of data or complex schemas. This is a common pain point for users.
  Fix: For performance-critical applications, consider using alternative libraries like `fastavro` (available on PyPI: `pip install fastavro`), which uses C extensions for significantly improved speed. `fastavro` provides a similar API but may not support Avro RPC.
• gotcha When reading Avro files, the reader's schema must be compatible with the writer's schema, adhering to Avro's schema evolution rules. Mismatched or missing fields (especially required ones) between reader and writer schemas can lead to errors or unexpected data during deserialization.
  Fix: Carefully manage schema evolution. Ensure that reader schemas are forward-compatible with writer schemas. For robust applications, always validate your data against the expected schema and handle potential schema resolution errors.

Install

• pip install avro Install stable release
• pip install avro[snappy,zstandard] Install with optional compression codecs

Imports

• avro.schema.parse

  import avro.schema
  schema = avro.schema.parse(json_schema_string)

  Prior to Avro 1.10.x, or if using the now deprecated 'avro-python3' package, 'Parse' (capital 'P') was used. For the current 'avro' package (Python 3 compatible), use 'parse' (lowercase 'p').
• DataFileReader

  from avro.datafile import DataFileReader

• DataFileWriter

  from avro.datafile import DataFileWriter

• DatumReader

  from avro.io import DatumReader

• DatumWriter

  from avro.io import DatumWriter

Quickstart

This quickstart demonstrates how to define an Avro schema, serialize Python dictionaries (records) into an Avro data file (here, in-memory using `io.BytesIO`), and then deserialize them back into Python dictionaries. It uses `avro.schema.parse` to load the schema, `DataFileWriter` and `DatumWriter` to write, and `DataFileReader` and `DatumReader` to read.

import avro.schema
from avro.datafile import DataFileReader, DataFileWriter
from avro.io import DatumReader, DatumWriter
import io

# Define schema
schema_str = '''
{
    "type": "record",
    "name": "User",
    "fields": [
        {"name": "name", "type": "string"},
        {"name": "favorite_number", "type": ["int", "null"]},
        {"name": "favorite_color", "type": ["string", "null"]}
    ]
}
'''
schema = avro.schema.parse(schema_str)

# Prepare data
users = [
    {"name": "Alyssa", "favorite_number": 256, "favorite_color": "red"},
    {"name": "Ben", "favorite_number": 7, "favorite_color": "blue"},
    {"name": "Charlie", "favorite_number": null, "favorite_color": "green"},
    {"name": "David", "favorite_number": 42, "favorite_color": null}
]

# Write data to an in-memory Avro file
# Using io.BytesIO for an in-memory file-like object
output_stream = io.BytesIO()
writer = DataFileWriter(output_stream, DatumWriter(), schema)
for user in users:
    writer.append(user)
writer.close()

# Reset stream position to read from the beginning
output_stream.seek(0)

# Read data from the in-memory Avro file
reader = DataFileReader(output_stream, DatumReader())
print("Reading Avro data:")
for user in reader:
    print(user)
reader.close()

output_stream.close()