{"id":758,"library":"fastavro","title":"Fast Avro for Python","description":"Fastavro is a high-performance Python library for reading and writing Avro files. It provides a significantly faster alternative to the official Apache Avro Python library, leveraging C extensions (Cython) for optimal speed. The library supports various compression codecs and is actively maintained, making it a popular choice for high-throughput Avro serialization and deserialization in Python applications.","status":"active","version":"1.12.1","language":"python","source_language":"en","source_url":"https://github.com/fastavro/fastavro","tags":["avro","serialization","deserialization","performance","binary","data format"],"install":[{"cmd":"pip install fastavro","lang":"bash","label":"Install stable release"}],"dependencies":[{"reason":"Provides fast Snappy, Zstandard, and LZ4 compression/decompression. Recommended over `python-snappy` for Snappy.","package":"cramjam","optional":true},{"reason":"Provides Snappy compression/decompression. Deprecated in favor of `cramjam` for better compatibility and features.","package":"python-snappy","optional":true},{"reason":"Provides Zstandard compression/decompression.","package":"python-zstandard","optional":true},{"reason":"Provides LZ4 compression/decompression.","package":"python-lz4","optional":true},{"reason":"Provides Bzip2 compression/decompression.","package":"bz2file","optional":true},{"reason":"Provides XZ compression/decompression. Built-in to Python 3.3+.","package":"lzma","optional":true}],"imports":[{"symbol":"writer","correct":"from fastavro import writer"},{"symbol":"reader","correct":"from fastavro import reader"},{"note":"While `parse_schema` exists in `fastavro.schema`, it's idiomatic to import directly from `fastavro`.","wrong":"from fastavro.schema import parse_schema","symbol":"parse_schema","correct":"from fastavro import parse_schema"}],"quickstart":{"code":"import io\nfrom fastavro import writer, reader, parse_schema\n\n# 1. Define an Avro schema\nschema = {\n    'doc': 'A simple user record.',\n    'name': 'User',\n    'namespace': 'example.avro',\n    'type': 'record',\n    'fields': [\n        {'name': 'name', 'type': 'string'},\n        {'name': 'favorite_number', 'type': ['int', 'null'], 'default': None},\n        {'name': 'favorite_color', 'type': ['string', 'null'], 'default': 'green'}\n    ]\n}\n\n# It's optional but recommended to parse the schema once for performance\nparsed_schema = parse_schema(schema)\n\n# 2. Prepare some records\nrecords = [\n    {'name': 'Alice', 'favorite_number': 256, 'favorite_color': 'blue'},\n    {'name': 'Bob', 'favorite_number': 7, 'favorite_color': None},\n    {'name': 'Charlie', 'favorite_number': None, 'favorite_color': 'red'}\n]\n\n# 3. Write records to an in-memory Avro file (BytesIO)\nbytes_writer = io.BytesIO()\nwriter(bytes_writer, parsed_schema, records, codec='deflate')\n\n# 4. Read records back from the in-memory Avro file\nbytes_writer.seek(0) # Rewind the buffer to the beginning\navro_reader = reader(bytes_writer)\n\nread_records = []\nfor record in avro_reader:\n    read_records.append(record)\n\nprint(\"Original Records:\", records)\nprint(\"Read Records:\", read_records)\n\n# Verify that read records match original records\nassert records == read_records\nprint(\"Successfully wrote and read Avro records!\")","lang":"python","description":"This quickstart demonstrates how to define an Avro schema, write a list of Python dictionaries (records) into an in-memory Avro binary format using `fastavro.writer`, and then read those records back using `fastavro.reader`. It also highlights the use of `parse_schema` for efficiency and `codec` for compression."},"warnings":[{"fix":"Do not rely on a global cache for schemas. Pass parsed schemas explicitly or re-parse them as needed. `parse_schema` returns a parsed schema object that should be passed to `writer` or `reader` functions.","message":"The global cache of parsed schemas was removed in version 0.24.0. Code that relied on manipulating or accessing this global cache via `parse_schema` would have broken.","severity":"breaking","affected_versions":">=0.24.0"},{"fix":"Install `cramjam` (`pip install cramjam`) and ensure it's available in your environment for Snappy compression.","message":"Using `python-snappy` for Snappy compression is deprecated. `fastavro` recommends `cramjam` for Snappy, Zstandard, and LZ4 compression due to better compatibility and features.","severity":"deprecated","affected_versions":">=1.9.0"},{"fix":"Always ensure your reader schema is compatible with the writer schema, especially when dealing with schema evolution. Use `parse_schema` and provide both `writer_schema` (from the file) and `reader_schema` (your application's expected schema) to `fastavro.reader` for schema resolution.","message":"Reading Avro data with an incompatible reader schema (e.g., missing fields, mismatched types) can lead to `SchemaResolutionError` or incorrect data during deserialization.","severity":"gotcha","affected_versions":"All"},{"fix":"Ensure all required fields are present in your Python dictionary records before passing them to `fastavro.writer`. For optional fields, explicitly use `null` if the field is omitted, or define a default value in the schema.","message":"When serializing data, if a field is marked as 'required' in the Avro schema but is missing from the Python dictionary record, `fastavro` will raise an error.","severity":"gotcha","affected_versions":"All"},{"fix":"Open the file with `mode='a+b'` and call `writer(file_object, None, more_records)`.","message":"When appending records to an existing Avro file, the file must be opened in `a+b` mode (read and append binary). Passing `None` as the schema to the `writer` function is recommended, as the existing file's schema will be reused. Using `ab` mode or providing a schema will likely lead to errors.","severity":"gotcha","affected_versions":"All"},{"fix":"Avoid using `expand=True` if you intend to reuse the parsed schema for reading or writing. Instead, manage referenced schemas via the `named_schemas` argument in `parse_schema` if you have complex, inter-dependent schemas.","message":"Using `parse_schema(..., expand=True)` generates a schema that may not fully conform to the Avro specification for all scenarios, especially when dealing with referenced schemas. The output of this function with `expand=True` should generally be considered for output/inspection only and not passed directly to `reader` or `writer` functions, as it might cause exceptions.","severity":"gotcha","affected_versions":"All"},{"fix":"Be aware of the return type when `return_record_name` is set. Adjust your code to unpack the `(name, value)` tuple or use `return_record_name_override=True` if you prefer a simpler return for single-type unions.","message":"When reading a union of records, if `return_record_name=True` is specified in `reader()`, the result for a union type will be a tuple `(record_name, record_value)`. If a union contains only one record type, `return_record_name_override=True` can modify this behavior to return just the record value, without the name tuple.","severity":"gotcha","affected_versions":"All"}],"env_vars":null,"last_verified":"2026-05-12T18:39:58.615Z","next_check":"2026-06-27T00:00:00.000Z","problems":[{"fix":"Run `pip install fastavro` in your terminal to install the library.","cause":"The `fastavro` library has not been installed in your current Python environment.","error":"ModuleNotFoundError: No module named 'fastavro'"},{"fix":"Ensure that your data strictly adheres to the provided Avro schema. When reading, verify that the reader's schema is compatible with the writer's schema, paying close attention to field names, types, and the use of aliases or union types. Validate your schema structure, especially for enums or complex types.","cause":"This error occurs when the data being written or read does not conform to the Avro schema, or when there's an incompatibility between the writer's schema and the reader's schema, often related to missing required fields, mismatched types, or issues with aliases/named schemas.","error":"fastavro.exceptions.SchemaResolutionError"},{"fix":"Ensure that the records you are passing to `fastavro.writer` are lists of dictionaries, and that each dictionary's values precisely match the types defined in your Avro schema for the corresponding fields. For example, if your schema defines a record, you must pass a Python dictionary for that record, not a string or other type.","cause":"This typically happens when attempting to write data that is not in the expected dictionary format, or when a field within a record has an incorrect data type (e.g., passing a string where an integer is expected) according to the Avro schema.","error":"TypeError: Expected dict, got str"},{"fix":"Install the required compression library. For Snappy, run `pip install python-snappy` (or `pip install cramjam` for `fastavro` versions that use it for Snappy). Similarly, for Zstandard, install `pip install python-zstandard` (or `pip install backports.zstd` for older Python versions).","cause":"You are attempting to read or write an Avro file compressed with the 'snappy' codec, but the necessary `python-snappy` library (or `cramjam` for newer `fastavro` versions) is not installed in your environment.","error":"ValueError: snappy codec is supported but you need to install python-snappy"}],"ecosystem":"pypi","meta_description":null,"install_score":100,"install_tag":"verified","quickstart_score":80,"quickstart_tag":"verified","pypi_latest":"1.12.2","cli_name":null,"install_checks":{"last_tested":"2026-05-12","tag":"verified","tag_description":"installs cleanly on critical runtimes, fast import, recently tested","results":[{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":" $EXIT -eq 0 ","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":null,"import_time_s":0.07,"mem_mb":2.3,"disk_size":"25.9M"},{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.07,"mem_mb":2.3,"disk_size":"26.1M"},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":" $EXIT -eq 0 ","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":1.8,"import_time_s":0.05,"mem_mb":2.3,"disk_size":"30M"},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.05,"mem_mb":2.3,"disk_size":"30M"},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":" $EXIT -eq 0 ","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":null,"import_time_s":0.08,"mem_mb":2.2,"disk_size":"28.3M"},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.09,"mem_mb":2.2,"disk_size":"28.5M"},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":" $EXIT -eq 0 ","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":1.9,"import_time_s":0.08,"mem_mb":2.2,"disk_size":"32M"},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.07,"mem_mb":2.2,"disk_size":"32M"},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":" $EXIT -eq 0 ","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":null,"import_time_s":0.06,"mem_mb":1.9,"disk_size":"20.3M"},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.11,"mem_mb":1.9,"disk_size":"20.5M"},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":" $EXIT -eq 0 ","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":1.7,"import_time_s":0.06,"mem_mb":1.9,"disk_size":"24M"},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.06,"mem_mb":1.9,"disk_size":"24M"},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":" $EXIT -eq 0 ","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":null,"import_time_s":0.05,"mem_mb":1.9,"disk_size":"19.9M"},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.05,"mem_mb":1.9,"disk_size":"20.0M"},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":" $EXIT -eq 0 ","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":1.6,"import_time_s":0.05,"mem_mb":1.7,"disk_size":"24M"},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.05,"mem_mb":1.7,"disk_size":"24M"},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":" $EXIT -eq 0 ","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":null,"import_time_s":0.07,"mem_mb":2.3,"disk_size":"25.3M"},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.06,"mem_mb":2.3,"disk_size":"25.4M"},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":" $EXIT -eq 0 ","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":2.2,"import_time_s":0.06,"mem_mb":2.3,"disk_size":"29M"},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.06,"mem_mb":2.3,"disk_size":"29M"}]},"quickstart_checks":{"last_tested":"2026-04-24","tag":"verified","tag_description":"quickstart runs on critical runtimes, recently tested","results":[{"runtime":"python:3.10-alpine","exit_code":0},{"runtime":"python:3.10-slim","exit_code":0},{"runtime":"python:3.11-alpine","exit_code":0},{"runtime":"python:3.11-slim","exit_code":0},{"runtime":"python:3.12-alpine","exit_code":0},{"runtime":"python:3.12-slim","exit_code":0},{"runtime":"python:3.13-alpine","exit_code":0},{"runtime":"python:3.13-slim","exit_code":0},{"runtime":"python:3.9-alpine","exit_code":0},{"runtime":"python:3.9-slim","exit_code":0}]}}