{"id":360,"library":"msgpack","title":"MessagePack for Python","description":"MessagePack is an efficient binary serialization format, offering a faster and smaller alternative to JSON for data exchange across multiple languages. The Python library provides CPython bindings and a pure Python implementation for reading and writing MessagePack data. The current stable version is 1.1.2, actively maintained with regular releases.","status":"active","version":"1.1.2","language":"python","source_language":"en","source_url":"https://github.com/msgpack/msgpack-python","tags":["serialization","binary","messagepack","msgpack"],"install":[{"cmd":"pip install msgpack","lang":"bash","label":"Install latest stable version"}],"dependencies":[],"imports":[{"symbol":"msgpack","correct":"import msgpack"},{"note":"Used for handling MessagePack extension types.","symbol":"ExtType","correct":"from msgpack import ExtType"},{"note":"Used for handling MessagePack timestamp extension type.","symbol":"Timestamp","correct":"from msgpack import Timestamp"}],"quickstart":{"code":"import msgpack\nimport os\n\ndata_to_pack = {\n    'name': 'Alice',\n    'age': 30,\n    'is_student': True,\n    'grades': [95, 88, 72.5]\n}\n\n# Serialize data to bytes\npacked_data = msgpack.packb(data_to_pack)\nprint(f\"Packed data (bytes): {packed_data}\")\n\n# Deserialize data from bytes\nunpacked_data = msgpack.unpackb(packed_data, raw=False)\nprint(f\"Unpacked data: {unpacked_data}\")\n\n# Example with file I/O\nfile_path = os.path.join(os.path.dirname(__file__), 'example.msgpack')\nwith open(file_path, 'wb') as f:\n    msgpack.pack(data_to_pack, f)\n\nwith open(file_path, 'rb') as f:\n    loaded_data = msgpack.unpack(f, raw=False)\nprint(f\"Loaded data from file: {loaded_data}\")\n\nos.remove(file_path)","lang":"python","description":"Demonstrates basic one-shot serialization and deserialization using `packb` and `unpackb`, and also shows how to use `pack` and `unpack` with file-like objects for streaming."},"warnings":[{"fix":"Uninstall the old `msgpack-python` package: `pip uninstall msgpack-python`, then install the current package: `pip install msgpack`.","message":"The PyPI package name changed from `msgpack-python` to `msgpack` in version 0.5. When upgrading from older versions (0.4 or earlier), you must `pip uninstall msgpack-python` before running `pip install -U msgpack` to avoid conflicts and ensure the correct package is used.","severity":"breaking","affected_versions":"< 0.5"},{"fix":"Review your packing/unpacking calls: use `raw=True` for unpacking if raw bytes are expected or for old formats; use `use_bin_type=False` if you need to pack into the old 'raw' type. Adjust `max_buffer_size` or `strict_map_key=False` if dealing with large/unusual data or old formats.","message":"Version 1.0 introduced several breaking changes: dropped Python 2 support for the C extension (pure Python fallback is used), `Packer.use_bin_type` now defaults to `True` (bytes are encoded in bin type), the `encoding` option was removed (UTF-8 is always used), `Unpacker.raw` now defaults to `False` (decodes to Python `str`), and default `max_buffer_size` changed to 100MiB with `strict_map_key` defaulting to `True`.","severity":"breaking","affected_versions":"1.0.0 and later"},{"fix":"Explicitly set `use_bin_type=False` during packing to use the old 'raw' type, and `raw=True` during unpacking if you need Python `bytes` objects instead of decoded `str` objects, especially when dealing with non-UTF-8 compatible binary data or older MessagePack implementations.","message":"When packing or unpacking data containing binary strings or interacting with older MessagePack formats, the default settings for `use_bin_type` and `raw` might lead to unexpected behavior (e.g., `UnicodeDecodeError`). By default, `packb` uses `use_bin_type=True` for `bytes` (new spec 'bin' type), and `unpackb` uses `raw=False` to decode to `str`.","severity":"gotcha","affected_versions":"All versions (behavioral)"},{"fix":"Implement custom serialization logic by providing a `default` callable to `msgpack.packb()` (or `Packer`) to convert custom objects into MessagePack-supported types. For deserialization, use an `object_hook` or `ext_hook` callable with `msgpack.unpackb()` (or `Unpacker`) to reconstruct your custom objects.","message":"MessagePack does not automatically serialize complex custom Python objects (e.g., instances of user-defined classes). Attempting to pack such objects directly will result in a `TypeError`.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Ensure file opening modes are always specified as binary, e.g., `with open('data.msgpack', 'wb') as f:`.","message":"When performing file I/O with `msgpack.pack()` and `msgpack.unpack()`, always open files in binary mode (`'wb'` for writing, `'rb'` for reading). Using text mode (`'w'` or `'r'`) will corrupt the binary MessagePack data.","severity":"gotcha","affected_versions":"All versions"}],"env_vars":null,"last_verified":"2026-05-12T13:17:01.076Z","next_check":"2026-07-11T00:00:00.000Z","problems":[{"fix":"Provide a `default` function to `msgpack.packb()` that converts unsupported types into a MessagePack-serializable format, often by representing them as dictionaries or using `ExtType` for custom types.\n\n```python\nimport msgpack\nimport datetime\n\ndef default_serializer(obj):\n    if isinstance(obj, datetime.datetime):\n        return {'__datetime__': obj.isoformat()}\n    # Add other custom types here\n    raise TypeError(f\"Object of type {obj.__class__.__name__} is not JSON serializable\")\n\ndata = {'timestamp': datetime.datetime.now(), 'value': 123}\npacked_data = msgpack.packb(data, default=default_serializer)\nprint(packed_data)\n\n# To unpack, you'd need a corresponding object hook or ext_hook\n# For simplicity, here's a basic unpack for the example above:\ndef datetime_object_hook(obj):\n    if '__datetime__' in obj:\n        return datetime.datetime.fromisoformat(obj['__datetime__'])\n    return obj\n\nunpacked_data = msgpack.unpackb(packed_data, raw=False, object_hook=datetime_object_hook)\nprint(unpacked_data)\n```","cause":"msgpack does not inherently know how to serialize custom Python objects or certain complex built-in types (like datetime or set), requiring a custom serialization function.","error":"TypeError: can't serialize <class 'your_module.YourCustomClass'>"},{"fix":"When dealing with streams or concatenated MessagePack objects, use `msgpack.Unpacker` to process data incrementally, or ensure `unpackb()` receives only a single, complete MessagePack object.\n\n```python\nimport msgpack\n\n# Example of multiple objects packed together (causes ExtraData with unpackb)\npacked_data_stream = msgpack.packb({'a': 1}) + msgpack.packb({'b': 2})\n\n# Incorrect: will raise ExtraData\ntry:\n    obj = msgpack.unpackb(packed_data_stream)\nexcept msgpack.exceptions.ExtraData as e:\n    print(f\"Caught expected error: {e}\")\n\n# Correct: using Unpacker for stream processing\nunpacker = msgpack.Unpacker(raw=False) # raw=False for Python strings\nunpacker.feed(packed_data_stream)\nfor obj in unpacker:\n    print(f\"Unpacked object: {obj}\")\n```","cause":"This error occurs when `msgpack.unpackb()` is given a byte string that contains more than one complete MessagePack object, or unexpected trailing data after a single object.","error":"msgpack.exceptions.ExtraData: unpack(b) received extra data."},{"fix":"Ensure the input byte string is a complete and valid MessagePack object. When reading from a stream or network, confirm that all parts of the MessagePack message have been received before attempting to unpack. Use `msgpack.Unpacker`'s `feed` method and iterate over it to handle partial data gracefully.\n\n```python\nimport msgpack\n\n# Example of incomplete data\nincomplete_data = msgpack.packb({'key': 'value'})[:-5] # Truncate 5 bytes\n\n# Incorrect: will raise ValueError\ntry:\n    obj = msgpack.unpackb(incomplete_data)\nexcept ValueError as e:\n    print(f\"Caught expected error: {e}\")\n\n# Correct: When data might be incomplete, use Unpacker\nunpacker = msgpack.Unpacker(raw=False)\n# Simulate receiving data in chunks\nunpacker.feed(incomplete_data)\n\n# The loop won't yield anything until a complete object is fed\nfor obj in unpacker:\n    print(f\"Unpacked object: {obj}\")\n\n# If later, the rest of the data arrives\nremaining_data = msgpack.packb({'key': 'value'})[-5:] # The missing 5 bytes\nunpacker.feed(remaining_data)\nfor obj in unpacker:\n    print(f\"Successfully unpacked with remaining data: {obj}\")\n```","cause":"The input bytes provided to `msgpack.unpackb()` are truncated, corrupted, or do not contain enough data to form a complete and valid MessagePack object.","error":"ValueError: Unpack failed: incomplete input"},{"fix":"When packing, ensure `bytes` objects are treated as MessagePack's binary type by passing `use_bin_type=True` to `msgpack.packb()`. When unpacking, pass `raw=False` to `msgpack.unpackb()` or `msgpack.Unpacker` to automatically decode MessagePack string types to Python `str`, while preserving MessagePack binary types as Python `bytes`.\n\n```python\nimport msgpack\n\n# Scenario 1: Packing bytes without use_bin_type=True (problematic)\nbinary_data = b'\\x80\\x01\\x02\\x03' # Non-UTF-8 bytes\ntry:\n    # This might pack it as a raw string if use_bin_type is not explicit\n    packed_raw = msgpack.packb(binary_data, use_bin_type=False)\n    # Unpacking without raw=True (Python 3 default) will try to decode as string\n    # and fail if the packed 'raw' contains invalid UTF-8\n    # In msgpack 1.1.2, use_bin_type=True is often default/recommended.\n    # However, if explicitly set to False during packing and then unpacked without raw=True (default), it can cause this.\n    print(\"Attempting to unpack potentially problematic data...\")\n    msgpack.unpackb(packed_raw, raw=False)\nexcept UnicodeDecodeError as e:\n    print(f\"Caught expected UnicodeDecodeError: {e}\")\n\n# Correct way: Pack bytes as MessagePack's binary type\npacked_correctly = msgpack.packb(binary_data, use_bin_type=True)\n\n# Unpack with raw=False (default for Python strings) to get correct Python types\nunpacked_data = msgpack.unpackb(packed_correctly, raw=False)\nprint(f\"Unpacked correctly (bytes object): {unpacked_data}\")\n\n# If you specifically want raw bytes for strings as well, use raw=True during unpacking\npacked_str = msgpack.packb('hello world')\nunpacked_raw_str = msgpack.unpackb(packed_str, raw=True)\nprint(f\"Unpacked string as bytes: {unpacked_raw_str}\")\n```","cause":"This error occurs during unpacking when MessagePack attempts to decode a byte sequence as a UTF-8 string, but the sequence actually represents raw binary data or is not valid UTF-8. This often happens when `bytes` objects are packed without `use_bin_type=True` or unpacked with default settings that try to interpret everything as strings.","error":"UnicodeDecodeError: 'utf-8' codec can't decode byte 0x... in position ...: invalid start byte"},{"fix":"Provide a `default` function to `msgpack.packb` to convert unsupported types (e.g., `datetime`) into a serializable format like an ISO string or timestamp.\n\n```python\nimport msgpack\nimport datetime\n\ndef default_serializer(obj):\n    if isinstance(obj, datetime.datetime):\n        return obj.isoformat()\n    raise TypeError(f\"Object of type {obj.__class__.__name__} is not msgpack serializable\")\n\ndata = {'timestamp': datetime.datetime.now()}\npacked_data = msgpack.packb(data, default=default_serializer)\nprint(packed_data)\n```","cause":"`msgpack` does not natively serialize all Python types like `datetime` objects, requiring a custom serialization handler.","error":"TypeError: Cannot serialize 'datetime.datetime' object"}],"ecosystem":"pypi","meta_description":null,"install_score":100,"install_tag":"verified","quickstart_score":80,"quickstart_tag":"verified","pypi_latest":null,"install_checks":{"last_tested":"2026-05-12","tag":"verified","tag_description":"installs cleanly on critical runtimes, fast import, recently tested","results":[{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.01,"mem_mb":0.3,"disk_size":"18.8M"},{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.01,"mem_mb":0.3,"disk_size":"18.8M"},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0,"mem_mb":0.3,"disk_size":"20M"},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0,"mem_mb":0.3,"disk_size":"20M"},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.01,"mem_mb":0.4,"disk_size":"20.7M"},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.01,"mem_mb":0.4,"disk_size":"20.7M"},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.01,"mem_mb":0.4,"disk_size":"22M"},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.01,"mem_mb":0.4,"disk_size":"22M"},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0,"mem_mb":0.1,"disk_size":"12.6M"},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0,"mem_mb":0.1,"disk_size":"12.6M"},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0,"mem_mb":0.1,"disk_size":"13M"},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0,"mem_mb":0.1,"disk_size":"13M"},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0,"mem_mb":0.3,"disk_size":"12.2M"},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0,"mem_mb":0.3,"disk_size":"12.3M"},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0,"mem_mb":0.1,"disk_size":"13M"},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0,"mem_mb":0.1,"disk_size":"13M"},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.01,"mem_mb":0.3,"disk_size":"18.3M"},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0,"mem_mb":0.3,"disk_size":"18.3M"},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.01,"mem_mb":0.3,"disk_size":"19M"},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0,"mem_mb":0.3,"disk_size":"19M"}]},"quickstart_checks":{"last_tested":"2026-04-23","tag":"verified","tag_description":"quickstart runs on critical runtimes, recently tested","results":[{"runtime":"python:3.10-alpine","exit_code":0},{"runtime":"python:3.10-slim","exit_code":0},{"runtime":"python:3.11-alpine","exit_code":0},{"runtime":"python:3.11-slim","exit_code":0},{"runtime":"python:3.12-alpine","exit_code":0},{"runtime":"python:3.12-slim","exit_code":0},{"runtime":"python:3.13-alpine","exit_code":0},{"runtime":"python:3.13-slim","exit_code":0},{"runtime":"python:3.9-alpine","exit_code":0},{"runtime":"python:3.9-slim","exit_code":0}]}}