{"id":4638,"library":"msgpack-numpy","title":"Msgpack-NumPy","description":"This package provides encoding and decoding routines that enable the serialization and deserialization of numerical and array data types from NumPy using the highly efficient MessagePack format. It also supports serialization of Python's native complex data types. The current version is 0.4.8, and it maintains compatibility with Python versions 2.7 and 3.5+.","status":"active","version":"0.4.8","language":"en","source_language":"en","source_url":"https://github.com/lebedov/msgpack-numpy","tags":["serialization","numpy","msgpack","data interchange","binary","data science"],"install":[{"cmd":"pip install msgpack-numpy","lang":"bash","label":"Install stable version"}],"dependencies":[{"reason":"Core serialization library.","package":"msgpack"},{"reason":"Provides the array and numerical types to be serialized.","package":"numpy"}],"imports":[{"note":"Common alias for convenience.","symbol":"msgpack_numpy","correct":"import msgpack_numpy as m"},{"note":"Applies NumPy-aware encoding/decoding globally to msgpack functions.","symbol":"patch","correct":"m.patch()"},{"note":"Manual encoder for NumPy types.","symbol":"encode","correct":"msgpack_numpy.encode"},{"note":"Manual decoder for NumPy types.","symbol":"decode","correct":"msgpack_numpy.decode"}],"quickstart":{"code":"import msgpack\nimport msgpack_numpy as m\nimport numpy as np\n\n# Easiest way: Monkey-patch msgpack to be numpy-aware\nm.patch()\n\n# Example NumPy array\ndata = {'array': np.array([1.2, 3.4, 5.6], dtype=np.float32), 'scalar': np.float64(7.8)}\n\n# Serialize\npacked_data = msgpack.packb(data, use_bin_type=True)\nprint(f\"Packed size: {len(packed_data)} bytes\")\n\n# Deserialize\nunpacked_data = msgpack.unpackb(packed_data, raw=False)\n\n# Verify\nprint(f\"Original array type: {type(data['array'])}, dtype: {data['array'].dtype}\")\nprint(f\"Unpacked array type: {type(unpacked_data[b'array'])}, dtype: {unpacked_data[b'array'].dtype}\")\nprint(f\"Original scalar type: {type(data['scalar'])}, value: {data['scalar']}\")\nprint(f\"Unpacked scalar type: {type(unpacked_data[b'scalar'])}, value: {unpacked_data[b'scalar']}\")\n\n# Ensure unpacked arrays are modifiable if needed (they are read-only by default)\noriginal_array = unpacked_data[b'array'].copy()\noriginal_array[0] = 99.9\nprint(f\"Modified array: {original_array}\")","lang":"python","description":"The quickest way to use msgpack-numpy is to call `m.patch()` after importing `msgpack` and `msgpack_numpy`. This automatically configures `msgpack.packb` and `msgpack.unpackb` to handle NumPy data types. It's crucial to use `use_bin_type=True` during packing for binary data and `raw=False` (default) for unpacking strings."},"warnings":[{"fix":"Ensure `msgpack-python` is uninstalled before installing or upgrading `msgpack`.","message":"When upgrading the underlying `msgpack` library from versions `0.4` or earlier to `0.5` or later, the package name on PyPI changed from `msgpack-python` to `msgpack`. Users must `pip uninstall msgpack-python` before `pip install -U msgpack` to prevent conflicts. This directly affects `msgpack-numpy` installations.","severity":"breaking","affected_versions":"msgpack < 0.5 transitioning to >= 0.5"},{"fix":"If modifications are needed, create a writable copy using `.copy()`: `my_array = unpacked_data[b'my_array'].copy()`.","message":"NumPy arrays deserialized by `msgpack-numpy` are read-only views of the underlying data buffer to optimize memory usage. Attempting to modify them directly will result in an error.","severity":"gotcha","affected_versions":"All versions"},{"fix":"If type preservation is not critical and absolute minimal size is, consider implementing a custom encoder/decoder for specific use cases or alternative serialization formats that explicitly sacrifice type information for size.","message":"The primary design goal of `msgpack-numpy` is the preservation of numerical data types, which inherently adds some storage overhead to the serialized data (e.g., storing dtype, shape, kind).","severity":"gotcha","affected_versions":"All versions"},{"fix":"For extremely large arrays (e.g., 10GB+), consider alternative serialization methods like `pickle`, HDF5, Zarr, or breaking the data into smaller chunks if `msgpack` is strictly required.","message":"`msgpack` (and by extension `msgpack-numpy`) has limits on object sizes. For instance, the maximum length of a binary object is `(2^32)-1` bytes (approximately 4GB). Attempting to serialize NumPy arrays significantly larger than this limit can fail.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Avoid using NumPy object arrays with `msgpack-numpy` if possible. If necessary, consider a custom encoder/decoder to handle the specific objects efficiently, or be aware of the `pickle` overhead and security implications.","message":"NumPy arrays with `dtype='O'` (object type) are serialized/deserialized using Python's `pickle` module as a fallback within `msgpack-numpy`. This negates the efficiency benefits of `msgpack` and can introduce security risks if deserializing untrusted data.","severity":"gotcha","affected_versions":"All versions"}],"env_vars":null,"last_verified":"2026-04-12T00:00:00.000Z","next_check":"2026-07-11T00:00:00.000Z"}