msgpack-numpy-opentensor

Common errors

• TypeError: buffer is too small for requested array

  cause This error often occurs when deserializing multi-dimensional NumPy arrays, potentially due to an incompatibility between `msgpack-numpy` (or `msgpack-numpy-opentensor`), `msgpack`, and `numpy` versions, or an internal buffer handling issue during array reconstruction.
  fix Ensure all related packages (`msgpack-numpy-opentensor`, `msgpack`, `numpy`) are updated to their latest compatible versions. If the problem persists, review the specific `numpy` version requirements for `msgpack-numpy` or consider reporting the issue with full version details.

• KeyError: b'nd' (or similar during unpackb with m.patch())

  cause When `m.patch()` is used, the global `object_hook` for `msgpack` is set. If `msgpack.unpackb` then encounters a dictionary that is *not* a serialized NumPy array (e.g., a regular Python dictionary), the `msgpack_numpy_opentensor` decoder might incorrectly attempt to interpret it as an array, leading to a `KeyError` because expected NumPy metadata keys (like `b'nd'`) are missing.
  fix Avoid using `m.patch()` for general `msgpack` usage. Instead, explicitly pass `default=m.encode` to `msgpack.packb` and `object_hook=m.decode` to `msgpack.unpackb` only when you expect NumPy arrays to be present in the data being serialized/deserialized. This provides more granular control and prevents unintended decoding attempts.

• TypeError: must be str, not bytes (or similar for datetime objects in arrays)

  cause `msgpack-numpy-opentensor` primarily focuses on numerical data types. Serializing NumPy arrays containing complex Python objects like `datetime` objects, especially if they have `dtype='O'` (object), can lead to type errors as `msgpack-numpy`'s default handlers may not know how to convert these specific objects.
  fix Convert `numpy.datetime64` arrays or arrays of Python `datetime` objects to a simpler, serializable format (e.g., integer timestamps, ISO 8601 strings) before packing. Alternatively, implement custom `default` encoder and `object_hook` decoder functions that specifically handle `datetime` objects within your NumPy array serialization pipeline.

Warnings

• breaking The upstream `opentensor/msgpack-numpy` GitHub repository, linked as this package's source, released `v1.0.0` with a breaking change: it disables `pickle` by default. This will prevent deserialization of NumPy arrays with `dtype='O'` (object arrays) that were serialized with pickle enabled in older versions. While `msgpack-numpy-opentensor` on PyPI is currently `0.5.0`, this change may propagate to future versions.
  Fix: Review `dtype='O'` usage and consider custom serializers for such arrays if you rely on pickle for compatibility. Be prepared to update serialization/deserialization logic if upgrading to a version with this change.
• gotcha NumPy arrays with `dtype='O'` (object arrays) are serialized/deserialized using Python's `pickle` module as a fallback by `msgpack-numpy` (and, by extension, likely `msgpack-numpy-opentensor`). This introduces significant performance overhead and poses security risks when deserializing data from untrusted sources due to pickle's arbitrary code execution capabilities.
  Fix: Avoid `dtype='O'` for sensitive or performance-critical data. Consider explicitly converting object arrays to more primitive types or implementing custom, secure encoders/decoders for specific object types.
• gotcha NumPy arrays deserialized by `msgpack-numpy` (and thus, `msgpack-numpy-opentensor`) are read-only by default. Attempting to modify them directly will raise a `ValueError` or `AttributeError`.
  Fix: If modification is required, explicitly create a writable copy of the array after deserialization, e.g., `modified_array = unpacked_array.copy()`.
• gotcha The underlying `msgpack` library has limitations on the maximum size of individual binary or string objects, typically around 4.3 GB. Attempting to serialize a single NumPy array that exceeds this limit may result in serialization errors.
  Fix: For extremely large NumPy arrays, consider chunking them into smaller pieces before serialization, using alternative serialization formats designed for larger-than-memory data, or streaming solutions.

Install

• pip install msgpack-numpy-opentensor PyPI Installation

Imports

• patch
  wrong:

  from msgpack_numpy_opentensor import patch

  correct:

  import msgpack
  import msgpack_numpy_opentensor as m
  m.patch()

  The `patch` function is typically called after importing both `msgpack` and the library, usually aliased for convenience. Direct import might not correctly set up monkey-patching for `msgpack`.
• encode
  wrong:

  from msgpack_numpy_opentensor import encode

  correct:

  import msgpack_numpy_opentensor as m
  msgpack.packb(data, default=m.encode)

  Similar to `patch`, `encode` and `decode` are usually accessed via the aliased module for consistency with `msgpack`'s `default` and `object_hook` parameters.
• decode
  wrong:

  from msgpack_numpy_opentensor import decode

  correct:

  import msgpack_numpy_opentensor as m
  msgpack.unpackb(packed_data, object_hook=m.decode)

  Similar to `patch`, `encode` and `decode` are usually accessed via the aliased module for consistency with `msgpack`'s `default` and `object_hook` parameters.

Quickstart

This quickstart demonstrates how to serialize a NumPy array into MessagePack binary format and then deserialize it back into a NumPy array using the `msgpack-numpy-opentensor` library. It shows both packing (`packb`) and unpacking (`unpackb`) using the provided encoder and decoder functions, highlighting that deserialized arrays are typically read-only.

import numpy as np
import msgpack
import msgpack_numpy_opentensor as m

# Create a NumPy array
x = np.random.rand(5, 5)

# Pack the NumPy array using msgpack-numpy-opentensor's encoder
# Optionally, you can call m.patch() to monkey-patch msgpack globally
# m.patch()
packed_x = msgpack.packb(x, default=m.encode)

# Unpack the bytes back into a NumPy array using the decoder
unpacked_x = msgpack.unpackb(packed_x, object_hook=m.decode, raw=False)

print("Original array:\n", x)
print("Unpacked array:\n", unpacked_x)
print("Arrays are equal:", np.array_equal(x, unpacked_x))
print("Unpacked array is read-only:", not unpacked_x.flags['WRITEABLE'])