msgspec

raw JSON →
0.20.0 verified Tue May 12 auth: no python install: verified quickstart: verified

msgspec is a fast serialization and validation library, with builtin support for JSON, MessagePack, YAML, and TOML. It features high-performance encoders/decoders, zero-cost schema validation using Python type annotations, and a speedy `Struct` type. The library is actively maintained with frequent releases.

pip install msgspec
error TypeError: Required field 'b' cannot follow optional fields. Either reorder the struct fields, or set `kw_only=True` in the struct definition.
cause Python function signature rules, which `msgspec.Struct` adheres to, require that all fields without a default value (required fields) must be defined before any fields with a default value (optional fields).
fix
Reorder the fields in your msgspec.Struct definition so that all required fields come before optional fields, or set kw_only=True in the Struct definition to make all fields keyword-only.
import msgspec

# Fix 1: Reorder fields
class ValidOrder(msgspec.Struct):
    a: str  # Required
    b: int = 0  # Optional

# Fix 2: Use kw_only=True
class KeywordOnly(msgspec.Struct, kw_only=True):
    a: str = "" # Optional, but position doesn't matter for kw_only
    b: int # Required
error msgspec.ValidationError: Expected `str`, got `int` - at `$.groups[0]`
cause This error occurs during decoding when the input data does not conform to the expected Python type annotations defined in the `msgspec.Struct` (or other specified type). In this example, an integer was found where a string was expected within a list.
fix
Ensure that the input data matches the type annotations specified in your msgspec.Struct or the type argument passed to the decoder. Inspect the input JSON/MessagePack and the msgspec.Struct definition to find the mismatch.
import msgspec

class User(msgspec.Struct):
    name: str
    groups: list[str] = msgspec.field(default_factory=list)

# Correct input: groups contains only strings
valid_data = b'{"name":"bob","groups":["devops"]}'
user = msgspec.json.decode(valid_data, type=User)
print(user)

# Original problematic input (assuming it contained an int where str was expected)
# invalid_data = b'{"name":"bob","groups":["devops", 123]}'
# try:
#     msgspec.json.decode(invalid_data, type=User)
# except msgspec.ValidationError as e:
#     print(e)
error TypeError: Encoding objects of type numpy.float64 is unsupported.
cause `msgspec` does not natively support encoding all arbitrary third-party types, such as NumPy data types, directly. It expects standard Python types or `msgspec.Struct` instances.
fix
Convert unsupported third-party types to a supported Python native type (e.g., float, int, list, dict) before encoding with msgspec. For custom types, you can also provide an enc_hook to the encoder.
import msgspec
import numpy as np

# Original problematic code:
# mjson.encode(np.float64(1.0))

# Fix 1: Convert to a native Python float
value_np = np.float64(1.0)
encoded_data = msgspec.json.encode(float(value_np))
print(encoded_data)

# Fix 2: Use an enc_hook for custom handling
def numpy_encoder_hook(obj):
    if isinstance(obj, np.ndarray):
        return obj.tolist()
    if isinstance(obj, np.generic):
        return obj.item()
    raise NotImplementedError

encoder = msgspec.json.Encoder(enc_hook=numpy_encoder_hook)
encoded_data_with_hook = encoder.encode(np.array([1.0, np.float64(2.0)]))
print(encoded_data_with_hook)
error AttributeError: immutable type: 'Point'
cause This error occurs when attempting to modify an attribute of a `msgspec.Struct` instance that was defined with `frozen=True`. Frozen structs are immutable after initialization, preventing any attribute changes.
fix
If you need to modify the object, either define the msgspec.Struct without frozen=True, or create a new instance with the desired changes.
import msgspec

class Point(msgspec.Struct, frozen=True):
    x: float
    y: float

p = Point(1.0, 2.0)

# Original problematic code:
# p.x = 2.0

# Fix: Create a new instance with updated values
p_new = Point(x=3.0, y=p.y) # Or use msgspec.structs.replace if you have many fields
print(p_new)

# If mutability is desired, define the struct without frozen=True
class MutablePoint(msgspec.Struct):
    x: float
    y: float
m_p = MutablePoint(1.0, 2.0)
m_p.x = 3.0
print(m_p)
error TypeError: Type unions may not contain more than one str-like type (`str`, `Enum`, ...)
cause `msgspec` enforces a restriction that type unions cannot contain multiple 'string-like' types (e.g., `str` and `bytes`) to avoid ambiguity, especially when handling different serialization formats like JSON (which has only strings) and MessagePack (which distinguishes strings and binary data).
fix
Refactor your type annotations to avoid unions with multiple string-like types. If you need to handle both str and bytes, consider using a single, unambiguous type or process bytes separately (e.g., base64 encode/decode if using JSON) or use msgspec.Raw for manual handling.
import msgspec
from typing import Union

# Original problematic code:
# class TestData(msgspec.Struct):
#     content: Union[str, bytes]

# Fix 1: Use a single, unambiguous type
class TestDataStr(msgspec.Struct):
    content: str

class TestDataBytes(msgspec.Struct):
    content: bytes

# Fix 2: If you must handle both, define separate structs and use a Tagged Union
# This allows msgspec to differentiate between them
class MyStrData(msgspec.Struct, tag='str_data'):
    value: str

class MyBytesData(msgspec.Struct, tag='bytes_data'):
    value: bytes

class Wrapper(msgspec.Struct):
    data: Union[MyStrData, MyBytesData]

encoder = msgspec.msgpack.Encoder()
decoder = msgspec.msgpack.Decoder(Wrapper)

wrapped_str = Wrapper(MyStrData('hello'))
encoded_str = encoder.encode(wrapped_str)
decoded_str = decoder.decode(encoded_str)
print(decoded_str)

wrapped_bytes = Wrapper(MyBytesData(b'world'))
encoded_bytes = encoder.encode(wrapped_bytes)
decoded_bytes = decoder.decode(encoded_bytes)
print(decoded_bytes)
breaking msgspec 0.19.0 dropped support for Python 3.8. Users on Python 3.8 or older must upgrade their Python version or stay on msgspec < 0.19.0.
fix Upgrade Python to 3.9+.
breaking In msgspec 0.19.0, the `encode_into` method (an advanced API) now expands the buffer if it's smaller than the offset, which is a breaking change for specific buffer management use cases.
fix Review usage of `Encoder.encode_into` and ensure buffer management aligns with the new behavior, potentially pre-allocating larger buffers or handling re-allocation.
deprecated The `from_builtins` method was removed in msgspec 0.19.0. Users should now use `msgspec.convert` for similar functionality.
fix Replace calls to `from_builtins` with `msgspec.convert`.
gotcha When using `msgspec.msgpack` to decode into `memoryview` objects, the original input message buffer will be kept in memory as long as the `memoryview` is alive. This can lead to unexpectedly high memory usage if not carefully managed, especially with large messages or long-lived `memoryview` instances.
fix Only use `memoryview` for zero-copy scenarios when strict performance is needed and memory lifecycle is precisely controlled. For most cases, prefer `bytes` or `bytearray` to ensure the input buffer can be garbage collected.
gotcha Mutable default values (e.g., `list`, `set`, `dict`) on `msgspec.Struct` fields will be shared across all instances if not defined using `msgspec.field(default_factory=...)`. This is a common Python pitfall.
fix Always use `msgspec.field(default_factory=my_callable)` for mutable default values, where `my_callable` is a zero-argument function that returns a new mutable object (e.g., `list`, `set`, `dict`).
gotcha The `msgspec.structs.fields()` function can be significantly slower (up to 20x) than `dataclasses.fields()` because it re-inspects type annotations on every invocation. Avoid frequent calls to `msgspec.structs.fields()` in performance-critical loops.
fix Cache the result of `msgspec.structs.fields()` if it's called repeatedly for the same `Struct` type, or redesign code to minimize its usage in hot paths.
breaking In msgspec 0.7.0, the `nogc` struct option was renamed to `gc`. To disable garbage collection for a Struct instance, you must now specify `gc=False` instead of `nogc=True`.
fix Update `Struct` definitions and instantiation calls from `nogc=True` to `gc=False`.
breaking Type hints using the `X | Y` syntax (e.g., `str | None`) for unions are only supported in Python 3.10 and later. When using `msgspec.Struct` on Python 3.9 or older, this syntax will raise a TypeError.
fix For Python 3.9 and older, replace type hints like `email: str | None` with `email: typing.Optional[str]` or `email: typing.Union[str, None]`. Remember to import `typing` if not already done.
gotcha Msgspec performs strict type validation by default. It does not automatically coerce values from one primitive type to another (e.g., a string '123' will not be converted to an integer 123) during decoding. Attempting to decode input data where a field's type does not strictly match its schema definition will raise a `ValidationError`.
fix Ensure input data types precisely match the `msgspec.Struct` field annotations. If type coercion is required, implement custom hooks or preprocess the data before decoding.
pip install msgspec[yaml,toml]
python os / libc variant status wheel install import disk
3.10 alpine (musl) msgspec wheel - 0.04s 18.4M
3.10 alpine (musl) yaml,toml wheel - 0.04s 20.7M
3.10 alpine (musl) msgspec - - 0.04s 18.4M
3.10 alpine (musl) yaml,toml - - 0.04s 20.7M
3.10 slim (glibc) msgspec wheel 1.6s 0.03s 19M
3.10 slim (glibc) yaml,toml wheel 1.8s 0.03s 22M
3.10 slim (glibc) msgspec - - 0.03s 19M
3.10 slim (glibc) yaml,toml - - 0.03s 22M
3.11 alpine (musl) msgspec wheel - 0.05s 20.3M
3.11 alpine (musl) yaml,toml wheel - 0.06s 22.7M
3.11 alpine (musl) msgspec - - 0.06s 20.3M
3.11 alpine (musl) yaml,toml - - 0.06s 22.7M
3.11 slim (glibc) msgspec wheel 1.9s 0.05s 21M
3.11 slim (glibc) yaml,toml wheel 1.8s 0.05s 24M
3.11 slim (glibc) msgspec - - 0.04s 21M
3.11 slim (glibc) yaml,toml - - 0.04s 24M
3.12 alpine (musl) msgspec wheel - 0.03s 12.2M
3.12 alpine (musl) yaml,toml wheel - 0.03s 14.6M
3.12 alpine (musl) msgspec - - 0.03s 12.2M
3.12 alpine (musl) yaml,toml - - 0.03s 14.6M
3.12 slim (glibc) msgspec wheel 1.7s 0.03s 13M
3.12 slim (glibc) yaml,toml wheel 1.7s 0.03s 16M
3.12 slim (glibc) msgspec - - 0.04s 13M
3.12 slim (glibc) yaml,toml - - 0.03s 16M
3.13 alpine (musl) msgspec wheel - 0.03s 11.9M
3.13 alpine (musl) yaml,toml wheel - 0.03s 14.4M
3.13 alpine (musl) msgspec - - 0.03s 11.8M
3.13 alpine (musl) yaml,toml - - 0.03s 14.2M
3.13 slim (glibc) msgspec wheel 1.5s 0.03s 12M
3.13 slim (glibc) yaml,toml wheel 1.7s 0.03s 16M
3.13 slim (glibc) msgspec - - 0.03s 12M
3.13 slim (glibc) yaml,toml - - 0.03s 15M
3.9 alpine (musl) msgspec wheel - 0.04s 17.9M
3.9 alpine (musl) yaml,toml wheel - 0.03s 20.2M
3.9 alpine (musl) msgspec - - 0.04s 17.9M
3.9 alpine (musl) yaml,toml - - 0.04s 20.2M
3.9 slim (glibc) msgspec wheel 1.9s 0.03s 18M
3.9 slim (glibc) yaml,toml wheel 2.2s 0.04s 21M
3.9 slim (glibc) msgspec - - 0.03s 18M
3.9 slim (glibc) yaml,toml - - 0.03s 21M

Defines a simple `User` struct, encodes it to JSON, and then decodes it back. It also demonstrates how `msgspec` handles type validation errors during decoding. Uses `msgspec.field(default_factory=set)` for mutable default values to prevent common Python footguns.

import msgspec

class User(msgspec.Struct):
    name: str
    age: int
    email: str | None = None
    groups: set[str] = msgspec.field(default_factory=set)

alice = User(name="alice", age=30, groups={"admin", "dev"})
print(f"Original User: {alice}")

# Encode to JSON
json_data = msgspec.json.encode(alice)
print(f"Encoded JSON: {json_data.decode()}")

# Decode from JSON
decoded_alice = msgspec.json.decode(json_data, type=User)
print(f"Decoded User: {decoded_alice}")

# Example of validation error
try:
    msgspec.json.decode(b'{"name":"bob","age":"25"}', type=User)
except msgspec.ValidationError as e:
    print(f"Validation Error: {e}")