Python Varint Implementation

1.0.2 · maintenance · verified Thu Apr 16

The `varint` library provides a simple Python implementation for variable-length encoding and decoding of integers. This encoding scheme is primarily useful for creating more compact representations, particularly for sequences or arrays of frequently small integers, to save memory. While Python's native integer objects have overhead, varint encoding can be advantageous when storing many small numbers in a bytearray.

Common errors

```
ImportError: No module named 'varint'
```
cause The `varint` library has not been installed or is not accessible in the current Python environment.

fix Install the library using pip: `pip install varint`
```
TypeError: argument must be a bytes-like object, not 'int' or TypeError: expected a bytes-like object, not int
```
cause Attempting to pass an integer directly to `decode_bytes` or `decode_stream` instead of a bytes object or a stream.

fix Ensure that you are passing a `bytes` object (e.g., `b'\xc2\x02'`) to `decode_bytes` or a `BytesIO` stream to `decode_stream`, not the raw integer value.
```
EOFError: Read from empty stream
```
cause The `decode_stream` function attempted to read beyond the end of the provided stream, indicating incomplete or malformed varint data.

fix Verify that the byte stream contains valid and complete varint-encoded data. This often happens if the stream is consumed or positioned incorrectly before decoding.

Warnings

gotcha Using varint encoding for individual integers in Python can *increase* memory usage due to Python's object and `bytearray` overhead, rather than reducing it. The memory benefit is primarily realized when encoding and storing *arrays or lists* of many small integers.
Fix: Evaluate actual memory usage for your specific use case, especially for single integer storage, and consider the processing overhead incurred by encoding/decoding. This library is best for sequences of small integers.
deprecated The `varint` library has not seen updates since its last release on February 24, 2016. While functional, this means it may not incorporate newer Python features, performance optimizations, or receive bug fixes.
Fix: For new projects, consider more actively maintained alternatives if specific performance or modern Python feature support is critical. For existing projects, be aware of the lack of ongoing development.
gotcha There is a processing overhead associated with encoding and decoding integers using varint. This library is not a drop-in replacement for native integer storage if raw performance and direct random access are paramount.
Fix: Benchmark and profile your application to ensure the memory savings (if any) outweigh the computational cost of encoding and decoding, particularly for high-throughput scenarios.

Install

pip install varint Install latest version

Imports

encode
```
from varint import encode
```
For encoding a single integer into varint bytes.
decode_bytes
```
from varint import decode_bytes
```
For decoding varint bytes directly from a bytes-like object.
decode_stream
```
from varint import decode_stream
```
For decoding varint bytes from a file-like object (stream).

Quickstart

This example demonstrates how to encode an integer into variable-length bytes and then decode it back using both `decode_bytes` for a direct bytes object and `decode_stream` for a file-like object (BytesIO).

from varint import encode, decode_bytes, decode_stream
from io import BytesIO

# Encode an integer
number_to_encode = 300
encoded_bytes = encode(number_to_encode)
print(f"Encoded {number_to_encode}: {encoded_bytes.hex()}")

# Decode from bytes
decoded_number_from_bytes = decode_bytes(encoded_bytes)
print(f"Decoded from bytes: {decoded_number_from_bytes}")

# Decode from a stream
stream = BytesIO(encoded_bytes)
decoded_number_from_stream = decode_stream(stream)
print(f"Decoded from stream: {decoded_number_from_stream}")

# Example with a larger number
large_number = 123456789
encoded_large = encode(large_number)
print(f"\nEncoded {large_number}: {encoded_large.hex()}")
decoded_large = decode_bytes(encoded_large)
print(f"Decoded large number: {decoded_large}")

view raw JSON →