CircularDict

1.9 · active · verified Sun Apr 12

CircularDict is a high-performance Python data structure (version 1.9) that combines the functionality of dictionaries and circular buffers. It allows defining constraints on size (number of items) and memory usage, automatically removing the oldest entries when limits are exceeded. This makes it ideal for caching large data structures while maintaining control over the memory footprint. The library appears to be actively maintained, with regular updates.

Warnings

gotcha If you attempt to add a single item (key-value pair) whose memory footprint alone exceeds the `maxsize_bytes` limit, a `MemoryError` will be raised. This means `maxsize_bytes` applies to individual items as well as the total.
Fix: Ensure that the `maxsize_bytes` parameter is large enough to accommodate the largest single item you intend to store. Consider using `sys.getsizeof()` to estimate object sizes, though actual dictionary overhead adds complexity.
gotcha While CircularDict inherits from Python's OrderedDict and maintains insertion order, its core functionality involves automatically removing the 'oldest' items when size or memory limits are hit. This modification can lead to a `RuntimeError` if you iterate over the dictionary while it is being modified by this automatic removal process.
Fix: Avoid iterating over a `CircularDict` simultaneously with operations that might trigger item removal (e.g., adding new items when at capacity). If iteration is necessary, consider iterating over a copy of the keys (`list(my_dict.keys())`) or values.
gotcha The `maxsize_bytes` parameter accounts for the total memory footprint, including both keys and values, and the internal overhead of the dictionary structure itself. Predicting exact memory usage can be challenging due to Python's object model and varying overheads, which might lead to unexpected removals if not carefully estimated.
Fix: Perform careful testing with representative data to determine appropriate `maxsize_bytes` values. Consider adding a buffer to your calculated `maxsize_bytes` to account for potential overheads or variations in object sizing.

Install

pip install circular-dict Install with pip

Imports

CircularDict
```
from circular_dict import CircularDict
```

Quickstart

Demonstrates initializing CircularDict with `maxlen` to limit item count and `maxsize_bytes` to limit memory. Shows how older items are automatically removed when limits are exceeded. Note that `maxsize_bytes` behavior is sensitive to the actual memory footprint of keys and values.

import os
from circular_dict import CircularDict

# Initialize a CircularDict with a maximum length of 3 items
my_dict_maxlen = CircularDict(maxlen=3)
my_dict_maxlen['key1'] = 'value1'
my_dict_maxlen['key2'] = 'value2'
my_dict_maxlen['key3'] = 'value3'
print(f"Initial maxlen dict: {list(my_dict_maxlen.keys())}")
my_dict_maxlen['key4'] = 'value4' # 'key1' is automatically removed
print(f"After adding key4 (key1 removed): {list(my_dict_maxlen.keys())}")

# Initialize a CircularDict with a maximum memory usage of 4MB
# Note: actual memory usage depends on content; this is an example.
# For demonstration, we'll use a smaller, illustrative byte size.
# Real-world usage requires careful calculation of object sizes.
# Using a small maxsize_bytes for demonstration of its behavior.
my_dict_maxsize = CircularDict(maxsize_bytes=100) # 100 bytes approx
my_dict_maxsize['a'] = '1234567890' # ~10 bytes for value + key overhead
my_dict_maxsize['b'] = 'abcdefghij' # ~10 bytes for value + key overhead
# Adding more items will cause older ones to be removed to stay under 100 bytes.
# This is illustrative; actual byte size calculations are complex.
print(f"\nInitial maxsize dict: {list(my_dict_maxsize.keys())}")
my_dict_maxsize['c'] = 'klmnopqrst' * 5 # A larger string
print(f"After adding larger key 'c': {list(my_dict_maxsize.keys())}")
# Depending on exact memory model, 'a' and 'b' might be removed.

view raw JSON →