Zict: Composable Mutable Mappings

3.0.0 · active · verified Thu Apr 09

Zict provides a collection of abstract MutableMapping classes that can be composed to build complex storage systems. It allows developers to create custom dictionaries that manage data across different locations (in-memory, on disk), apply various eviction policies (like LRU), and handle data transformations (compression, serialization). This enables intuitive interfaces over sophisticated storage strategies. The library is currently at version 3.0.0 and appears to be actively maintained.

Warnings

gotcha When using `zict.Sieve` in a multithreaded environment, be aware that the `__getitem__` and `__setitem__` methods of the *underlying mappings* are not protected by locks. This means if multiple threads access the same underlying mapping concurrently via Sieve, race conditions could occur. Only `__contains__` and `__delitem__` methods of underlying mappings are protected if they are fast.
Fix: Implement external locking mechanisms or ensure thread-safe underlying mappings for concurrent `__getitem__` and `__setitem__` operations when using `zict.Sieve`.
gotcha Mappings that interface with system resources (e.g., `zict.File`, `zict.LMDB`) must have their `close()` method called to release those resources. Failure to do so can lead to resource leaks, file locks, or data corruption, especially in long-running applications or when reopening mappings.
Fix: Always call the `.close()` method on `zict` mappings that manage external resources when they are no longer needed. Consider using a `with` statement if the mapping supports the context manager protocol, or a `try...finally` block.
gotcha `zict.LMDB` requires the `lmdb` Python package to be installed separately to function correctly. It is not listed as a direct dependency of `zict` itself. Without it, attempting to use `zict.LMDB` will result in an `ImportError`.
Fix: If planning to use `zict.LMDB`, ensure `pip install lmdb` is run in your environment.

Install

pip install zict Install stable version

Imports

File
```
from zict import File
```
Func
```
from zict import Func
```
LRU
```
from zict import LRU
```
Buffer
```
from zict import Buffer
```
LMDB
```
from zict import LMDB
```
Requires `lmdb` to be installed separately for full functionality.
Sieve
```
from zict import Sieve
```

Quickstart

This example demonstrates how to compose `zict` mappings to create a multi-layered storage system. It sets up an LRU cache (`LRU`) backed by a function-transforming mapping (`Func`) for serialization and compression, which in turn writes to disk using a file-based mapping (`File`). This allows efficient in-memory caching with persistent, compressed storage.

import pickle
import zlib
import os
import shutil
from zict import File, Func, LRU

# Create a temporary directory for file storage
storage_dir = 'zict_example_storage'
if os.path.exists(storage_dir):
    shutil.rmtree(storage_dir)

# 1. File: A mapping that stores values as files in a directory
a = File(storage_dir, mode='a')

# 2. Func: A mapping that applies functions (e.g., compress/decompress, dumps/loads)
#    to values as they are set or retrieved from the underlying mapping.
b = Func(zlib.compress, zlib.decompress, a)
c = Func(pickle.dumps, pickle.loads, b)

# 3. LRU: A Least Recently Used (LRU) cache that wraps another mapping.
#    When the LRU overflows, items are evicted to the underlying mapping (c).
d = LRU(10, c) # Keep 10 items in the LRU cache

# Store and retrieve items
d['x'] = [1, 2, 3]
d['y'] = {'a': 1, 'b': 2}

print(f"Value for 'x': {d['x']}")
print(f"Value for 'y': {d['y']}")

# Populate more items to trigger LRU eviction
for i in range(15):
    d[str(i)] = i * 10

# Accessing 'x' brings it back to the fast cache (LRU)
print(f"Accessing 'x' again: {d['x']}")

# Clean up resources (important for File, LMDB, etc.)
d.close()
# Clean up the temporary storage directory
shutil.rmtree(storage_dir)

view raw JSON →