LMDB Python Binding

Warnings

• gotcha Keys and values in LMDB must be byte strings. Python strings (unicode) need to be explicitly encoded (e.g., `s.encode('utf-8')`) before being stored, and decoded (`b.decode('utf-8')`) after retrieval.
  Fix: Always `.encode()` Python strings to bytes before `put()` and `.decode()` bytes to strings after `get()` or cursor iteration.
• gotcha The `map_size` parameter in `lmdb.open()` sets the maximum size of the database. It is crucial to set this value sufficiently large upfront, especially if the database is opened by multiple processes. Modifying `map_size` from multiple processes concurrently can lead to catastrophic data loss.
  Fix: Estimate your maximum database size and set `map_size` generously when initially opening the environment. Ensure `map_size` is consistent if multiple processes access the same database. If resizing is necessary, ensure only one process is active.
• gotcha When using named databases (sub-databases), the `max_dbs` parameter must be set in `lmdb.open()` during the *first* opening of the environment by any process or thread. This allocates shared memory resources for the maximum number of named databases.
  Fix: Determine the maximum number of named databases you will use and specify `max_dbs` in `lmdb.open()` when the environment is created or first accessed.
• gotcha Write transactions (`env.begin(write=True)`) must be committed using `txn.commit()`. If not explicitly committed (e.g., if an exception occurs within the `with` block without proper handling), changes will be rolled back by default.
  Fix: Always use `with env.begin(write=True) as txn:` for write operations. The context manager handles `commit()` on success and `abort()` on exceptions automatically.
• breaking Version 1.4.1 and newer of `py-lmdb` dropped support for Python 2.x and older Python 3.x versions. Specifically, Python 3.9 or newer is now required.
  Fix: Ensure your project runs on Python 3.9 or a newer compatible version. For older Python environments, use `py-lmdb < 1.4.1` (e.g., `pip install 'lmdb<1.4.1'`).

Install

• pip install lmdb Install `lmdb`

Imports

• lmdb

  import lmdb

Quickstart

This quickstart demonstrates how to open an LMDB environment, write key-value pairs, read specific values, iterate through all entries using a cursor, and properly close the environment. Note the use of byte strings for keys and values, and the importance of `map_size` and `max_dbs` configuration. The example also includes basic cleanup.

import lmdb
import os
import shutil

db_path = './my_lmdb_data'

# Ensure cleanup for demonstration
if os.path.exists(db_path):
    shutil.rmtree(db_path)

# 1. Open an LMDB environment
# map_size: Maximum size of the database. Crucial to set correctly.
# max_dbs: Max number of named databases (sub-databases).
env = lmdb.open(db_path, map_size=10*1024*1024, max_dbs=10, subdir=True)

# 2. Write data to the database
with env.begin(write=True) as txn:
    txn.put(b'my_key_1', b'my_value_1')
    txn.put(b'my_key_2', b'my_value_2')
    txn.put(b'another_key', b'another_value')
    print("Wrote data to LMDB.")

# 3. Read data from the database
with env.begin() as txn:
    value1 = txn.get(b'my_key_1')
    value_non_existent = txn.get(b'non_existent_key')
    print(f"Value for my_key_1: {value1.decode('utf-8') if value1 else None}")
    print(f"Value for non_existent_key: {value_non_existent}")

    # Iterate through all key-value pairs
    print("\nAll items in DB:")
    for key, value in txn.cursor():
        print(f"  {key.decode('utf-8')}: {value.decode('utf-8')}")

# 4. Close the environment
env.close()
print("\nLMDB environment closed.")

# Cleanup (optional, for examples)
if os.path.exists(db_path):
    shutil.rmtree(db_path)
    print(f"Cleaned up database at {db_path}")