PyMongo In-Memory

0.5.0 · active · verified Thu Apr 16

PyMongo In-Memory is a Python library that provides an ephemeral in-memory MongoDB server, primarily designed for testing and continuous integration environments. It achieves this by downloading and running a real `mongod` binary configured to use the in-memory storage engine. The library is actively maintained, with frequent updates that include support for newer MongoDB versions and bug fixes, typically released every few months.

Common errors

```
pymongo.errors.ServerSelectionTimeoutError: [Errno 111] Connection refused
```
cause The in-memory MongoDB daemon failed to start or was not accessible within the default timeout. Common causes include port conflicts, issues downloading the MongoDB binary, or incorrect OS/MongoDB version detection.

fix Check logs for `pymongo-inmemory` (set `PYMONGOIM__DEBUG=True` for verbose output). Try specifying a different `mongod_port` (e.g., `PYMONGOIM__MONGOD_PORT=27018`). Ensure your system has internet access for the initial binary download. If on Linux with a newer MongoDB version, confirm `operating_system` and `os_version` are explicitly set.
```
DBException in initAndListen, terminating, attr:{error:"Location28662: Cannot start server. Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, but the specified storage engine was 'inMemory'."}
```
cause You are attempting to start `pymongo-inmemory` with the in-memory storage engine in a directory that previously stored data using a persistent storage engine like WiredTiger.

fix Provide a new, empty directory for `mongod_data_folder` (e.g., `PYMONGOIM__MONGOD_DATA_FOLDER=/tmp/mongo_test_data`) or manually clear the contents of the conflicting `dbPath` before starting the server.
```
pymongo.errors.OperationFailure: The 'aggregate' command resulted in an OOM (out of memory) error :: caused by :: WiredTigerKVEngine::memory_walk (memory) exceeded its maximum memory limit
```
cause Even though `pymongo-inmemory` uses an in-memory storage engine, MongoDB itself has memory limits for certain operations (especially aggregations). Large datasets or complex queries can exceed these, leading to an Out-Of-Memory error.

fix Optimize your queries to reduce memory footprint, use `allowDiskUse=True` in aggregation pipelines if applicable (though this would bypass the 'in-memory' aspect), or increase the system's available memory. For `pymongo-inmemory` specifically, you might need to adjust the `inMemorySizeGB` if it were configurable at that level (it's often handled by MongoDB's defaults).
```
TypeError: expected str, bytes or os.PathLike object, not int on self._mongod.start()
```
cause This error can occur if a configuration parameter expecting a string path or similar object receives an integer, often related to port or folder configurations, especially in older versions or with specific Python environments.

fix Ensure all path-related or string-expected configuration values (e.g., `mongod_port` if directly passed, `download_folder`, `extract_folder`, `mongod_data_folder`) are passed as strings. For `mongod_port`, the library usually handles type coercion, but direct manipulation might bypass it. Review your configuration, especially if upgrading from older versions.

Warnings

breaking The default fallback for the storage engine changed in v0.5.0 for MongoDB versions greater than 6.0. If you were implicitly relying on previous fallback behavior for newer MongoDB versions, your tests might behave differently or require explicit `storage_engine` configuration.
Fix: Explicitly configure the `storage_engine` parameter in your `setup.cfg` or environment variables if you encounter unexpected behavior with MongoDB versions > 6.0.
gotcha For MongoDB versions greater than 4.0.23, generic Linux builds are not available from MongoDB. If you specify a newer `mongo_version` without an explicit `operating_system` (e.g., 'ubuntu', 'debian'), `pymongo-inmemory` will default to MongoDB 4.0.23, which might not be the version you intend to test against.
Fix: Always explicitly set `operating_system` (e.g., `PYMONGOIM__OPERATING_SYSTEM=ubuntu`) and `os_version` for MongoDB versions beyond 4.0.23 when running on Linux to ensure the correct binary is downloaded.
gotcha The in-memory storage engine does *not* persist data to disk. All data is lost when the `mongod` process shuts down. This is by design for testing, but crucial to understand if expecting any persistence.
Fix: Do not use `pymongo-inmemory` for any scenario where data persistence across process restarts is required. It is strictly for ephemeral testing.
gotcha If you try to initialize `pymongo-inmemory` in a `dbPath` that contains data files created by a different MongoDB storage engine (e.g., WiredTiger), `mongod` will fail to start with an error like 'Detected data files in ... created by the 'wiredTiger' storage engine, but the specified storage engine was 'inMemory''.
Fix: Ensure the `mongod_data_folder` is either a new, empty directory or one explicitly cleaned before starting the in-memory server. You can configure this via `PYMONGOIM__MONGOD_DATA_FOLDER` or `setup.cfg`.
gotcha The `pymongo-inmemory` library needs to download the appropriate MongoDB binary for your operating system and requested version. This requires an active internet connection on first use for a specific MongoD version/OS combination. Subsequent runs will use a cached binary.
Fix: Ensure network connectivity or pre-configure `use_local_mongod` if running in an isolated environment. The `download_folder` and `extract_folder` can also be customized.

Install

pip install pymongo-inmemory Install PyMongo In-Memory

Imports

MongoClient
wrong:
```
from pymongo import MongoClient
```
correct:
```
from pymongo_inmemory import MongoClient
```
While `pymongo.MongoClient` is the standard, `pymongo_inmemory.MongoClient` wraps it to automatically manage the in-memory server.
Mongod
```
from pymongo_inmemory.mongod import Mongod
```
For advanced control over the MongoDB daemon lifecycle, you can directly import and manage the `Mongod` class.

Quickstart

This quickstart demonstrates how to use `pymongo-inmemory.MongoClient` within a context manager. The server is automatically started upon entering the `with` block and stopped upon exiting. Basic CRUD operations are shown, connecting to a 'testdb' database and a 'test_collection'. You can specify the MongoDB version via environment variables or `setup.cfg`.

import os
from pymongo_inmemory import MongoClient

# Configure MongoDB version via environment variable for consistency
os.environ['PYMONGOIM__MONGO_VERSION'] = os.environ.get('PYMONGOIM__MONGO_VERSION', '6.0')

with MongoClient() as client:
    db = client['testdb']
    collection = db['test_collection']

    # Insert a document
    doc = {"name": "Alice", "age": 30}
    insert_result = collection.insert_one(doc)
    print(f"Inserted document with ID: {insert_result.inserted_id}")

    # Find a document
    found_doc = collection.find_one({"name": "Alice"})
    print(f"Found document: {found_doc}")

    # Update a document
    collection.update_one({"name": "Alice"}, {"$set": {"age": 31}})
    updated_doc = collection.find_one({"name": "Alice"})
    print(f"Updated document: {updated_doc}")

    # Delete a document
    collection.delete_one({"name": "Alice"})
    remaining_doc = collection.find_one({"name": "Alice"})
    print(f"Remaining document: {remaining_doc}")

print("In-memory MongoDB server automatically stopped.")

view raw JSON →