{"id":7092,"library":"cmreshandler","title":"CMRESHandler","description":"CMRESHandler is a Python library that provides an Elasticsearch logging appender compatible with the standard `logging` library. It enables logging application events to Elasticsearch in JSON format, supporting various authentication types (Basic, Kerberos, AWS Signed) and flexible index naming frequencies (daily, weekly, monthly, yearly). The library is currently at version 1.0.0, released in 2017. Although functional, the project appears to be unmaintained, with a community fork existing due to its lack of active development.","status":"abandoned","version":"1.0.0","language":"en","source_language":"en","source_url":"https://github.com/cmanaha/python-elasticsearch-logger","tags":["logging","elasticsearch","handler","log management"],"install":[{"cmd":"pip install CMRESHandler","lang":"bash","label":"Install stable release"}],"dependencies":[{"reason":"Core dependency for connecting to Elasticsearch.","package":"elasticsearch","optional":false},{"reason":"Used for HTTP communication with Elasticsearch.","package":"requests","optional":false},{"reason":"Required for Python 2; built-in for Python 3.","package":"enum","optional":true},{"reason":"Optional dependency for Kerberos authentication.","package":"requests-kerberos","optional":true},{"reason":"Optional dependency for AWS IAM user authentication.","package":"requests-aws4auth","optional":true}],"imports":[{"note":"The PyPI package name is CMRESHandler, but the import path uses the cmreslogging namespace.","wrong":"from CMRESHandler import CMRESHandler","symbol":"CMRESHandler","correct":"from cmreslogging.handlers import CMRESHandler"},{"note":"AuthType is an inner enum class of CMRESHandler.","symbol":"AuthType","correct":"CMRESHandler.AuthType"},{"note":"IndexNameFrequency is an inner enum class of CMRESHandler.","symbol":"IndexNameFrequency","correct":"CMRESHandler.IndexNameFrequency"}],"quickstart":{"code":"import logging\nimport os\nfrom cmreslogging.handlers import CMRESHandler\n\n# Configure basic logging\nlog = logging.getLogger(\"PythonTest\")\nlog.setLevel(logging.INFO)\n\n# Elasticsearch host configuration\nES_HOST = os.environ.get('ES_HOST', 'localhost')\nES_PORT = int(os.environ.get('ES_PORT', 9200))\nES_INDEX_NAME = os.environ.get('ES_INDEX_NAME', 'python_app_logs')\n\n# Optional: Basic Auth details\nES_USER = os.environ.get('ES_USER')\nES_PASSWORD = os.environ.get('ES_PASSWORD')\n\nhosts = [{'host': ES_HOST, 'port': ES_PORT}]\nauth_type = CMRESHandler.AuthType.NO_AUTH\nauth_details = None\n\nif ES_USER and ES_PASSWORD:\n    auth_type = CMRESHandler.AuthType.BASIC_AUTH\n    auth_details = (ES_USER, ES_PASSWORD)\n\ntry:\n    handler = CMRESHandler(\n        hosts=hosts,\n        auth_type=auth_type,\n        auth_details=auth_details,\n        es_index_name=ES_INDEX_NAME,\n        es_additional_fields={'App': 'MyAppName', 'Environment': 'Dev'},\n        # Ensure flush frequency is reasonable for testing\n        flush_frequency_in_sec=1\n    )\n    log.addHandler(handler)\n\n    log.info(\"This is an info statement that will be logged into Elasticsearch.\")\n    log.warning(\"A warning message from the application.\")\n    try:\n        1 / 0\n    except ZeroDivisionError:\n        log.exception(\"An exception occurred, should be logged with traceback.\")\n\n    print(\"Log messages sent to Elasticsearch (check your ES_INDEX_NAME in Elasticsearch).\")\n    # In a real application, you might need a small delay or explicit flush\n    # if the application exits immediately after logging.\n    # For this quickstart, we rely on flush_frequency_in_sec.\n\nexcept Exception as e:\n    print(f\"Failed to initialize CMRESHandler or send logs: {e}\")\n    print(\"Ensure Elasticsearch is running and accessible at {ES_HOST}:{ES_PORT}.\")\n","lang":"python","description":"This quickstart demonstrates how to configure and use CMRESHandler with basic authentication (if environment variables are set) to send log messages to an Elasticsearch instance. Logs will include additional custom fields and exceptions will be sent with full tracebacks. The `flush_frequency_in_sec` is set to 1 for quick testing, but should be adjusted for production."},"warnings":[{"fix":"There is no direct fix within CMRESHandler as the project is unmaintained. Consider migrating to a maintained Elasticsearch logging library or forking and modifying the handler's serialization logic to remove the deprecated `_type` field.","message":"Older Elasticsearch versions used `_type` in bulk requests. This is deprecated in newer Elasticsearch versions (e.g., 6.x and above) in favor of a single `_doc` type or removing types entirely. This handler might send deprecated `_type` fields, leading to warnings or errors in modern Elasticsearch clusters.","severity":"breaking","affected_versions":"<=1.0.0 with Elasticsearch >=6.x"},{"fix":"Set `flush_frequency_in_sec` to a low value (e.g., 1 second) during development or for applications with short lifespans. For critical logs, consider explicitly calling `handler.flush()` before application exit, though this might not be thread-safe depending on implementation details.","message":"Logs might not appear immediately in Elasticsearch because CMRESHandler buffers messages and flushes them periodically or when the buffer is full. If an application exits before a flush, buffered logs may be lost.","severity":"gotcha","affected_versions":"All versions"},{"fix":"No direct fix for users. This requires a change in the library's source code to set the `daemon` attribute directly during thread initialization. As the project is unmaintained, consider using a different logging handler or forking the library.","message":"The `threading.Thread.setDaemon` method, potentially used internally by CMRESHandler for its buffer flushing thread, has been deprecated in Python 3.10. Running the library on Python 3.10+ might produce deprecation warnings.","severity":"deprecated","affected_versions":"All versions with Python >=3.10"},{"fix":"Downgrade your `elasticsearch` client library to a version compatible with `cmreshandler` (e.g., `pip install elasticsearch<7.0.0`) or update the `cmreslogging/handlers.py` source code to reflect the current import path (e.g., `from elasticsearch.connection import RequestsHttpConnection`).","message":"The `RequestsHttpConnection` import path within the `elasticsearch` client library has changed across major versions. Older versions of CMRESHandler might fail to import if the installed `elasticsearch` client library is too new.","severity":"breaking","affected_versions":"All versions with `elasticsearch` client library >= 7.x"}],"env_vars":null,"last_verified":"2026-04-16T00:00:00.000Z","next_check":"2026-07-15T00:00:00.000Z","problems":[{"fix":"Ensure your system's hostname resolves correctly, or try setting your hostname to `localhost` (e.g., `sudo hostname localhost` on Linux/macOS). For Docker environments, ensure network settings allow internal resolution.","cause":"This error often occurs in local development environments, particularly when the hostname cannot be resolved or is incorrectly configured, preventing the internal timer thread from being properly initialized.","error":"'CMRESHandler' object has no attribute '_timer'"},{"fix":"Install an older, compatible version of the `elasticsearch` client library: `pip install elasticsearch<7.0.0` or manually modify the `cmreslogging/handlers.py` file to correct the import path.","cause":"The `elasticsearch` Python client library changed the location of `RequestsHttpConnection` in newer versions (e.g., 7.x and above). CMRESHandler was designed for older versions of the client.","error":"ImportError: cannot import name 'RequestsHttpConnection' from 'elasticsearch'"},{"fix":"For testing, set `flush_frequency_in_sec` to a small value (e.g., 1). In production, ensure the application runs long enough for flushes to occur, or consider explicitly calling `handler.flush()` before exiting to ensure all buffered logs are sent.","cause":"CMRESHandler buffers log messages and flushes them to Elasticsearch at a set frequency or when the buffer is full. If your application exits quickly or logs infrequently, messages might remain in the buffer.","error":"Logs are not appearing in Elasticsearch, and there are no errors."},{"fix":"Inspect your log messages and Elasticsearch index mappings. Ensure that fields being logged (especially dynamic ones from `extra` or `es_additional_fields`) consistently have the same data type. If necessary, explicitly define a flexible mapping in Elasticsearch for such fields (e.g., using `object` type or disabling `dynamic` mapping for specific fields), or rename conflicting fields in your application.","cause":"This Elasticsearch error indicates that you are attempting to index data where a field (e.g., 'args') has been previously mapped as one type (e.g., `long` for numbers) but is now being sent with a different type (e.g., `text` for strings). This commonly happens with dynamic fields when different log messages use the same field name for different data types.","error":"IllegalArgumentException: mapper [args] of different type, current_type [long], merged_type [text]"}]}