CMRESHandler

Common errors

• 'CMRESHandler' object has no attribute '_timer'

  cause This error often occurs in local development environments, particularly when the hostname cannot be resolved or is incorrectly configured, preventing the internal timer thread from being properly initialized.
  fix Ensure your system's hostname resolves correctly, or try setting your hostname to `localhost` (e.g., `sudo hostname localhost` on Linux/macOS). For Docker environments, ensure network settings allow internal resolution.

• ImportError: cannot import name 'RequestsHttpConnection' from 'elasticsearch'

  cause The `elasticsearch` Python client library changed the location of `RequestsHttpConnection` in newer versions (e.g., 7.x and above). CMRESHandler was designed for older versions of the client.
  fix Install an older, compatible version of the `elasticsearch` client library: `pip install elasticsearch<7.0.0` or manually modify the `cmreslogging/handlers.py` file to correct the import path.

• Logs are not appearing in Elasticsearch, and there are no errors.

  cause CMRESHandler buffers log messages and flushes them to Elasticsearch at a set frequency or when the buffer is full. If your application exits quickly or logs infrequently, messages might remain in the buffer.
  fix For testing, set `flush_frequency_in_sec` to a small value (e.g., 1). In production, ensure the application runs long enough for flushes to occur, or consider explicitly calling `handler.flush()` before exiting to ensure all buffered logs are sent.

• IllegalArgumentException: mapper [args] of different type, current_type [long], merged_type [text]

  cause This Elasticsearch error indicates that you are attempting to index data where a field (e.g., 'args') has been previously mapped as one type (e.g., `long` for numbers) but is now being sent with a different type (e.g., `text` for strings). This commonly happens with dynamic fields when different log messages use the same field name for different data types.
  fix Inspect your log messages and Elasticsearch index mappings. Ensure that fields being logged (especially dynamic ones from `extra` or `es_additional_fields`) consistently have the same data type. If necessary, explicitly define a flexible mapping in Elasticsearch for such fields (e.g., using `object` type or disabling `dynamic` mapping for specific fields), or rename conflicting fields in your application.

Warnings

• breaking Older Elasticsearch versions used `_type` in bulk requests. This is deprecated in newer Elasticsearch versions (e.g., 6.x and above) in favor of a single `_doc` type or removing types entirely. This handler might send deprecated `_type` fields, leading to warnings or errors in modern Elasticsearch clusters.
  Fix: There is no direct fix within CMRESHandler as the project is unmaintained. Consider migrating to a maintained Elasticsearch logging library or forking and modifying the handler's serialization logic to remove the deprecated `_type` field.
• gotcha Logs might not appear immediately in Elasticsearch because CMRESHandler buffers messages and flushes them periodically or when the buffer is full. If an application exits before a flush, buffered logs may be lost.
  Fix: Set `flush_frequency_in_sec` to a low value (e.g., 1 second) during development or for applications with short lifespans. For critical logs, consider explicitly calling `handler.flush()` before application exit, though this might not be thread-safe depending on implementation details.
• deprecated The `threading.Thread.setDaemon` method, potentially used internally by CMRESHandler for its buffer flushing thread, has been deprecated in Python 3.10. Running the library on Python 3.10+ might produce deprecation warnings.
  Fix: No direct fix for users. This requires a change in the library's source code to set the `daemon` attribute directly during thread initialization. As the project is unmaintained, consider using a different logging handler or forking the library.
• breaking The `RequestsHttpConnection` import path within the `elasticsearch` client library has changed across major versions. Older versions of CMRESHandler might fail to import if the installed `elasticsearch` client library is too new.
  Fix: Downgrade your `elasticsearch` client library to a version compatible with `cmreshandler` (e.g., `pip install elasticsearch<7.0.0`) or update the `cmreslogging/handlers.py` source code to reflect the current import path (e.g., `from elasticsearch.connection import RequestsHttpConnection`).

Install

• pip install CMRESHandler Install stable release

Imports

• CMRESHandler
  wrong:

  from CMRESHandler import CMRESHandler

  correct:

  from cmreslogging.handlers import CMRESHandler

  The PyPI package name is CMRESHandler, but the import path uses the cmreslogging namespace.
• AuthType

  CMRESHandler.AuthType

  AuthType is an inner enum class of CMRESHandler.
• IndexNameFrequency

  CMRESHandler.IndexNameFrequency

  IndexNameFrequency is an inner enum class of CMRESHandler.

Quickstart

This quickstart demonstrates how to configure and use CMRESHandler with basic authentication (if environment variables are set) to send log messages to an Elasticsearch instance. Logs will include additional custom fields and exceptions will be sent with full tracebacks. The `flush_frequency_in_sec` is set to 1 for quick testing, but should be adjusted for production.

import logging
import os
from cmreslogging.handlers import CMRESHandler

# Configure basic logging
log = logging.getLogger("PythonTest")
log.setLevel(logging.INFO)

# Elasticsearch host configuration
ES_HOST = os.environ.get('ES_HOST', 'localhost')
ES_PORT = int(os.environ.get('ES_PORT', 9200))
ES_INDEX_NAME = os.environ.get('ES_INDEX_NAME', 'python_app_logs')

# Optional: Basic Auth details
ES_USER = os.environ.get('ES_USER')
ES_PASSWORD = os.environ.get('ES_PASSWORD')

hosts = [{'host': ES_HOST, 'port': ES_PORT}]
auth_type = CMRESHandler.AuthType.NO_AUTH
auth_details = None

if ES_USER and ES_PASSWORD:
    auth_type = CMRESHandler.AuthType.BASIC_AUTH
    auth_details = (ES_USER, ES_PASSWORD)

try:
    handler = CMRESHandler(
        hosts=hosts,
        auth_type=auth_type,
        auth_details=auth_details,
        es_index_name=ES_INDEX_NAME,
        es_additional_fields={'App': 'MyAppName', 'Environment': 'Dev'},
        # Ensure flush frequency is reasonable for testing
        flush_frequency_in_sec=1
    )
    log.addHandler(handler)

    log.info("This is an info statement that will be logged into Elasticsearch.")
    log.warning("A warning message from the application.")
    try:
        1 / 0
    except ZeroDivisionError:
        log.exception("An exception occurred, should be logged with traceback.")

    print("Log messages sent to Elasticsearch (check your ES_INDEX_NAME in Elasticsearch).")
    # In a real application, you might need a small delay or explicit flush
    # if the application exits immediately after logging.
    # For this quickstart, we rely on flush_frequency_in_sec.

except Exception as e:
    print(f"Failed to initialize CMRESHandler or send logs: {e}")
    print("Ensure Elasticsearch is running and accessible at {ES_HOST}:{ES_PORT}.")