Asynchronous Logstash Logging Handler

Warnings

• breaking Version 4.0.0 and newer drop support for Python versions 3.8, 3.9, and 3.10.
  Fix: Upgrade to Python 3.11 or newer, or stick to `python-logstash-async` versions < 4.0.0.
• gotcha Using `AsynchronousLogstashHandler` without specifying a `database_path` will result in the loss of unsent log messages if the application process restarts or crashes.
  Fix: Always provide a `database_path` argument to `AsynchronousLogstashHandler` to enable event persistence to a SQLite database. For example: `database_path='logstash_events.db'`.
• gotcha Using multiple instances of `AsynchronousLogstashHandler` with *different* `database_path` settings in the same process will not work as expected. Only the `database_path` from the first handler that emits a log event will be used by the single underlying `LogProcessingWorker`. Using `python-logstash-async` with standard Python `multiprocessing` can also be problematic due to how the worker thread and database are managed.
  Fix: For multiple handlers in a single process, configure them to use the *same* `database_path`. Avoid using `AsynchronousLogstashHandler` directly in multi-process scenarios where logs need to be flushed from each child process independently; consider using `logging.QueueHandler` with a `QueueListener` in the parent process or writing to files and using a tool like Filebeat for robust multi-process logging.
• gotcha Older versions (prior to 4.0.2) might experience hangs on socket errors or improper shutdown of UDP sockets, potentially leading to resource leaks or unresponsiveness.
  Fix: Upgrade to version 4.0.2 or newer to benefit from fixes addressing socket handling and preventing hangs during network errors.
• gotcha When adding `extra` fields to log messages, ensure the keys do not clash with reserved names used by Python's logging system (e.g., `levelname`, `asctime`, `filename`).
  Fix: Refer to Python's `logging.Formatter` documentation for a list of reserved attributes. Use a `extra_prefix` in `LogstashFormatter` to avoid conflicts (e.g., `extra_prefix='my_app_data'`).

Install

• pip install python-logstash-async Install latest version

Imports

• AsynchronousLogstashHandler

  from logstash_async.handler import AsynchronousLogstashHandler

• LogstashFormatter

  from logstash_async.formatter import LogstashFormatter

• HttpTransport

  from logstash_async.transport import HttpTransport

• BeatsTransport

  from logstash_async.transport import BeatsTransport

Quickstart

This quickstart demonstrates how to configure `python-logstash-async` with `AsynchronousLogstashHandler` and `LogstashFormatter`. It sets up a logger to send INFO, WARNING, and ERROR messages to a Logstash instance, including extra fields. A `database_path` is specified for persistent storage of events in case Logstash is unreachable or the application restarts. The example also includes a console handler for immediate feedback during development.

import logging
import os
from logstash_async.handler import AsynchronousLogstashHandler
from logstash_async.formatter import LogstashFormatter

# Configure Logstash host and port (replace with your Logstash details)
LOGSTASH_HOST = os.environ.get('LOGSTASH_HOST', 'localhost')
LOGSTASH_PORT = int(os.environ.get('LOGSTASH_PORT', '5959')) # Or 5000 for standard TCP/UDP
LOGSTASH_DATABASE_PATH = os.environ.get('LOGSTASH_DB_PATH', 'logstash_events.db')

# Get a logger instance
logger = logging.getLogger('my_app_logger')
logger.setLevel(logging.INFO)

# Create a Logstash formatter
formatter = LogstashFormatter(message_type='python-logstash', extra_prefix='dev', extra={'application': 'my-python-app'})

# Create an asynchronous Logstash handler
# It's recommended to specify a database_path for persistence across restarts
handler = AsynchronousLogstashHandler(
    host=LOGSTASH_HOST,
    port=LOGSTASH_PORT,
    database_path=LOGSTASH_DATABASE_PATH,
    # For TCP/Beats with SSL, set ssl_enable=True and configure certs
    # ssl_enable=True,
    # ssl_verify=True,
    # ca_certs='/path/to/ca.crt'
)

handler.setFormatter(formatter)
logger.addHandler(handler)

# Add a console handler for local debugging
console_handler = logging.StreamHandler()
console_handler.setFormatter(logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s'))
logger.addHandler(console_handler)

# Log some messages
logger.info('This is an info message from the quickstart.', extra={'user_id': 123})
logger.warning('A warning occurred with some extra data.', extra={'component': 'auth', 'status': 'failed'})
logger.error('An error message with detailed context.')

# Ensure all queued logs are sent before exiting
# In a real application, this might be handled by an atexit hook or proper shutdown logic
logger.info('Flushing pending log events...')
handler.close() # This will attempt to flush remaining events