{"id":5037,"library":"python-logstash-async","title":"Asynchronous Logstash Logging Handler","description":"Python-logstash-async is an asynchronous Python logging handler designed to send log events to a remote Logstash instance. It processes log events in a separate worker thread to avoid blocking the main application, crucial for performance-sensitive applications like web services. It supports TCP, UDP, and Beats protocols, with optional SSL for TCP, and can persist unsent logs to a SQLite database. The library is actively maintained with regular releases addressing bug fixes and introducing new features.","status":"active","version":"4.1.0","language":"en","source_language":"en","source_url":"https://github.com/eht16/python-logstash-async","tags":["logging","logstash","async","elk","observability"],"install":[{"cmd":"pip install python-logstash-async","lang":"bash","label":"Install latest version"}],"dependencies":[{"reason":"Requires Python 3.11 or newer for version 4.0.0+.","package":"python"}],"imports":[{"symbol":"AsynchronousLogstashHandler","correct":"from logstash_async.handler import AsynchronousLogstashHandler"},{"symbol":"LogstashFormatter","correct":"from logstash_async.formatter import LogstashFormatter"},{"symbol":"HttpTransport","correct":"from logstash_async.transport import HttpTransport"},{"symbol":"BeatsTransport","correct":"from logstash_async.transport import BeatsTransport"}],"quickstart":{"code":"import logging\nimport os\nfrom logstash_async.handler import AsynchronousLogstashHandler\nfrom logstash_async.formatter import LogstashFormatter\n\n# Configure Logstash host and port (replace with your Logstash details)\nLOGSTASH_HOST = os.environ.get('LOGSTASH_HOST', 'localhost')\nLOGSTASH_PORT = int(os.environ.get('LOGSTASH_PORT', '5959')) # Or 5000 for standard TCP/UDP\nLOGSTASH_DATABASE_PATH = os.environ.get('LOGSTASH_DB_PATH', 'logstash_events.db')\n\n# Get a logger instance\nlogger = logging.getLogger('my_app_logger')\nlogger.setLevel(logging.INFO)\n\n# Create a Logstash formatter\nformatter = LogstashFormatter(message_type='python-logstash', extra_prefix='dev', extra={'application': 'my-python-app'})\n\n# Create an asynchronous Logstash handler\n# It's recommended to specify a database_path for persistence across restarts\nhandler = AsynchronousLogstashHandler(\n    host=LOGSTASH_HOST,\n    port=LOGSTASH_PORT,\n    database_path=LOGSTASH_DATABASE_PATH,\n    # For TCP/Beats with SSL, set ssl_enable=True and configure certs\n    # ssl_enable=True,\n    # ssl_verify=True,\n    # ca_certs='/path/to/ca.crt'\n)\n\nhandler.setFormatter(formatter)\nlogger.addHandler(handler)\n\n# Add a console handler for local debugging\nconsole_handler = logging.StreamHandler()\nconsole_handler.setFormatter(logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s'))\nlogger.addHandler(console_handler)\n\n# Log some messages\nlogger.info('This is an info message from the quickstart.', extra={'user_id': 123})\nlogger.warning('A warning occurred with some extra data.', extra={'component': 'auth', 'status': 'failed'})\nlogger.error('An error message with detailed context.')\n\n# Ensure all queued logs are sent before exiting\n# In a real application, this might be handled by an atexit hook or proper shutdown logic\nlogger.info('Flushing pending log events...')\nhandler.close() # This will attempt to flush remaining events","lang":"python","description":"This quickstart demonstrates how to configure `python-logstash-async` with `AsynchronousLogstashHandler` and `LogstashFormatter`. It sets up a logger to send INFO, WARNING, and ERROR messages to a Logstash instance, including extra fields. A `database_path` is specified for persistent storage of events in case Logstash is unreachable or the application restarts. The example also includes a console handler for immediate feedback during development."},"warnings":[{"fix":"Upgrade to Python 3.11 or newer, or stick to `python-logstash-async` versions < 4.0.0.","message":"Version 4.0.0 and newer drop support for Python versions 3.8, 3.9, and 3.10.","severity":"breaking","affected_versions":"4.0.0+"},{"fix":"Always provide a `database_path` argument to `AsynchronousLogstashHandler` to enable event persistence to a SQLite database. For example: `database_path='logstash_events.db'`.","message":"Using `AsynchronousLogstashHandler` without specifying a `database_path` will result in the loss of unsent log messages if the application process restarts or crashes.","severity":"gotcha","affected_versions":"All versions"},{"fix":"For multiple handlers in a single process, configure them to use the *same* `database_path`. Avoid using `AsynchronousLogstashHandler` directly in multi-process scenarios where logs need to be flushed from each child process independently; consider using `logging.QueueHandler` with a `QueueListener` in the parent process or writing to files and using a tool like Filebeat for robust multi-process logging.","message":"Using multiple instances of `AsynchronousLogstashHandler` with *different* `database_path` settings in the same process will not work as expected. Only the `database_path` from the first handler that emits a log event will be used by the single underlying `LogProcessingWorker`. Using `python-logstash-async` with standard Python `multiprocessing` can also be problematic due to how the worker thread and database are managed.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Upgrade to version 4.0.2 or newer to benefit from fixes addressing socket handling and preventing hangs during network errors.","message":"Older versions (prior to 4.0.2) might experience hangs on socket errors or improper shutdown of UDP sockets, potentially leading to resource leaks or unresponsiveness.","severity":"gotcha","affected_versions":"< 4.0.2"},{"fix":"Refer to Python's `logging.Formatter` documentation for a list of reserved attributes. Use a `extra_prefix` in `LogstashFormatter` to avoid conflicts (e.g., `extra_prefix='my_app_data'`).","message":"When adding `extra` fields to log messages, ensure the keys do not clash with reserved names used by Python's logging system (e.g., `levelname`, `asctime`, `filename`).","severity":"gotcha","affected_versions":"All versions"}],"env_vars":null,"last_verified":"2026-04-12T00:00:00.000Z","next_check":"2026-07-11T00:00:00.000Z"}