Python Logstash

0.4.8 · active · verified Mon Apr 13

python-logstash is a Python logging handler for Logstash. It allows applications to send log messages to a Logstash instance using UDP, TCP, or AMQP protocols. The library is currently at version 0.4.8 and focuses on providing reliable, direct log forwarding, with updates addressing Python 3 compatibility and bug fixes.

Warnings

gotcha The default `LogstashHandler` uses UDP. If you intend to use TCP for more reliable delivery, you must explicitly use `TCPLogstashHandler`.
Fix: Use `from logstash import TCPLogstashHandler` and instantiate `TCPLogstashHandler(host, port, version=1)` instead of `LogstashHandler`.
gotcha When using `extra` fields in log records, ensure the dictionary keys do not clash with reserved names used by the Python logging system (e.g., `pathname`, `lineno`, `levelname`).
Fix: Refer to Python's `logging.Formatter` documentation for a list of reserved `LogRecord` attribute names. Prefix custom fields (e.g., `my_custom_field`) or group them under a distinct key if conflicts are a concern.
gotcha Logstash itself requires proper configuration (e.g., `udp { port => 5959 codec => json }`) to receive messages. Common issues arise from misconfigured Logstash inputs (wrong port, protocol, or codec).
Fix: Verify your Logstash configuration file (`logstash.conf`) matches the protocol and port used by the Python handler and includes a `json` codec for parsing the incoming messages. Check Logstash logs for errors.
breaking Version 0.4.8 fixed Python 3 issues with JSON serialization. Earlier 0.4.x versions might have had broken JSON serialization for Python 3 environments, potentially leading to malformed logs.
Fix: Upgrade to `python-logstash==0.4.8` or newer to ensure correct JSON serialization in Python 3.
gotcha For high-performance or web applications, consider `python-logstash-async` (a fork) if synchronous logging blocking your main thread is an issue. The original `python-logstash` is synchronous.
Fix: If asynchronous logging is desired, install and use `python-logstash-async` (`pip install python-logstash-async`) which provides non-blocking log event submission.
deprecated The `distutils` module, used by older setup tools, is deprecated in Python 3.10+ and removed in Python 3.12. While not a direct API change, it can affect installation or packaging processes.
Fix: Ensure you are using a recent version of `setuptools` that relies on `build` and `installer` instead of `distutils`. This is usually handled by `pip` automatically with up-to-date environments.

Install

pip install python-logstash Install stable version

Imports

LogstashHandler
```
from logstash import LogstashHandler
```
This handler sends logs via UDP by default.
TCPLogstashHandler
```
from logstash import TCPLogstashHandler
```
Use this explicitly for TCP transport.
AMQPLogstashHandler
```
from logstash import AMQPLogstashHandler
```
Requires 'pika' package to be installed.

Quickstart

This quickstart demonstrates how to configure the Python logging module to send messages to Logstash using the default UDPLogstashHandler. It also shows how to include custom 'extra' fields in your log entries. Remember to set `version=1` for compatibility with modern Logstash versions and ensure your Logstash input is configured for UDP on the specified port.

import logging
import logstash
import sys
import os

# Configure Logstash host and port (use environment variables for production)
LOGSTASH_HOST = os.environ.get('LOGSTASH_HOST', 'localhost')
LOGSTASH_PORT = int(os.environ.get('LOGSTASH_PORT_UDP', 5959))

test_logger = logging.getLogger('my_logstash_app')
test_logger.setLevel(logging.INFO)

# Add Logstash UDP handler (default behavior)
test_logger.addHandler(logstash.LogstashHandler(LOGSTASH_HOST, LOGSTASH_PORT, version=1))

# Optional: Add Logstash TCP handler (uncomment and adjust port as needed)
# LOGSTASH_TCP_PORT = int(os.environ.get('LOGSTASH_PORT_TCP', 5000))
# test_logger.addHandler(logstash.TCPLogstashHandler(LOGSTASH_HOST, LOGSTASH_TCP_PORT, version=1))

# Log messages
test_logger.info('python-logstash: test logstash info message.')

# Add extra fields to logstash message
extra_data = {
    'test_string': 'python version: ' + repr(sys.version_info),
    'test_boolean': True,
    'test_dict': {'a': 1, 'b': 'c'},
    'test_float': 1.23,
    'test_integer': 123,
    'test_list': [1, 2, '3'],
}
test_logger.info('python-logstash: test extra fields', extra=extra_data)

try:
    1 / 0
except:
    test_logger.exception('python-logstash-app: Exception with stack trace!')

print(f"Logs sent to Logstash at {LOGSTASH_HOST}:{LOGSTASH_PORT} (UDP)")

view raw JSON →