Logstash Python Formatter

Common errors

• ModuleNotFoundError: No module named 'logstash_formatter'

  cause The package was installed using a hyphenated name (`logstash-python-formatter`) or the import statement used a hyphenated name, but the actual installed package name on PyPI uses underscores.
  fix Ensure you installed with `pip install logstash_formatter` and imported with `from logstash_formatter import ...`.

• TypeError: Object of type <SomeObject> is not JSON serializable

  cause You are trying to log an object that the standard `json` module cannot convert into a JSON string, often when passed in `extra` or directly as `msg`.
  fix The `LogstashFormatter` has a `json_default` parameter. By default, it coerces unknown types to strings. If you have custom objects, either ensure they are converted to strings/JSON-serializable types before logging, or provide a custom `json_default` callable to the `LogstashFormatterV1` constructor to handle their serialization.

• Logstash shows 'msg' field instead of '@message' or 'message'

  cause You might be using `LogstashFormatter` instead of `LogstashFormatterV1`, or your Logstash configuration expects a different field name for the message.
  fix Use `LogstashFormatterV1` explicitly as it handles the `msg` to `message` conversion for Logstash 1.2.0 schema. Ensure your Logstash input configuration is set up to parse JSON correctly and expects the `message` field.

Warnings

• gotcha The PyPI package name is `logstash_formatter` (with an underscore) while the library is often referred to as `logstash-python-formatter` (with a hyphen). Ensure `pip install logstash_formatter` and `from logstash_formatter import ...` for correct usage.
  Fix: Always use `logstash_formatter` for installation and imports.
• gotcha When using `LogstashFormatterV1`, the original `msg` field in a LogRecord is specially handled and updated to `message` in the output JSON. If `msg` is a dictionary, its contents are merged as top-level fields instead of being treated as a message string.
  Fix: Be aware that direct message strings become `message`, while dictionaries passed as `msg` are flattened. Use the `extra` parameter for additional structured data alongside a string message.
• gotcha The keys in the `extra` dictionary passed to logging calls should not clash with standard LogRecord attributes (e.g., `name`, `levelname`, `lineno`). Overlapping keys might lead to unexpected behavior or overwriting of standard fields.
  Fix: Consult Python's `logging.Formatter` documentation for a list of reserved `LogRecord` attributes. Prefix custom `extra` fields (e.g., `my_app_field`) to avoid conflicts.
• deprecated This library (version 0.1.2, last updated 2018) appears to be in maintenance mode or largely inactive. More actively developed alternatives like `python-logstash` or `python-logstash-async` might offer broader features, better performance (e.g., asynchronous handling), and more current Logstash schema support.
  Fix: For new projects or if encountering limitations, consider evaluating `python-logstash` or `python-logstash-async` as alternatives, especially if asynchronous logging or newer Logstash features are required.

Install

• pip install logstash_formatter Install via pip

Imports

• LogstashFormatterV1
  wrong:

  from logstash_python_formatter import LogstashFormatterV1

  correct:

  from logstash_formatter import LogstashFormatterV1

  The PyPI package name uses underscores ('logstash_formatter'), while the project name often uses hyphens. Ensure you import from the correct package name.

Quickstart

This quickstart demonstrates how to configure the standard Python logging module to use `LogstashFormatterV1`. It sets up a basic logger with a stream handler, applies the formatter, and shows examples of logging simple messages, messages with extra fields, and exceptions, all formatted as JSON. The `LogstashFormatterV1` handles conversion of `msg` to `@message` and integrates `extra` dictionary fields.

import logging
from logstash_formatter import LogstashFormatterV1
import sys

# Configure a logger
logger = logging.getLogger('my_app')
logger.setLevel(logging.INFO)

# Create a StreamHandler to output logs to console
handler = logging.StreamHandler(sys.stdout)

# Create an instance of LogstashFormatterV1
# You can pass 'fmt' as a JSON string to add default extra fields or override source_host
# For example: formatter = LogstashFormatterV1(fmt='{"extra": {"service": "my-service"}}')
formatter = LogstashFormatterV1()

# Set the formatter for the handler
handler.setFormatter(formatter)

# Add the handler to the logger
logger.addHandler(handler)

# Log messages
logger.info('This is a simple info message.')
logger.warning('This is a warning message with some context.', extra={'user_id': 123, 'transaction_id': 'abc-123'})

try:
    1 / 0
except ZeroDivisionError:
    logger.error('An error occurred!', exc_info=True, extra={'error_code': 500})

# Logging a dictionary directly as a message (fields are merged into the top level)
logger.info({'event': 'user_login', 'username': 'testuser'})