Fluent Logger for Python

Warnings

• breaking Version 0.11.0 dropped official support for Python 3.5 and 3.6. The library now requires Python 3.7 or newer.
  Fix: Upgrade your Python environment to 3.7+ before upgrading fluent-logger to v0.11.0 or newer.
• gotcha The `event.Event()` API does not provide a mechanism to check for success or failure of log delivery, unlike directly using `FluentSender.emit()` which returns a boolean.
  Fix: For critical logging paths requiring delivery confirmation, consider using `fluent.sender.FluentSender` directly and calling its `emit()` method, which returns `True` on success or `False` on error. You can then use `logger.last_error()` to retrieve error details.
• breaking In v0.11.0, `FluentSender` introduced a new `forward_packet_error` option. When set to `True` (default behavior in previous versions was to catch), the sender will no longer catch exceptions during event serialization (e.g., if `msgpack` fails), allowing errors to propagate.
  Fix: If you relied on silent error suppression during serialization, you might need to adjust your error handling logic or explicitly set `forward_packet_error=False` when initializing `FluentSender` or `FluentHandler` (if supported directly). It's generally recommended to handle these exceptions for robustness.
• gotcha When using `FluentRecordFormatter`, how Python's `extra` dictionary for log records is processed can be non-obvious. It might either be merged directly into the top-level record or nested under an 'extra' key, potentially leading to unexpected field names or data structure in Fluentd.
  Fix: Explicitly define all desired fields in your `FluentRecordFormatter`'s `fmt` dictionary. For dynamic `extra` fields, configure `FluentRecordFormatter` with `exclude_attrs` to control which default `LogRecord` attributes are logged, or handle the merging in a custom formatter or `buffer_overflow_handler` if the default behavior is not suitable. Ensure your Fluentd configuration expects the structure sent by the formatter.
• gotcha `FluentSender.setup()` creates a global singleton sender instance. While convenient for simple applications, this can be problematic in complex applications requiring multiple distinct Fluentd connections or different root tags, especially in multi-threaded environments or when testing.
  Fix: For more control, instantiate `FluentSender` objects directly. `FluentSender` now supports `None` as a root tag, allowing multiple root tags within a single connection. This provides greater flexibility and isolation between loggers.

Install

• pip install fluent-logger Install stable version

Imports

• FluentSender

  from fluent import sender
  logger = sender.FluentSender('app_tag', host='localhost', port=24224)

  The top-level package for imports is `fluent`, not `fluent_logger` or `fluent-logger`.
• FluentHandler

  from fluent import handler
  fluent_handler = handler.FluentHandler('app.tag')

  The top-level package for imports is `fluent`, not `fluent_logger` or `fluent-logger`.
• FluentRecordFormatter

  from fluent import handler
  formatter = handler.FluentRecordFormatter({'host': '%(hostname)s', 'where': '%(module)s.%(funcName)s'})

  The top-level package for imports is `fluent`, not `fluent_logger` or `fluent-logger`.

Quickstart

This quickstart demonstrates how to configure the standard Python `logging` module to send structured logs to Fluentd using `FluentHandler` and `FluentRecordFormatter`. It includes capturing extra contextual data and exception tracebacks.

import logging
from fluent import handler
import os

# Configure Fluentd to listen on 0.0.0.0:24224 (default)
# For a quick test, you can run Fluentd with:
# <source>
#   @type forward
#   port 24224
#   bind 0.0.0.0
# </source>
# <match app.**>
#   @type stdout
# </match>

# Set up a Python logger
logger = logging.getLogger('my_app')
logger.setLevel(logging.INFO)

# Configure FluentHandler
fluent_host = os.environ.get('FLUENTD_HOST', 'localhost')
fluent_port = int(os.environ.get('FLUENTD_PORT', 24224))

# The tag 'app.follow' will be used in Fluentd configuration to route logs
h = handler.FluentHandler('app.follow', host=fluent_host, port=fluent_port)

# Optional: Configure a formatter for structured logs
# The dictionary keys are the output field names in Fluentd
custom_format = {
    'host': '%(hostname)s',
    'where': '%(module)s.%(funcName)s',
    'type': '%(levelname)s',
    'message': '%(message)s',
    'stack_trace': '%(exc_text)s'
}
formatter = handler.FluentRecordFormatter(custom_format)
h.setFormatter(formatter)

# Add the FluentHandler to the logger
logger.addHandler(h)

try:
    logger.info('This is an info message from Python!', extra={'user_id': 123, 'action': 'login'})
    raise ValueError("Something went wrong here!")
except ValueError as e:
    logger.error('An error occurred.', exc_info=True, extra={'error_code': 500})

print(f"Logs sent to Fluentd at {fluent_host}:{fluent_port} with tag 'app.follow'")
print("Check your Fluentd/Fluent Bit output.")