Drain3 Log Template Miner

Warnings

• breaking Version 0.9.0 introduced significant breaking changes, including a major refactor of the API. The `LogViewer` module was removed, and `TemplateMinerConfig` was refactored from a static method `load()` to a class that can be instantiated and configured. Imports for `Drain3` and `TemplateMinerConfig` have also changed.
  Fix: Review the official documentation and update imports and configuration instantiation. For example, `config = TemplateMinerConfig()` instead of `config = TemplateMinerConfig.load('config.ini')`, and `from drain3 import Drain3`.
• gotcha Drain3 does not automatically persist its state (the learned log templates) by default. If your application restarts without explicit state saving and loading, it will lose all learned templates and start learning from scratch, leading to reprocessing and inconsistent cluster IDs.
  Fix: Configure a persistence backend (FILE, REDIS, or HTTP) using `TemplateMinerConfig`. Call `drain.save_state()` periodically or upon application shutdown, and `drain.load_state()` upon application startup to ensure state is maintained. Example: `config.persistence_type = 'FILE'; config.file_persistence_path = 'path/to/state.bin'`.
• gotcha The quality of log templates heavily depends on the `TemplateMinerConfig` parameters, especially `drain_sim_th` (similarity threshold) and `depth`. Incorrect settings can lead to overly generic or too specific templates, reducing the effectiveness of log parsing.
  Fix: Experiment with `drain_sim_th` and `depth` based on your log data characteristics. A lower `drain_sim_th` (e.g., 0.3-0.4) creates more general templates, while a higher value (e.g., 0.6-0.8) creates more specific ones. `depth` controls how many tokens are considered for tree traversal before a new node is created.

Install

• pip install drain3 Install stable version

Imports

• Drain3

  from drain3 import Drain3

  The main `Drain3` class was moved to the top-level package in v0.9.0. Older versions used `from drain3.drain import Drain3`.
• TemplateMinerConfig

  from drain3.template_miner_config import TemplateMinerConfig

  The `TemplateMinerConfig` class was moved and significantly refactored in v0.9.0. It is now imported from its own module.

Quickstart

This quickstart demonstrates how to initialize Drain3 with a basic configuration, add log messages, and retrieve the identified log clusters. For production use, consider configuring persistence (file, Redis, or HTTP) via `TemplateMinerConfig` to save and load the model state.

from drain3 import Drain3
from drain3.template_miner_config import TemplateMinerConfig
import os

# Configure Drain3. For production, consider loading from a file or Redis.
# Example for file persistence:
# config = TemplateMinerConfig.load('drain3.ini')
# Ensure 'persist_state_to_file' and 'state_file_path' are set in config.

config = TemplateMinerConfig()
config.load_default_config()
config.drain_sim_th = 0.4
config.depth = 4
# If you want to use file persistence (ensure directory exists and is writable):
# config.persistence_type = 'FILE'
# config.file_persistence_path = os.path.join(os.getcwd(), 'drain3_state.bin')

drain = Drain3(config)

log_messages = [
    "081109 203619 143 INFO dfs.DataNode$PacketResponder: PacketResponder "
    "0 for block blk_3886504917409280145 terminating",
    "081109 203619 369 INFO dfs.DataNode$PacketResponder: PacketResponder "
    "0 for block blk_-6755409170280820986 terminating",
    "081109 203620 357 INFO dfs.DataNode$PacketResponder: PacketResponder "
    "2 for block blk_814013142207908518 terminating",
    "081109 203620 543 INFO dfs.DataNode$DataXceiver: Receiving block blk_-6755409170280820986 "
    "src: /10.250.9.141:50106 dest: /10.250.9.141:50010",
]

for log_message in log_messages:
    cluster_id = drain.add_log_message(log_message)
    print(f"Log: '{log_message}' -> Cluster ID: {cluster_id}")

# After processing, it's good practice to save the state if using persistence
# if config.persistence_type == 'FILE':
#     drain.save_state()

print("\n--- Current Clusters ---")
for cluster in drain.drain.clusters:
    print(cluster)