Concurrent Log Handler

0.9.29 · active · verified Fri Apr 10

A robust drop-in replacement for Python's `RotatingFileHandler` and `TimedRotatingFileHandler` that supports safe concurrent writes from multiple processes and threads. It also includes features like gzip compression and better Windows compatibility. The library is actively maintained, with frequent minor releases addressing bugs and adding features, typically every few months.

Warnings

breaking Python 2.x support was dropped. Versions 0.9.23 and higher require Python 3.6 or higher.
Fix: Upgrade your Python environment to 3.6 or newer if you are using an older Python 2.x interpreter.
gotcha When using `fork()` to create new processes, log handlers inherited by child processes could experience race conditions, leading to corrupted logs or deadlocks.
Fix: As of 0.9.29, inherited handlers automatically reopen lock files for isolation. However, it is still strongly recommended to re-initialize your logging configuration *after* calling `os.fork()` in child processes to ensure full isolation and robust behavior.
deprecated The `queue.py` `setup_logging_queues()` function has been deprecated and should no longer be used.
Fix: Remove calls to `setup_logging_queues()`. The standard synchronous logging calls with this handler are generally sufficient and recommended.
gotcha Before 0.9.28, if an application shut down during a scheduled rollover time (e.g., for `ConcurrentTimedRotatingFileHandler`), the handler might miss the rollover entirely and skip to the next scheduled time without rotating accumulated logs.
Fix: Upgrade to version 0.9.28 or later. The handler now performs catch-up rollovers on initialization if scheduled times were missed.
gotcha Since 0.9.26, the handler keeps file handles open (`keep_file_open=True` by default) for performance. However, on Windows, the log file must still be closed after each write for rollovers to occur correctly due to filesystem API behavior.
Fix: No direct fix needed, but be aware of this internal behavior. If you experience unexpected issues with file handles, you can revert to the old behavior by explicitly setting `keep_file_open=False` in the handler constructor, though this may impact performance.

Install

pip install concurrent-log-handler Install stable version

Imports

ConcurrentRotatingFileHandler
```
from concurrent_log_handler import ConcurrentRotatingFileHandler
```
The standard RotatingFileHandler does not provide robust concurrent access from multiple processes.
ConcurrentTimedRotatingFileHandler
```
from concurrent_log_handler import ConcurrentTimedRotatingFileHandler
```
The standard TimedRotatingFileHandler does not provide robust concurrent access from multiple processes.

Quickstart

This example demonstrates how to set up a basic logger using `ConcurrentRotatingFileHandler` to ensure log file rotation works safely across multiple processes, preventing data loss or corruption due to concurrent writes. It configures a log file to rotate after reaching 10MB, keeping 5 backup copies.

import logging
import os
from concurrent_log_handler import ConcurrentRotatingFileHandler

log_file_path = os.path.join(os.getcwd(), 'app.log')

# Configure logging
log = logging.getLogger('my_app')
log.setLevel(logging.INFO)

# Use ConcurrentRotatingFileHandler
# Rotate log after reaching 10MB, keep 5 old copies.
# Set encoding to 'utf-8' to avoid UnicodeEncodeError on some systems.
handler = ConcurrentRotatingFileHandler(
    log_file_path, 'a', 10 * 1024 * 1024, 5, encoding='utf-8'
)

formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
log.addHandler(handler)

log.info('This is an informational message.')
log.warning('This is a warning message.')
log.error('This is an error message.')

print(f"Log messages written to: {log_file_path}")

view raw JSON →