Pygtail

Common errors

• UnicodeDecodeError: 'charmap' codec can't decode byte 0x__ in position __: character maps to <undefined>

  cause Attempting to read a log file with an encoding (e.g., UTF-8) that differs from the system's default encoding, particularly common on Windows systems expecting CP1252.
  fix Pass the `encoding` parameter to the `Pygtail` constructor: `tail = Pygtail('your.log', encoding='utf8')`. Adjust 'utf8' to the actual encoding of your log file if different.

• Pygtail stops updating offset file and always returns 0 for stats.

  cause This can occur when log rotation is performed by moving the original log file and creating a new empty one, rather than truncating it (e.g., `copytruncate`). Pygtail, by default, might not recognize this as a rotation if the inode remains the same but the file content is reset.
  fix Enable `copytruncate` support by initializing `Pygtail` with `copytruncate=True`: `tail = Pygtail('your.log', copytruncate=True)`. If log rotation involves compression, `pygtail` might also need custom log patterns using the `--log-pattern` command-line option, though this is not directly supported in the Python API for dynamic patterns.

• FileNotFoundError: [Errno 2] No such file or directory: 'some.log.offset'

  cause The default offset file name is derived from the log file. If Pygtail cannot create or access this file due to permissions or an invalid path, it will raise this error.
  fix Ensure the directory where the log file resides (or where you specify the offset file) has write permissions for the user running the Pygtail script. Alternatively, specify an explicit and writable `offset_file` path: `tail = Pygtail('some.log', offset_file='/var/log/my_app/some.log.offset')`.

Warnings

• gotcha When log files are rotated using a 'copytruncate' method (where the file is emptied and rewritten), Pygtail might not detect this by default, leading to missing new entries. This is because the inode remains the same, but the content is reset.
  Fix: Initialize `Pygtail` with `copytruncate=True` (e.g., `Pygtail('logfile.log', copytruncate=True)`). This enables support for `copytruncate`-style rotation.
• gotcha On Windows, Pygtail may encounter `UnicodeDecodeError` when reading log files containing non-ASCII characters, especially if the file was generated on a Linux system or uses UTF-8 encoding.
  Fix: Specify the correct encoding when initializing `Pygtail` using the `encoding` parameter (e.g., `Pygtail('logfile.log', encoding='utf8')`).
• gotcha Pygtail automatically creates an offset file (e.g., `logfile.log.offset`) to track read progress. If this offset file is deleted or becomes corrupted, Pygtail will re-read the entire log file from the beginning, which can lead to duplicate processing of log entries.
  Fix: Manage the lifecycle of the offset file carefully. If you need to re-read a file from the end if the offset file is missing, use the `read_from_end=True` parameter. For temporary reading without persistent offsets, you can specify `offset_file='/dev/null'` (on Unix-like systems) or use `save_on_end=False` and manually manage offsets with `tail.with_offsets()` and `tail.write_offset_to_file()`.

Install

• pip install pygtail Install with pip

Imports

• Pygtail
  wrong:

  import pygtail

  correct:

  from pygtail import Pygtail

  The primary class 'Pygtail' needs to be imported directly, not accessed via the top-level package name.

Quickstart

This quickstart demonstrates how to use Pygtail to read new lines from a log file. It first creates a dummy log file, reads its initial content, then appends new lines, and finally uses Pygtail again to read only the newly added lines. Pygtail automatically manages an offset file (by default `logfile.offset`) to keep track of already read content.

import sys
import os
from pygtail import Pygtail

# Create a dummy log file for demonstration
log_file_path = 'example.log'
with open(log_file_path, 'w') as f:
    f.write('Line 1\n')
    f.write('Line 2\n')

print(f"Reading new lines from {log_file_path}:")
for line in Pygtail(log_file_path):
    sys.stdout.write(line)

# Simulate new log entries
with open(log_file_path, 'a') as f:
    f.write('Line 3 (new)\n')
    f.write('Line 4 (new)\n')

print("\nReading newly added lines:")
for line in Pygtail(log_file_path):
    sys.stdout.write(line)

# Clean up dummy log and offset files
if os.path.exists(log_file_path):
    os.remove(log_file_path)
if os.path.exists(log_file_path + '.offset'):
    os.remove(log_file_path + '.offset')